Breast cancer lymph node benignity-malignancy classification system and method based on preoperative and intraoperative images
By combining preoperative multimodal imaging and intraoperative fluorescence imaging, generative adversarial networks and deep learning techniques are used to classify benign and malignant lymph nodes in breast cancer, solving the problem of incorrect lymph node removal in existing technologies and achieving more accurate analysis and assisted diagnosis and treatment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF AUTOMATION CHINESE ACAD OF SCI
- Filing Date
- 2022-08-05
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies suffer from errors in lymph node dissection in breast cancer, and current medical image analysis methods typically only process information at a single point in time, failing to construct a complete computer-aided diagnostic scheme. Furthermore, deep learning-based fluorescence image processing is insufficient.
By combining preoperative multimodal images and intraoperative fluorescence images, image enhancement and feature extraction were performed using generative adversarial networks, deep residual networks, and Transformer encoders to construct a classification model for benign and malignant lymph nodes in breast cancer. The model was then classified based on the results of image analysis at different stages.
It reduces the probability of incorrect lymph node removal in breast cancer, improves the accuracy and precision of analysis results, and provides a more comprehensive computer-aided diagnosis and treatment solution.
Smart Images
Figure CN115222992B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and specifically relates to a classification system and method for benign and malignant lymph nodes of breast cancer based on preoperative and intraoperative images. Background Technology
[0002] Medical imaging provides visual information related to tumors. Through different artificial intelligence coding methods, the most relevant features in the images to disease progression can be deeply mined, thereby achieving image classification and assisting in clinical diagnosis.
[0003] Current medical image analysis methods mainly fall into two categories: those based on predefined features and those based on deep learning. These two methods typically operate independently for a specific task. However, in practical applications, decision support usually involves a comprehensive consideration of long-term, multimodal medical image analysis results. Since images at different time points contain different information, relying solely on the results of a single image analysis is insufficient. Furthermore, targeted processing of images of a specific modality can provide more accurate predictive models, while research on the correlation between intraoperative fluorescence image sequences and lymph node metastasis status using deep learning is still in its early stages.
[0004] Therefore, this field needs to accurately analyze images of patients at different stages of disease progression, and also needs to combine the analysis results of different stages to realize a complete computer-aided diagnosis and treatment plan to meet the needs of assisting clinical diagnosis. Summary of the Invention
[0005] To address the aforementioned problems in the prior art, namely the issue of incorrect lymph node dissection in breast cancer, this invention provides a classification system for benign and malignant breast cancer lymph nodes based on preoperative and intraoperative images. The classification system includes:
[0006] The preoperative multimodal image preprocessing and encoding module extracts high-dimensional predefined features related to lymph nodes based on the region of interest in the preoperative multimodal images of patients with metastatic breast cancer in lymph nodes.
[0007] The intraoperative fluorescence image sequence preprocessing and encoding module enhances the patient's intraoperative fluorescence imaging sequence by generating adversarial network variants, and then encodes the overall sequence based on deep residual network and Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images.
[0008] The intraoperative fluorescence intensity time-varying sequence preprocessing and encoding module obtains the time-varying pixel intensity of fluorescence brightness in the enhanced sequence through the region of interest of the fluorescence imaging sequence, fits the biophysical model parameters, and obtains the local features of intraoperative lymph nodes in fluorescence images.
[0009] The intraoperative sentinel / non-sentinel lymph node benign / malignant classification module is based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images. It constructs and trains an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model, and uses the trained model to classify breast cancer lymph nodes as benign or malignant.
[0010] In some preferred embodiments, the preoperative multimodal imaging includes multi-sequence magnetic resonance imaging, mammography images, and ultrasound images from the patient's preoperative examination.
[0011] In some preferred embodiments, the region of interest in the preoperative multimodal image and the region of interest in the fluorescence imaging sequence are tumor areas where the proportion of surface hemorrhage or edema is below a set threshold.
[0012] In some preferred embodiments, the high-dimensional predefined features include shape features, intensity features, texture features, and wavelet features.
[0013] In some preferred embodiments, the generative adversarial network variant is built on a deep residual convolutional neural network, comprising one convolutional module and four residual modules connected in sequence;
[0014] The deep residual network includes sequentially connected convolutional modules, residual modules, and global average pooling layers. The convolutional kernels of the convolutional layers of the modules are 3×3×3.
[0015] The Transformer encoder includes two sequentially connected Transformer coding layers, and the Transformer coding layers include an 8-head attention mechanism.
[0016] In some preferred embodiments, the biophysical model parameters include the inflow behavior and exponential decay of the fluorescence signal.
[0017] In some preferred embodiments, the method for classifying the benign or malignant nature of breast cancer lymph nodes is as follows:
[0018] The process of assessing sentinel lymph nodes is terminated if the result is negative; otherwise, further assessment is needed to determine the benign or malignant nature of non-sentinel lymph nodes.
[0019] In another aspect, the present invention proposes a method for classifying benign and malignant breast cancer lymph nodes based on preoperative and intraoperative imaging, the classification method comprising:
[0020] Step S100: Extract high-dimensional predefined features related to lymph nodes based on the region of interest of preoperative multimodal images of breast cancer patients with lymph node metastasis;
[0021] Step S200: Image enhancement of the patient's intraoperative fluorescence imaging sequence is performed by generating an adversarial network variant to obtain the patient's enhanced intraoperative fluorescence imaging sequence;
[0022] Step S300: Encode the patient's intraoperative enhanced fluorescence imaging sequence based on a deep residual network and a Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images;
[0023] Step S400: Obtain the time-varying pixel intensity of fluorescence brightness in the intraoperative enhanced fluorescence imaging sequence of the patient through the region of interest of the fluorescence imaging sequence, fit the biophysical model parameters, and obtain the local features of intraoperative lymph nodes in the fluorescence image.
[0024] Step S500: Based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images, construct and train an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model.
[0025] Step S600: Acquire preoperative / intraoperative images of the patient and classify benign / malignant lymph nodes of breast cancer using a trained intraoperative sentinel / non-sentinel lymph node benign / malignant classification model.
[0026] In some preferred embodiments, the biophysical model parameters are expressed as follows:
[0027]
[0028] in, Indicates fluorescence intensity, This indicates the time elapsed between the start of the video and the tracer injection. and These represent the injection and evacuation times of the coded tracer, respectively. A unitless number representing the existence and persistence of intensity oscillations. This indicates the overall intensity of the fluorescence profile.
[0029] In some preferred embodiments, step S600 includes:
[0030] Step S601: Perform high-dimensional predefined features of preoperative multimodal images and stitch together global and local features of intraoperative fluorescence images to obtain stitched features;
[0031] Step S602: Input the spliced features into the fully connected layer of the trained intraoperative sentinel lymph node benign and malignant classification model to obtain the benign and malignant probability of the sentinel lymph node.
[0032] Step S603: If the probability of benign or malignant sentinel lymph node is lower than a set threshold, the sentinel lymph node is negative; otherwise, the sentinel lymph node is positive, and the process proceeds to step S604.
[0033] Step S604: Input the spliced features into the fully connected layer of the trained intraoperative non-sentinel lymph node benign / malignant classification model to obtain the benign / malignant probability of non-sentinel lymph nodes.
[0034] The beneficial effects of this invention are:
[0035] (1) The present invention is a classification system for benign and malignant lymph nodes of breast cancer based on preoperative and intraoperative images. It addresses the problems that existing medical image processing and modeling methods usually only extract and process information at a single time point, failing to construct a complete computer-aided diagnosis scheme, and the current fluorescence image processing steps based on deep learning are insufficient. It combines images of patients at different stages of disease progression for precise analysis, and combines the analysis results of different stages to achieve a complete computer-aided diagnosis and treatment scheme.
[0036] (2) The present invention is a classification system for benign and malignant lymph nodes of breast cancer based on preoperative and intraoperative images, which reduces the probability of incorrect lymph node dissection in existing breast cancer and effectively improves the accuracy and precision of the analysis results, thereby providing better auxiliary diagnosis and treatment options. Attached Figure Description
[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0038] Figure 1 This is a schematic diagram of the module composition of the breast cancer lymph node benign and malignant classification system based on preoperative and intraoperative images of the present invention;
[0039] Figure 2 This is a flowchart illustrating the method for classifying benign and malignant lymph nodes in breast cancer based on preoperative and intraoperative imaging, as per the present invention. Detailed Implementation
[0040] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0042] This invention provides a classification system for benign and malignant lymph nodes in breast cancer based on preoperative and intraoperative imaging, the classification system comprising:
[0043] The preoperative multimodal image preprocessing and encoding module extracts high-dimensional predefined features related to lymph nodes based on the region of interest in the preoperative multimodal images of patients with metastatic breast cancer in lymph nodes.
[0044] The intraoperative fluorescence image sequence preprocessing and encoding module enhances the patient's intraoperative fluorescence imaging sequence by generating adversarial network variants, and then encodes the overall sequence based on deep residual network and Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images.
[0045] The intraoperative fluorescence intensity time-varying sequence preprocessing and encoding module obtains the time-varying pixel intensity of fluorescence brightness in the enhanced sequence through the region of interest of the fluorescence imaging sequence, fits the biophysical model parameters, and obtains the local features of intraoperative lymph nodes in fluorescence images.
[0046] The intraoperative sentinel / non-sentinel lymph node benign / malignant classification module is based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images. It constructs and trains an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model, and uses the trained model to classify breast cancer lymph nodes as benign or malignant.
[0047] To more clearly explain the breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative imaging of this invention, the following will be combined with... Figure 1 The modules in the embodiments of the present invention will be described in detail below.
[0048] The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative images according to the first embodiment of the present invention includes a preoperative multimodal image preprocessing and encoding module, an intraoperative fluorescence image sequence preprocessing and encoding module, an intraoperative fluorescence intensity time-varying sequence preprocessing and encoding module, and an intraoperative sentinel / non-sentinel lymph node benign / malignant classification module. Each module is described in detail below:
[0049] The preoperative multimodal image preprocessing and encoding module extracts high-dimensional predefined features related to lymph nodes based on the regions of interest in preoperative multimodal images of patients with metastatic breast cancer in lymph nodes.
[0050] Preoperative multimodal imaging includes multi-sequence magnetic resonance imaging, mammography images, and ultrasound images from the patient's preoperative examination.
[0051] The region of interest includes the complete tumor area and some surrounding tissue. During the selection process, the tumor area should be selected as much as possible, and areas with obvious surface hemorrhage or edema should be avoided. That is, the region of interest is the tumor area where the proportion of surface hemorrhage or edema is less than a set threshold.
[0052] High-dimensional predefined features include shape features, intensity features, texture features, and wavelet features.
[0053] In one embodiment of the present invention, the extraction process of high-dimensional predefined features specifically includes:
[0054] Given mammograms, ultrasound, and MRI images obtained from preoperative breast cancer examinations;
[0055] The regions of interest (ROIs) of different modal images were manually delineated. The ROIs included the entire cancerous lesion area and the surrounding tissue.
[0056] For the regions of interest in magnetic resonance imaging, N4 bias field correction, resampling, and intensity normalization are performed; for the regions of interest in mammography and ultrasound, intensity normalization is performed to complete the preprocessing of the regions of interest.
[0057] Based on the preprocessed region of interest, high-dimensional predefined features are extracted. These high-dimensional predefined features include shape features, intensity features, texture features, and wavelet features.
[0058] The shape features include the length, volume, surface area, and edge smoothness of the lesion; the intensity features include the mean, variance, skewness, and kurtosis of the lesion's gray level; the texture features are extracted based on the Gray-Level Co-Occurrence Matrix (GLCM), Gray-Level Run Length Matrix (GLLM), and Gray-Level Size Zone Matrix (GLSZM); the wavelet features are extracted based on multi-scale wavelet filtering of the image, followed by feature extraction from images in different wavelet domains.
[0059] Finally, the features are stitched together and then filtered to obtain predefined multimodal radiomics features of lymph nodes, namely high-dimensional predefined features related to lymph nodes.
[0060] The intraoperative fluorescence image sequence preprocessing and encoding module enhances the patient's intraoperative fluorescence imaging sequence by generating adversarial network variants, and then encodes the overall sequence based on deep residual network and Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images.
[0061] The generative adversarial network variant is built on a deep residual convolutional neural network, consisting of one sequentially connected convolutional module and four residual modules;
[0062] The deep residual network includes sequentially connected convolutional modules, residual modules, and global average pooling layers, wherein the convolutional kernels of the convolutional layers are 3×3×3.
[0063] The Transformer encoder comprises two sequentially connected Transformer coding layers, each including an 8-head attention mechanism.
[0064] In one embodiment of the present invention, the process of extracting global features of intraoperative lymph nodes in fluorescence images specifically includes:
[0065] A sequence of fluorescence images was obtained by signal reconstruction after intraoperative injection of ICG (indocyanine green, ICG). The tracers used in intraoperative fluorescence imaging included indocyanine green (ICG) and methylene blue dye (BD). After absorption and scattering by the tissue, the fluorescence was transmitted to the body surface and collected and processed by a highly sensitive intraoperative optical detector to reconstruct the fluorescence image of the cancerous lesion area.
[0066] Image enhancement was performed on the patient's intraoperative fluorescence imaging sequence. The image enhancement included: 1. Downsampling the high-resolution image and then re-enlarging it to its original size using bicubic interpolation with a 4× scaling factor; 2. Training a generative adversarial network (GAN) by inputting pairs of high-resolution and low-resolution images. A least-squares GAN with 7 dense residual blocks and a spectral normalization layer was used in the generator; 3. The GAN was optimized using full gradient loss to produce sharpened edges and less artifact texture; 4. The original low-resolution image was input into the network to obtain the enhanced image sequence.
[0067] The enhanced image sequence is input into a deep residual convolutional neural network comprising 17 convolutional layers, including one sequentially connected convolutional module and four residual modules, and the output is N. M 512-dimensional feature vector;
[0068] N M The 512-dimensional feature vector is input into the Transformer encoder, which includes two sequentially connected Transformer coding layers. Each coding layer adopts an 8-head attention mechanism to obtain the 512-dimensional global feature encoding of intraoperative lymph node fluorescence images under the attention mechanism, that is, to obtain the global features of intraoperative lymph nodes in fluorescence images.
[0069] The intraoperative fluorescence intensity time-varying sequence preprocessing and encoding module obtains the time-varying pixel intensity of fluorescence brightness in the enhanced sequence through the region of interest of the fluorescence imaging sequence, fits the biophysical model parameters, and obtains the local features of intraoperative lymph nodes in fluorescence images.
[0070] The biophysical model parameters include the inflow behavior of fluorescence signals and exponential decay.
[0071] In one embodiment of the present invention, the process of extracting local features of intraoperative lymph nodes in fluorescence imaging specifically includes:
[0072] Manually delineate the region of interest in the fluorescence imaging sequence;
[0073] Calculate the time-varying pixel intensity of ICG luminance within the ROI of consecutive video frames;
[0074] The extracted intensity data were fitted to the following parametric curve derived from the biophysical model to capture the inflow behavior and exponential decay of ICG, as shown in Equation (1):
[0075]
[0076] in, Indicates fluorescence intensity, This indicates the time elapsed between the start of the video and the tracer injection. and These represent the injection and evacuation times of the coded tracer, respectively. A unitless number representing the existence and persistence of intensity oscillations. Indicates the overall intensity of the fluorescence profile;
[0077] The parameters are estimated by minimizing the mismatch between the observed data and the parameter curve, thus obtaining the local feature encoding of intraoperative lymph node fluorescence images, that is, obtaining the local features of intraoperative lymph nodes in fluorescence images.
[0078] The intraoperative sentinel / non-sentinel lymph node benign / malignant classification module is based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images. It constructs and trains an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model, and uses the trained model to classify breast cancer lymph nodes as benign or malignant.
[0079] The method for classifying benign and malignant lymph nodes in breast cancer is as follows:
[0080] The process of assessing sentinel lymph nodes is terminated if the result is negative; otherwise, further assessment is needed to determine the benign or malignant nature of non-sentinel lymph nodes.
[0081] In one embodiment of the present invention, the process of classifying benign and malignant lymph nodes in breast cancer specifically includes:
[0082] First, sentinel lymph node status prediction is performed: the preoperative multimodal radiomics features and the global and local features of intraoperative fluorescence imaging are respectively input into their respective fully connected layers, stitched together, and then input into the subsequent fully connected layers to output the benign and malignant probabilities of the sentinel lymph nodes. The cross-entropy loss function is used to update the parameters.
[0083] If the patient is predicted to be negative for sentinel lymph nodes, the prediction process is terminated.
[0084] If the patient is predicted to be positive for sentinel lymph nodes, repeat the above feature splicing and prediction steps. The predicted value is the probability of benign or malignant non-sentinel lymph node, and the parameters are updated using a class-balanced cross-entropy loss function.
[0085] The final result is the prediction of the patient's sentinel / non-sentinel lymph node status.
[0086] The second embodiment of the present invention provides a method for classifying benign and malignant breast cancer lymph nodes based on preoperative and intraoperative images, the classification method comprising:
[0087] Step S100: Based on the region of interest in the preoperative multimodal images of breast cancer patients with lymph node metastasis, extract high-dimensional predefined features related to lymph nodes.
[0088] Step S200: Image enhancement of the patient's intraoperative fluorescence imaging sequence is performed by generating an adversarial network variant to obtain the patient's enhanced intraoperative fluorescence imaging sequence.
[0089] Step S300: Encode the patient's intraoperative enhanced fluorescence imaging sequence based on a deep residual network and a Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images;
[0090] Step S400: Obtain the time-varying pixel intensity of fluorescence brightness in the intraoperative enhanced fluorescence imaging sequence of the patient through the region of interest of the fluorescence imaging sequence, fit the biophysical model parameters, and obtain the local features of intraoperative lymph nodes in the fluorescence image.
[0091] Step S500: Based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images, construct and train an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model.
[0092] Step S600: Acquire preoperative / intraoperative images of the patient and classify the benign or malignant nature of breast cancer lymph nodes using a trained intraoperative sentinel / non-sentinel lymph node benign / malignant classification model.
[0093] Step S601: Perform high-dimensional predefined features of preoperative multimodal images and stitch together global and local features of intraoperative fluorescence images to obtain stitched features;
[0094] Step S602: Input the spliced features into the fully connected layer of the trained intraoperative sentinel lymph node benign and malignant classification model to obtain the benign and malignant probability of the sentinel lymph node.
[0095] Step S603: If the probability of benign or malignant sentinel lymph node is lower than a set threshold, the sentinel lymph node is negative; otherwise, the sentinel lymph node is positive, and the process proceeds to step S604.
[0096] Step S604: Input the spliced features into the fully connected layer of the trained intraoperative non-sentinel lymph node benign / malignant classification model to obtain the benign / malignant probability of non-sentinel lymph nodes.
[0097] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related explanations of the methods described above can be found in the corresponding processes in the foregoing system embodiments, and will not be repeated here.
[0098] It should be noted that the breast cancer lymph node benign / malignant classification system and method based on preoperative and intraoperative images provided in the above embodiments are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.
[0099] A device according to a third embodiment of the present invention includes:
[0100] At least one processor; and
[0101] A memory communicatively connected to at least one of the processors; wherein,
[0102] The memory stores instructions that can be executed by the processor to implement the above-described method for classifying benign and malignant lymph nodes in breast cancer based on preoperative and intraoperative images.
[0103] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are executed by the computer to implement the above-described method for classifying benign and malignant lymph nodes of breast cancer based on preoperative and intraoperative images.
[0104] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0105] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.
[0106] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.
[0107] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.
[0108] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A classification system for benign and malignant lymph nodes in breast cancer based on preoperative and intraoperative images, characterized in that, The classification system includes: The preoperative multimodal image preprocessing and encoding module extracts high-dimensional predefined features related to lymph nodes based on the region of interest in the preoperative multimodal images of patients with metastatic breast cancer in lymph nodes. The intraoperative fluorescence image sequence preprocessing and encoding module enhances the patient's intraoperative fluorescence imaging sequence by generating adversarial network variants, and then encodes the overall sequence based on deep residual network and Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images. The intraoperative fluorescence intensity time-varying sequence preprocessing and encoding module obtains the time-varying pixel intensity of fluorescence brightness in the enhanced sequence through the region of interest of the fluorescence imaging sequence, fits the biophysical model parameters, and obtains the local features of intraoperative lymph nodes in fluorescence images. The intraoperative sentinel / non-sentinel lymph node benign / malignant classification module is based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images. It constructs and trains an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model, and uses the trained model to classify benign / malignant lymph nodes in breast cancer. The breast cancer lymph node benign and malignant classification system based on the preoperative and intraoperative images includes the following steps: Step S100, extracting high-dimensional predefined features related to lymph nodes based on the region of interest of the preoperative multimodal images of patients with metastatic breast cancer. Step S200: Image enhancement of the patient's intraoperative fluorescence imaging sequence is performed by generating an adversarial network variant to obtain the patient's enhanced intraoperative fluorescence imaging sequence; Step S300: Encode the patient's intraoperative enhanced fluorescence imaging sequence based on a deep residual network and a Transformer encoder to obtain the global features of intraoperative lymph nodes in fluorescence images; Step S400: Obtain the time-varying pixel intensity of fluorescence brightness in the intraoperative enhanced fluorescence imaging sequence of the patient through the region of interest of the fluorescence imaging sequence, fit the biophysical model parameters, and obtain the local features of intraoperative lymph nodes in the fluorescence image. Step S500: Based on the high-dimensional predefined features of preoperative multimodal images and the global and local features of intraoperative fluorescence images, construct and train an intraoperative sentinel / non-sentinel lymph node benign / malignant classification model. Step S600: Acquire preoperative / intraoperative images of the patient and classify benign and malignant lymph nodes of breast cancer using a trained intraoperative sentinel / non-sentinel lymph node benign and malignant classification model; The parameters of the biophysical model are expressed as follows: ； in, Indicates fluorescence intensity, This indicates the time elapsed between the start of the video and the tracer injection. and These represent the injection and evacuation times of the coded tracer, respectively. A unitless number representing the existence and persistence of intensity oscillations. This indicates the overall intensity of the fluorescence profile.
2. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative imaging according to claim 1, characterized in that, The preoperative multimodal imaging includes multi-sequence magnetic resonance imaging, mammography images, and ultrasound images from the patient's preoperative examination.
3. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative imaging according to claim 1, characterized in that, The region of interest in the preoperative multimodal images and the region of interest in the fluorescence imaging sequence are tumor areas where the proportion of surface hemorrhage or edema is lower than a set threshold.
4. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative images according to claim 1, characterized in that, The high-dimensional predefined features include shape features, intensity features, texture features, and wavelet features.
5. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative images according to claim 1, characterized in that, The generative adversarial network variant is built on a deep residual convolutional neural network, comprising one convolutional module and four residual modules connected in sequence; The deep residual network includes sequentially connected convolutional modules, residual modules, and global average pooling layers. The convolutional kernels of the convolutional layers in the modules are all 3×3×3. The Transformer encoder includes two sequentially connected Transformer coding layers, and the Transformer coding layers include an 8-head attention mechanism.
6. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative images according to claim 1, characterized in that, The biophysical model parameters include the inflow behavior and exponential decay of the fluorescence signal.
7. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative imaging according to claim 1, characterized in that, The method for classifying benign and malignant lymph nodes in breast cancer is as follows: The process of assessing sentinel lymph nodes is terminated if the result is negative; if the result is positive, further assessment is needed to determine the benign or malignant nature of non-sentinel lymph nodes.
8. The breast cancer lymph node benign / malignant classification system based on preoperative and intraoperative images according to claim 1, characterized in that, Step S600 includes: Step S601: Perform high-dimensional predefined features of preoperative multimodal images and stitch together global and local features of intraoperative fluorescence images to obtain stitched features; Step S602: Input the spliced features into the fully connected layer of the trained intraoperative sentinel lymph node benign and malignant classification model to obtain the benign and malignant probability of the sentinel lymph node. Step S603: If the probability of benign or malignant sentinel lymph node is lower than a set threshold, the sentinel lymph node is negative; otherwise, the sentinel lymph node is positive, and the process proceeds to step S604. Step S604: Input the spliced features into the fully connected layer of the trained intraoperative non-sentinel lymph node benign / malignant classification model to obtain the benign / malignant probability of non-sentinel lymph nodes.