Intraoperative fluorescence multi-mode image processing method for evaluating colorectal cancer lymph node metastasis state

Abstract

The invention provides an intraoperative fluorescence multi-mode image processing method for evaluating colorectal cancer lymph node metastasis state, which can be applied to the technical field of medical imaging. The method comprises the following steps: acquiring a multi-modal image of an intraoperative lymph node through imaging equipment, and performing data enhancement on the multi-modal image through an image denoising algorithm to obtain an enhanced multi-modal image; performing local feature extraction and global feature extraction on each mode in the enhanced multi-mode image, and splicing the local feature and the global feature of each mode to obtain a spliced feature of each mode; carrying out weighted fusion on splicing features of all modals in the enhanced multi-modal image, and carrying out feature sharing and same-dimension vector operation on a weighted fusion result to obtain multi-modal fusion features; and preprocessing the multi-modal fusion features, and carrying out feature mapping and classification processing on the preprocessed multi-modal fusion features to obtain metastatic state information of the lymph nodes in the operation.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and more specifically to an intraoperative fluorescence multimodal image processing method, model and model training method for evaluating the status of colorectal cancer lymph node metastasis. Background Art

[0002] Colorectal cancer (CRC) is the third most common cancer in the world and the fourth leading cause of cancer-related death. With the improvement of people's living standards and changes in eating habits, the incidence and mortality of colorectal cancer have increased significantly. Multidisciplinary comprehensive treatment collaboration with surgery as the main approach has become an important model and development direction for the clinical treatment of colorectal cancer. Lymph node metastasis is a typical feature of the progression of colorectal cancer. Accurately judging the status of lymph node metastasis and performing precise resection during surgery have a key impact on the patient's prognosis and quality of life.

[0003] However, the existing intraoperative fluorescence multimodal image processing technology solutions cannot process the fluorescence multimodal images of lymph nodes in a timely and efficient manner, resulting in the inability of relevant personnel to timely and accurately know the metastasis status of the lymph nodes. Summary of the invention

[0004] In view of the above problems, the present invention provides an intraoperative fluorescence multimodal image processing method, model and model training method for evaluating the status of colorectal cancer lymph node metastasis.

[0005] According to a first aspect of the present invention, there is provided an intraoperative fluorescence multimodal imaging processing method for evaluating the status of colorectal cancer lymph node metastasis, comprising:

[0006] Acquire multimodal images of intraoperative lymph nodes through imaging equipment, and perform data enhancement on the multimodal images through image denoising algorithms to obtain enhanced multimodal images, wherein the multimodal images include white light images, fluorescent images, and pseudo-color images;

[0007] Perform local feature extraction and global feature extraction on each modality in the enhanced multimodal image, and splice the local features and global features of each modality to obtain the splicing features of each modality;

[0008] The splicing features of all modalities in the enhanced multimodal image are weightedly fused, and the weighted fusion results are subjected to feature sharing and same-dimensional vector operations to obtain multimodal fusion features;

[0009] The multimodal fusion features are preprocessed, and feature mapping and classification are performed on the preprocessed multimodal fusion features to obtain the metastasis status information of the intraoperative lymph nodes.

[0010] According to an embodiment of the present invention, the above-mentioned method of acquiring a multimodal image of a lymph node during surgery by an imaging device and performing data enhancement on the multimodal image by an image denoising algorithm, the enhanced multimodal image includes:

[0011] Exciting the indocyanine green fluorescent dye at the intraoperative lymph node position by an excitation light source in the near-infrared zone 1 of the near-infrared light imaging device, and capturing the fluorescent signal emitted by the indocyanine green fluorescent dye by using the near-infrared light imaging device;

[0012] In the process of capturing fluorescence signals, near-infrared imaging equipment is used to synchronously collect multimodal images of preset resolution;

[0013] The multimodal image is denoised by combining bilateral filtering and Gaussian filtering to achieve image enhancement of the multimodal image, thereby obtaining an enhanced multimodal image.

[0014] According to an embodiment of the present invention, the local feature extraction and global feature extraction are performed on each modality in the enhanced multimodal image, and the local features and global features of each modality are spliced ​​to obtain the spliced ​​features of each modality, including:

[0015] Extracting the edge, texture and / or shape of each modality in the enhanced multimodal image to obtain initial local features of each modality;

[0016] Perform multi-layer convolution and pooling operations on the initial local features of each modality in the enhanced multimodal image to obtain the local features of each modality;

[0017] Perform feature extraction based on the self-attention mechanism for each modality in the enhanced multimodal image to obtain the global features of each modality;

[0018] The local features and global features of each mode are vector-concatenated to obtain the concatenated features of each mode.

[0019] According to an embodiment of the present invention, the weighted fusion of the splicing features of all modalities in the enhanced multimodal image is performed, and the weighted fusion result is subjected to feature sharing and same-dimensional vector operations to obtain the multimodal fusion features including:

[0020] Perform weighted fusion on the vector dimension of the splicing features of all modalities in the enhanced multimodal image to obtain a weighted fusion result;

[0021] Performing initial point convolution processing on the weighted fusion combination, and performing activation processing on the initial point convolution processing result to obtain an activation result;

[0022] Perform secondary point convolution on the activation result, and normalize the secondary point convolution result to obtain the initial multimodal fusion feature;

[0023] The initial multimodal fusion features and the concatenated features of each modality in the multimodal image are subjected to vector operations of the same dimension to obtain the multimodal fusion features.

[0024] According to an embodiment of the present invention, the above-mentioned multimodal fusion feature is obtained by performing a vector operation of the same dimension on the initial multimodal fusion feature and the splicing feature of each modality in the multimodal image, and includes:

[0025] Performing a vector product operation on the initial multimodal fusion feature, the splicing feature of the white light mode and the splicing feature of the pseudo-color mode in the multimodal image, to obtain a first intermediate multimodal fusion feature;

[0026] The initial multimodal fusion feature is operated, and the operation result is vector-multiplied with the splicing feature of the fluorescence modality in the multimodal image to obtain the second intermediate multimodal fusion feature;

[0027] A vector addition operation of the same dimension is performed on the first intermediate multimodal fusion feature and the second intermediate multimodal fusion feature to obtain a multimodal fusion feature.

[0028] According to a second aspect of the present invention, there is provided an intraoperative fluorescence multimodal image processing model for evaluating the status of colorectal cancer lymph node metastasis, comprising:

[0029] An image acquisition and enhancement module is used to acquire multimodal images of intraoperative lymph nodes through an imaging device, and to perform data enhancement on the multimodal images through an image denoising algorithm to obtain enhanced multimodal images, wherein the multimodal images include fluorescent images, white light images, and pseudo-color images;

[0030] A multimodal feature extraction module is used to extract local features and global features of each modality in the enhanced multimodal image, and to splice the local features and global features of each modality to obtain spliced ​​features of each modality;

[0031] The multimodal feature fusion module is used to perform weighted fusion on the spliced ​​features of all modalities in the enhanced multimodal image, and perform feature sharing and same-dimensional vector operations on the weighted fusion results to obtain multimodal fusion features;

[0032] The analysis and prediction module is used to preprocess the multimodal fusion features, and perform feature mapping and classification processing on the preprocessed multimodal fusion features to obtain the metastasis status information of the intraoperative lymph nodes.

[0033] According to an embodiment of the present invention, the multimodal feature extraction module includes a fluorescence multimodal extraction unit, a white light multimodal extraction unit and a pseudo-color multimodal extraction unit;

[0034] Wherein, each extraction unit of the multimodal feature extraction module includes a convolutional neural network branch for extracting local features and a Transformer branch for extracting global features;

[0035] The convolutional neural network branch for extracting local features includes an input layer, a convolutional layer, a plurality of residual neural networks with a preset structure, a first ReLUctant activation function, and an output layer;

[0036] Among them, the Transformer branch used to extract global features includes a linear mapping layer, multiple encoder layers and a multi-layer perceptron.

[0037] According to an embodiment of the present invention, the multimodal feature fusion module includes multiple point convolution layers, a second ReLU activation function and a Sigmoid activation function.

[0038] According to an embodiment of the present invention, the above-mentioned analysis and prediction module includes a multi-layer fully connected network, a third ReLU activation function, a Softmax activation function and an output layer.

[0039] According to a third aspect of the present invention, a training method for an intraoperative fluorescence multimodal image processing model is provided, which is applied to the above intraoperative fluorescence multimodal image processing model, comprising:

[0040] The constructed intraoperative fluorescence multimodal image processing model is used to enhance the multimodal image samples, and the enhanced multimodal image samples are divided into multimodal image training samples and multimodal image verification samples.

[0041] The intraoperative fluorescence multimodal image processing model is used to extract local and global features of multimodal image training samples, and the extracted local features and global features are spliced ​​to obtain sample splicing features.

[0042] The intraoperative fluorescence multimodal image processing model is used to perform weighted fusion on sample splicing features, and the sample weighted fusion results are subjected to feature sharing and same-dimensional vector operations to obtain sample multimodal fusion features.

[0043] The intraoperative fluorescence multimodal image processing model is used to preprocess the sample multimodal fusion features, and the preprocessed sample multimodal fusion features are subjected to feature mapping and classification to obtain the sample transfer status information;

[0044] The sample information and the label values ​​of the multimodal image training samples are processed using a predefined loss function to obtain a loss value, and the parameters of the intraoperative fluorescence multimodal image processing model are updated based on the loss value;

[0045] Iteratively perform sample image enhancement operations, feature extraction operations, multimodal feature fusion operations, and analysis and prediction operations until the preset conditions are met, thereby obtaining a trained intraoperative fluorescence multimodal image processing model;

[0046] The multimodal image verification samples were used to perform K-fold cross-validation on the trained intraoperative fluorescence multimodal image processing model to obtain a cross-validated intraoperative fluorescence multimodal image processing model.

[0047] The intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis provided by the present invention can significantly improve the prediction accuracy of lymph node metastasis based on multimodal fusion features by comprehensively utilizing fluorescence imaging, white light imaging and pseudo-color imaging features. In addition, the intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis provided by the present invention has the characteristics of real-time intraoperative prediction and high precision, and can give each in vivo lymph node a prediction probability during surgery, which is more conducive to auxiliary precision treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0049] Figure 1 is a flow chart of an intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis according to an embodiment of the present invention;

[0050] Figure 2 is a structural schematic diagram of an intraoperative fluorescence multimodal image processing model according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of the structure of a multimodal feature extraction module according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of the structure of a multimodal feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0054] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0055] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0056] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0057] It should be noted that in the technical solution of the present invention, the user information (including but not limited to user personal information, user medical image information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0059] Compared with preoperative imaging technology, intraoperative fluorescence imaging can identify the complete structure of benign and malignant tissues such as tumors, lymph nodes, and nerves in real time during tumor surgery, which helps to achieve better surgical results. The intraoperative fluorescence navigation system can not only perform surgical navigation during the operation, but also perform real-time fluorescence multimodal imaging to visualize tissues. Due to the rapid acquisition and low cost of intraoperative multimodal fluorescence imaging, using it to predict colorectal cancer lymph nodes not only reduces manpower and time costs, but also helps to assist clinical decision-making during surgery and improve prediction accuracy and efficiency.

[0060] The purpose of the present invention is to improve the prediction accuracy of lymph node metastasis by comprehensively utilizing and analyzing fluorescence imaging, white light imaging and pseudo-color imaging.

[0061] Figure 1 4 is a flowchart of an intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis according to an embodiment of the present invention.

[0062] like Figure 1 As shown, the above-mentioned intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis includes operations S110 to S140.

[0063] In operation S110, a multimodal image of the intraoperative lymph node is acquired by an imaging device, and data enhancement is performed on the multimodal image by an image denoising algorithm to obtain an enhanced multimodal image, wherein the multimodal image includes a white light image, a fluorescent image, and a pseudo-color image.

[0064] During the implementation of operation S110 , the user's permission is obtained, and the multimodal image data is processed under the user's permission.

[0065] In operation S120, local feature extraction and global feature extraction are performed on each modality in the enhanced multimodal image, and the local features and global features of each modality are spliced ​​to obtain spliced ​​features of each modality.

[0066] In operation S130, weighted fusion is performed on the spliced ​​features of all modalities in the enhanced multimodal image, and feature sharing and same-dimensional vector operations are performed on the weighted fusion results to obtain multimodal fusion features.

[0067] In operation S140, the multimodal fusion features are preprocessed, and feature mapping and classification processing are performed on the preprocessed multimodal fusion features to obtain intraoperative lymph node metastasis status information.

[0068] The purpose of implementing operation S140 is to analyze and predict the metastasis status information of the lymph nodes.

[0069] The above operations S110~S140 are all information processing operations performed by computers and other devices. It should be noted that the direct purpose of the above operations S110~S140 is not to obtain diagnostic results or health conditions, but to obtain intermediate information from lymph nodes. The above operations S110~S140 are used to process the intermediate information.

[0070] The intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis provided by the present invention can significantly improve the prediction accuracy of lymph node metastasis based on multimodal fusion features by comprehensively utilizing fluorescence imaging, white light imaging and pseudo-color imaging features. In addition, the intraoperative fluorescence multimodal image processing method for evaluating the status of colorectal cancer lymph node metastasis provided by the present invention has the characteristics of real-time intraoperative prediction and high precision, and can give each in vivo lymph node a prediction probability during surgery, which is more conducive to auxiliary precision treatment.

[0071] According to an embodiment of the present invention, the above-mentioned multimodal images of intraoperative lymph nodes are obtained by imaging equipment, and the multimodal images are data enhanced by an image denoising algorithm to obtain the enhanced multimodal images, including: exciting the indocyanine green fluorescent dye at the position of the intraoperative lymph nodes by an excitation light source in a near-infrared zone 1 of a near-infrared imaging device, and capturing the fluorescence signal emitted by the indocyanine green fluorescent dye by the near-infrared imaging device; in the process of capturing the fluorescence signal, synchronously collecting multimodal images of a preset resolution by the near-infrared imaging device; and denoising the multimodal images by a combination of bilateral filtering and Gaussian filtering to achieve image enhancement of the multimodal images, thereby obtaining the enhanced multimodal images.

[0072] The image acquisition and pre-processing process involved in the present invention will be further described in detail below through specific embodiments.

[0073] First, a highly sensitive near-infrared imaging device is used, using an excitation light source in the near-infrared region to excite the indocyanine green (ICG) fluorescent dye and capture the fluorescent signal it emits. The three-modality images were acquired at the same time using 1080p white light, fluorescence, and pseudo-color images. A combination of bilateral filtering and Gaussian filtering was used to remove noise from the acquired three-modality images to ensure image quality.

[0074] According to an embodiment of the present invention, the above-mentioned local feature extraction and global feature extraction are performed on each modality in the enhanced multimodal image, and the local features and global features of each modality are spliced ​​to obtain the spliced ​​features of each modality, including: extracting the edge, texture and / or shape of each modality in the enhanced multimodal image to obtain the initial local features of each modality; performing multi-layer convolution and pooling operations on the initial local features of each modality in the enhanced multimodal image to obtain the local features of each modality; performing feature extraction based on the self-attention mechanism on each modality in the enhanced multimodal image to obtain the global features of each modality; performing vector splicing operations on the local features and global features of each modality to obtain the spliced ​​features of each modality.

[0075] The local and global feature extraction process of the present invention is further described in detail below through specific embodiments.

[0076] The feature extraction process of the present invention is divided into a local feature extraction branch and a global feature extraction branch.

[0077] In the process of local feature extraction, CNN is used to perform deep feature learning on fluorescence, white light and pseudo-color images. CNN can effectively extract local features such as edges, textures and shapes, and is suitable for processing the high-dimensional characteristics of image data. Through multi-layer convolution and pooling operations, the model can gradually learn from low-level features to more advanced abstract features, enhancing the understanding of tissue structure. Under the technical enlightenment of the present invention, those skilled in the art can extract other local features except edges, textures and / or shapes according to their own needs.

[0078] In the process of global feature extraction, the Transformer architecture is combined to process and model long-distance dependencies in the image. Transformer can enhance the ability to capture global features of the image through the self-attention mechanism, helping the model to recognize more complex organizational patterns and structures. This method is particularly suitable for processing multimodal data, so that features between different modalities can be effectively fused and associated. The extracted features are spliced ​​to obtain trimodal features. (fluorescence mode), (white light mode) and (False color mode).

[0079] According to an embodiment of the present invention, the above-mentioned weighted fusion of the stitching features of all modalities in the enhanced multimodal image, and the feature sharing and same-dimensional vector operation of the weighted fusion result to obtain the multimodal fusion feature includes: weighted fusion of the stitching features of all modalities in the enhanced multimodal image in the vector dimension to obtain a weighted fusion result; performing a primary point convolution processing on the weighted fusion combination, and activating the primary point convolution processing result to obtain an activation result; performing a secondary point convolution processing on the activation result, and normalizing the secondary point convolution processing result to obtain an initial multimodal fusion feature; performing a same-dimensional vector operation on the initial multimodal fusion feature and the stitching features of each modality in the multimodal image to obtain a multimodal fusion feature.

[0080] According to an embodiment of the present invention, the above-mentioned performing a vector operation of the same dimension on the initial multimodal fusion feature and the splicing feature of each modality in the multimodal image to obtain the multimodal fusion feature includes: performing a vector product operation on the initial multimodal fusion feature and the splicing feature of the white light modality and the splicing feature of the pseudo-color modality in the multimodal image to obtain a first intermediate multimodal fusion feature; performing an operation on the initial multimodal fusion feature, and performing a vector product on the operation result with the splicing feature of the fluorescence modality in the multimodal image to obtain a second intermediate multimodal fusion feature; performing a vector addition operation of the same dimension on the first intermediate multimodal fusion feature and the second intermediate multimodal fusion feature to obtain the multimodal fusion feature.

[0081] The following is a further detailed description of the process of acquiring the multimodal fusion features of the present invention through specific embodiments.

[0082] In the process of obtaining multimodal fusion features: first, a weighted fusion algorithm is performed. , and After dimensional splicing, fusion and re-weighting are performed to ensure that the key information in different modes is retained to the greatest extent. , and Perform feature splicing to obtain features , ;Will Put it into the convolution operation of the combined point to get , as the weights of different modal features. Secondly, feature fusion is performed to concatenate and fuse the weighted three modal features to generate the fused feature The feature vector not only contains the key information from each mode, but also eliminates the noise and redundant information between modes to a certain extent.

[0083] Figure 24 is a schematic diagram of the structure of an intraoperative fluorescence multimodal image processing model according to an embodiment of the present invention.

[0084] like Figure 2 As shown, an intraoperative fluorescence multimodal image processing model 200 for evaluating the status of colorectal cancer lymph node metastasis includes an image acquisition and enhancement module 210, a multimodal feature extraction module 220, a multimodal feature fusion module 230 and an analysis and prediction module 240.

[0085] The image acquisition and enhancement module 210 is used to acquire multimodal images of intraoperative lymph nodes through an imaging device, and perform data enhancement on the multimodal images through an image denoising algorithm to obtain enhanced multimodal images, wherein the multimodal images include fluorescent images, white light images, and pseudo-color images.

[0086] The image acquisition and enhancement module 210 includes an image acquisition unit for acquiring high-resolution intraoperative fluorescence, white light and pseudo-color images using a near-infrared fluorescence device. The injected fluorescent dye is excited by the near-infrared fluorescence imaging device, and white light, fluorescence and pseudo-color images, i.e., multi-modal image data, are acquired synchronously; and an image preprocessing unit is used to enhance the image quality by using an image denoising preprocessing algorithm to ensure the clarity and consistency of the image.

[0087] The multimodal feature extraction module 220 is used to extract local features and global features of each modality in the enhanced multimodal image, and to splice the local features and global features of each modality to obtain spliced ​​features of each modality.

[0088] The multimodal feature fusion module 230 is used to perform weighted fusion on the spliced ​​features of all modalities in the enhanced multimodal image, and perform feature sharing and same-dimensional vector operations on the weighted fusion results to obtain multimodal fusion features.

[0089] The analysis and prediction module 240 is used to preprocess the multimodal fusion features, and perform feature mapping and classification processing on the preprocessed multimodal fusion features to obtain intraoperative lymph node metastasis status information.

[0090] According to an embodiment of the present invention, the above-mentioned multimodal feature extraction module includes a fluorescence multimodal extraction unit, a white light multimodal extraction unit and a pseudo-color multimodal extraction unit; wherein each extraction unit of the multimodal feature extraction module includes a convolutional neural network branch for extracting local features and a Transformer branch for extracting global features; wherein the convolutional neural network branch for extracting local features includes an input layer, a convolutional layer, a plurality of residual neural networks with a preset structure, a first ReLUctant activation function and an output layer; wherein the Transformer branch for extracting global features includes a linear mapping layer, a plurality of encoder layers and a multilayer perceptron.

[0091] The following is a specific embodiment and combined with the attached Figure 3 The multimodal feature extraction module involved in the present invention is further described in detail.

[0092] Figure 3 is a schematic diagram of the structure of a multimodal feature extraction module according to an embodiment of the present invention.

[0093] like Figure 3 As shown, the present invention establishes a dual-branch feature extraction module for each modality image, a local feature extraction branch network, and adopts a CNN structure. The specific architecture is that the input size of the input layer is (RGB image). Using an 18-layer ResNet structure, the convolution kernel size is , the stride is 1, and the activation function is ReLU. Every two layers of convolution are followed by a The maximum pooling layer reduces the feature dimension and finally outputs a The global feature extraction branch uses the Transformer architecture and sets the input layer size to be the same as the local feature. The encoder layer uses 6 Transformer encoders, each of which contains 8 self-attention heads, with a hidden layer dimension of 512 and a feedforward layer dimension of 2048. The output part is the same feature map as the local feature extraction branch. Then, the global and local features of the three modalities are added at the corresponding positions to obtain the corresponding modal features. , and , and input into the multimodal image feature fusion module for feature fusion.

[0094] According to an embodiment of the present invention, the multimodal feature fusion module includes multiple point convolution layers, a second ReLU activation function and a Sigmoid activation function.

[0095] The following is a specific embodiment and combined with the attached Figure 4 The multimodal feature fusion module 230 provided by the present invention is further described in detail.

[0096] Figure 4 is a schematic diagram of the structure of a multimodal feature fusion module according to an embodiment of the present invention.

[0097] like Figure 4 As shown in the figure, the multimodal feature fusion module first uses the weighted fusion algorithm to , and The three modal features are concatenated in dimension ,Right now By using point convolution to compress and restore feature dimensions to achieve feature sharing, that is, , where BN is the normalization layer and PWC is the point convolution layer. is the ReLU activation function. According to the modal characteristics, and and product, and Multiply and add the products to get the fused features .

[0098] According to an embodiment of the present invention, the above-mentioned analysis and prediction module includes a multi-layer fully connected network, a third ReLU activation function, a Softmax activation function and an output layer.

[0099] The structure of the analysis and prediction module of the present invention is further described in detail below through specific embodiments.

[0100] The analysis and prediction module is mainly composed of a multi-layer fully connected network (MLP), with 2 hidden layers, each containing 256 neurons, and the activation function is ReLU. The output layer dimension is 2 (transferred / not transferred), and the softmax activation function is used to calculate the classification probability.

[0101] According to an embodiment of the present invention, a training method for an intraoperative fluorescence multimodal image processing model is provided, which is applied to the above-mentioned intraoperative fluorescence multimodal image processing model, including: using the constructed intraoperative fluorescence multimodal image processing model to perform image enhancement on multimodal image samples, and dividing the enhanced multimodal image samples into multimodal image training samples and multimodal image verification samples; using the intraoperative fluorescence multimodal image processing model to extract local and global features of the multimodal image training samples, and performing feature splicing on the extracted local features and global features to obtain sample splicing features; using the intraoperative fluorescence multimodal image processing model to perform weighted fusion on the sample splicing features, and performing feature sharing and same-dimensional vector operations on the sample weighted fusion results to obtain sample multimodal fusion features; using the intraoperative fluorescence multimodal image processing model to perform weighted fusion on the sample splicing features, and performing feature sharing and same-dimensional vector operations on the sample weighted fusion results to obtain sample multimodal fusion features. The fluorescence multimodal image processing model preprocesses the multimodal fusion features of the samples, and performs feature mapping and classification on the preprocessed multimodal fusion features of the samples to obtain sample transfer status information; a predefined loss function is used to process the sample information and the label values ​​of the multimodal image training samples to obtain the loss value, and the parameters of the intraoperative fluorescence multimodal image processing model are updated based on the loss value; image enhancement operations, feature extraction operations, multimodal feature fusion operations, and analysis and prediction operations are iteratively performed on the samples until the preset conditions are met to obtain a trained intraoperative fluorescence multimodal image processing model; multimodal image verification samples are used to perform K-fold cross-validation on the trained intraoperative fluorescence multimodal image processing model to obtain a cross-validated intraoperative fluorescence multimodal image processing model.

[0102] The training method of the intraoperative fluorescence multimodal image processing model provided by the present invention is further described in detail below through specific embodiments.

[0103] During the training process, the loss function is combined with the class imbalance problem Improve model performance, including ,in and is the predicted probability. ,in is the sample imbalance weight, is the regulating factor, The probability of prediction. In the analysis and prediction decision process, after receiving the fused image feature vector, it is first preprocessed to ensure the quality and consistency of the input data. This includes standardization and normalization. The output layer is formed by establishing multiple layers of fully connected layers and linear layers. According to the analysis of the fused features, the model will generate corresponding evaluation results for each lymph node. K-fold cross validation is performed based on the trained model to verify the model performance. After verification, the model can provide the probability of lymph node metastasis status evaluation during surgery.

[0104] In summary, through detailed implementation process explanation module design and parameter setting, the above method provided by the present invention can effectively improve the assessment accuracy of lymph node metastasis status and provide strong support for clinical practice.

[0105] Compared with the prior art, the above method provided by the present invention has the characteristics of strong real-time performance and high accuracy. By effectively integrating multiple imaging information, the model can provide clinicians with reliable metastasis prediction and assist them in making more scientific decisions, thereby improving the effect of colorectal cancer surgery and the prognosis of patients.

[0106] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0108] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for intraoperative fluorescence multimodal imaging for evaluating the status of colorectal cancer lymph node metastasis, characterized in that: The method comprises: Acquire a multimodal image of the intraoperative lymph node by an imaging device, and perform data enhancement on the multimodal image by an image denoising algorithm to obtain an enhanced multimodal image, wherein the multimodal image includes a white light image, a fluorescent image, and a pseudo-color image; Performing local feature extraction and global feature extraction on each modality in the enhanced multimodal image, and splicing the local features and global features of each modality to obtain splicing features of each modality; Performing weighted fusion on the splicing features of all modalities in the enhanced multimodal image, and performing feature sharing and same-dimensional vector operations on the weighted fusion results to obtain multimodal fusion features; The multimodal fusion features are preprocessed, and feature mapping and classification processing are performed on the preprocessed multimodal fusion features to obtain the metastasis status information of the intraoperative lymph nodes.

2. The method according to claim 1, characterized in that The multimodal images of the intraoperative lymph nodes are acquired by an imaging device, and the multimodal images are data enhanced by an image denoising algorithm. The enhanced multimodal images include: Exciting the indocyanine green fluorescent dye at the intraoperative lymph node position by an excitation light source in a near-infrared zone 1 of the near-infrared light imaging device, and capturing the fluorescent signal emitted by the indocyanine green fluorescent dye by using the near-infrared light imaging device; In the process of capturing the fluorescence signal, using the near-infrared light imaging device to synchronously collect multimodal images of a preset resolution; The multimodal image is denoised by combining bilateral filtering and Gaussian filtering to achieve image enhancement of the multimodal image, thereby obtaining the enhanced multimodal image.

3. The method according to claim 1, characterized in that Performing local feature extraction and global feature extraction on each modality in the enhanced multimodal image, and splicing the local features and global features of each modality to obtain the splicing features of each modality includes: Extracting the edge, texture and / or shape of each modality in the enhanced multimodal image to obtain initial local features of each modality; Performing multi-layer convolution and pooling operations on the initial local features of each modality in the enhanced multi-modal image to obtain the local features of each modality; Performing feature extraction based on a self-attention mechanism on each modality in the enhanced multimodal image to obtain a global feature of each modality; Perform a vector concatenation operation on the local features and the global features of each mode to obtain the concatenated features of each mode.

4. The method according to claim 1, characterized in that: The splicing features of all modes in the enhanced multimodal image are weightedly fused, and the weighted fusion results are subjected to feature sharing and same-dimensional vector operations to obtain multimodal fusion features including: Performing weighted fusion on the vector dimension of the splicing features of all modalities in the enhanced multimodal image to obtain the weighted fusion result; Performing a primary point convolution process on the weighted fusion combination, and performing an activation process on the result of the primary point convolution process to obtain an activation result; Performing secondary point convolution processing on the activation result, and normalizing the secondary point convolution processing result to obtain an initial multimodal fusion feature; The initial multimodal fusion feature and the splicing feature of each modality in the multimodal image are subjected to vector operation of the same dimension to obtain the multimodal fusion feature.

5. The method according to claim 4, characterized in that The initial multimodal fusion feature and the splicing feature of each modality in the multimodal image are subjected to vector operation of the same dimension to obtain the multimodal fusion feature including: Performing a vector product operation on the initial multimodal fusion feature and the splicing feature of the white light modality and the splicing feature of the pseudo-color modality in the multimodal image to obtain a first intermediate multimodal fusion feature; The initial multimodal fusion feature is operated, and a vector product is performed on the operation result and the splicing feature of the fluorescence modality in the multimodal image to obtain a second intermediate multimodal fusion feature; Performing a vector addition operation of the same dimension on the first intermediate multimodal fusion feature and the second intermediate multimodal fusion feature to obtain the multimodal fusion feature.

6. An intraoperative fluorescence multimodal image processing model for evaluating the status of colorectal cancer lymph node metastasis, characterized in that: The model includes: An image acquisition and enhancement module, used to acquire multimodal images of intraoperative lymph nodes through an imaging device, and perform data enhancement on the multimodal images through an image denoising algorithm to obtain enhanced multimodal images, wherein the multimodal images include fluorescent images, white light images, and pseudo-color images; A multimodal feature extraction module, used for performing local feature extraction and global feature extraction on each modality in the enhanced multimodal image, and splicing the local features and global features of each modality to obtain splicing features of each modality; A multimodal feature fusion module is used to perform weighted fusion on the spliced ​​features of all modalities in the enhanced multimodal image, and perform feature sharing and same-dimensional vector operations on the weighted fusion results to obtain multimodal fusion features; The analysis and prediction module is used to preprocess the multimodal fusion features, and perform feature mapping and classification processing on the preprocessed multimodal fusion features to obtain the metastasis status information of the intraoperative lymph nodes.

7. The model according to claim 6, characterized in that The multimodal feature extraction module includes a fluorescence multimodal extraction unit, a white light multimodal extraction unit and a pseudo-color multimodal extraction unit; Wherein, each extraction unit of the multimodal feature extraction module includes a convolutional neural network branch for extracting local features and a Transformer branch for extracting global features; The convolutional neural network branch for extracting local features includes an input layer, a convolutional layer, a plurality of residual neural networks with a preset structure, a first ReLUctant activation function, and an output layer; The Transformer branch for extracting global features includes a linear mapping layer, multiple encoder layers and a multi-layer perceptron.

8. The model according to claim 6, characterized in that The multimodal feature fusion module includes multiple point convolution layers, a second ReLU activation function and a Sigmoid activation function.

9. The model according to claim 6, characterized in that The analysis and prediction module includes a multi-layer fully connected network, a third ReLU activation function, a Softmax activation function and an output layer.

10. A method for training an intraoperative fluorescence multimodal image processing model, applied to training the model described in any one of claims 6-9, characterized in that: The method comprises: The constructed intraoperative fluorescence multimodal image processing model is used to enhance the multimodal image samples, and the enhanced multimodal image samples are divided into multimodal image training samples and multimodal image verification samples. Using the intraoperative fluorescence multimodal image processing model to extract local and global features of the multimodal image training samples, and performing feature splicing on the extracted local features and global features to obtain sample splicing features; The sample splicing features are weightedly fused using the intraoperative fluorescence multimodal image processing model, and the sample weighted fusion results are subjected to feature sharing and same-dimensional vector operations to obtain sample multimodal fusion features; Preprocessing the sample multimodal fusion features by using the intraoperative fluorescence multimodal image processing model, and performing feature mapping and classification processing on the preprocessed sample multimodal fusion features to obtain sample transfer status information; Using a predefined loss function to process the sample information and the label value of the multimodal image training sample to obtain a loss value, and updating the parameters of the intraoperative fluorescence multimodal image processing model based on the loss value; Iteratively perform sample image enhancement operations, feature extraction operations, multimodal feature fusion operations, and analysis and prediction operations until the preset conditions are met, thereby obtaining a trained intraoperative fluorescence multimodal image processing model; The multimodal image verification samples are used to perform K-fold cross-validation on the trained intraoperative fluorescence multimodal image processing model to obtain a cross-validated intraoperative fluorescence multimodal image processing model.

Images