Automatic segmentation and recognition method and system for cancer image fused with multi-modal images

Abstract

This invention relates to the field of medical image processing technology, specifically to an automatic segmentation and recognition method and system for cancer images based on the fusion of multimodal images. The method includes: acquiring a multimodal medical image set containing computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images; generating spatially aligned data blocks through three-dimensional spatial registration and voxel alignment; extracting fusion features using an improved encoder-decoder network optimized based on the characteristics and spatial dependencies of different modal channels; obtaining enhanced semantic feature maps through multi-level context-aware processing; identifying suspected cancer regions through a region candidate generation network; and simultaneously outputting pixel-level segmentation masks and cancer type probabilities through a segmentation network, ultimately generating a three-dimensional segmentation and recognition result. This method can accurately capture multimodal-specific features, clearly distinguish the boundaries between lesions and normal tissue, simplify the processing flow, reduce errors, and achieve accurate and efficient segmentation and recognition of cancer lesions.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for automatic segmentation and recognition of cancer images by fusing multimodal images. Background Technology

[0002] Early and accurate segmentation and identification of cancer is an important prerequisite for clinical diagnosis and treatment planning. Currently, multimodal medical imaging combined with automatic segmentation and identification technology is often used in clinical practice to assist in diagnosis and treatment. Commonly used multimodal medical images include computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images. Conventional techniques often use standard encoder-decoder networks to extract and fuse features from multimodal images, and then obtain the segmentation and identification results of cancer lesions through simple region screening and segmentation processing.

[0003] Conventional standard encoder-decoder networks do not consider the differences in channel characteristics and spatial dependencies of images from different modalities. They use a uniform network structure for feature extraction and fusion, resulting in the inability to accurately capture the unique features of images from different modalities, and insufficient targeting and effectiveness of cross-modal feature fusion. Furthermore, conventional techniques do not perform dedicated context-aware processing on the fused feature maps, leading to insufficient semantic feature expression and difficulty in accurately distinguishing the boundaries between lesion regions and normal tissue. Additionally, candidate region generation and segmentation, as well as cancer type determination, are often processed independently in separate steps, easily resulting in regional localization errors, insufficient segmentation accuracy, and a disconnect between type determination and segmentation results, ultimately failing to achieve accurate and efficient segmentation and identification of cancer lesions. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and propose an automatic segmentation and recognition method and system for cancer images that integrates multimodal images.

[0005] To achieve the above objectives, the present invention employs the following technical solution: an automatic segmentation and recognition method for cancer images fused with multimodal imaging, comprising: Obtain a multimodal medical image set corresponding to the target case, wherein the multimodal medical image set includes at least computed tomography images, magnetic resonance imaging images, and positron emission tomography images; Perform three-dimensional spatial registration and voxel alignment on the multimodal medical image set to generate spatially aligned multimodal image data blocks; An improved encoder-decoder network is used to extract and fuse features from the spatially aligned multimodal image data blocks to generate a fused multimodal depth feature map. The improved encoder-decoder network optimizes the standard encoder-decoder structure based on the channel characteristics and spatial dependencies of different modal images. Perform multi-level context-aware processing on the fused multimodal deep feature map to generate an enhanced semantic feature map; The enhanced semantic feature map is input into the region candidate generation network to generate a set of candidate 3D bounding boxes containing suspected cancer regions. The segmentation network is invoked to process the image regions within the candidate 3D bounding box set, and the pixel-level segmentation mask and cancer type probability corresponding to each candidate region are output. Based on the pixel-level segmentation mask and the cancer type probability, the final three-dimensional cancer lesion segmentation and recognition result is generated.

[0006] As a further aspect of the present invention, the step of performing three-dimensional spatial registration and voxel alignment on the multimodal medical image set to generate spatially aligned multimodal image data blocks includes: The computed tomography images in the multimodal medical image set are selected as the reference reference images for spatial registration; Rigid registration calculations based on mutual information were performed on the magnetic resonance imaging images and positron emission tomography images and the reference reference image respectively to obtain their respective spatial transformation matrices; Using the spatial transformation matrix, three-dimensional spatial resampling and interpolation are performed on magnetic resonance imaging images and positron emission tomography images, so that each voxel is physically aligned with the corresponding voxel of the reference image. Spatially aligned computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images are stacked along the channel dimension to form the spatially aligned multimodal image data block.

[0007] As a further aspect of the present invention, the step of using an improved encoder-decoder network to extract and fuse features from the spatially aligned multimodal image data blocks to generate a fused multimodal depth feature map includes: The improved encoder-decoder network includes a multi-branch encoder and a single-branch decoder, wherein the multi-branch encoder is configured with an independent encoder branch for each modal image; The spatially aligned multimodal image data block is split into independent image channels according to image modality, and each channel is input into the corresponding encoder branch for depth feature extraction to obtain the depth feature sequence of each modality. A cross-modal feature interaction module is set at a specific level of a multi-branch encoder. The cross-modal feature interaction module receives feature maps of the same level from different encoder branches, calculates the feature correlation between modes through a cross-modal attention mechanism, performs feature weighted fusion, and generates an interactive fused feature map. The final output depth feature sequences of each encoder branch and the fused feature map after interaction are input into the single-branch decoder. The decoder gradually recovers the spatial resolution through upsampling and skip connections, and outputs the fused multimodal depth feature map containing rich multimodal information in the final layer.

[0008] As a further aspect of the present invention, the step of performing multi-level context-aware processing on the fused multimodal deep feature map to generate an enhanced semantic feature map includes: A spatial pyramid pooling structure is constructed, and pooling windows of different scales are applied to the fused multimodal deep feature map to extract multi-scale contextual features; By utilizing dilated convolution, parallel convolutional paths with different receptive fields are constructed on the fused multimodal deep feature map to capture multi-range contextual information; The extracted multi-scale context features and the captured multi-range context information are then concatenated with the original fused multimodal deep feature map. The concatenated feature map is processed by channel attention and spatial attention mechanisms to recalibrate the importance of different channels and spatial positions in the feature map, thereby generating the enhanced semantic feature map.

[0009] As a further aspect of the present invention, the step of inputting the enhanced semantic feature map into a region candidate generation network to generate a set of candidate 3D bounding boxes containing suspected cancer regions includes: In the region candidate generation network, multiple anchor boxes of different scales and aspect ratios are preset, and the anchor boxes are densely slid in the three-dimensional space corresponding to the enhanced semantic feature map. For each anchor box location, features of the corresponding region are extracted from the enhanced semantic feature map and input into two parallel fully connected layers. One fully connected layer is used to predict the probability that the anchor box contains a cancer target, and the other fully connected layer is used to predict the positional correction of the anchor box relative to the true cancer region bounding box. All anchor boxes are sorted according to the predicted probability of containing cancer targets, and the top few anchor boxes with the highest probability are selected as initial candidate boxes. The position and size of the initial candidate boxes are adjusted using the predicted bounding box position correction amount to obtain a set of candidate 3D bounding boxes with more accurate positions.

[0010] As a further aspect of the present invention, the step of calling the segmentation network to process the image regions within the candidate 3D bounding box set and outputting a pixel-level segmentation mask and cancer type probability corresponding to each candidate region includes: From the original spatially aligned multimodal image data blocks, the corresponding three-dimensional image region blocks are cropped according to the coordinates of the candidate three-dimensional bounding boxes; Each 3D image region block is input into the segmentation network, which adopts an encoder-decoder structure. The encoder downsamples the input region block to extract high-level semantic features, while the decoder gradually restores the resolution of the original region block through upsampling and skip connections. In the final output layer of the decoder, a segmentation head outputs the probability that each voxel belongs to a cancer lesion, forming a probability heatmap, and a classification head outputs the probability distribution of the entire region block belonging to a specific cancer type. Thresholding is applied to the probability heatmap, and voxels with a probability higher than a preset threshold are marked as cancer lesions. The pixel-level segmentation mask is generated, and the type with the highest probability in the probability distribution of the specific cancer type is recorded as the predicted cancer type of the candidate region.

[0011] As a further aspect of the present invention, the training process of the segmentation network includes: Obtain a multimodal image training dataset with pixel-level segmentation annotations and cancer type annotations; Three-dimensional image regions containing cancer lesions are extracted from the multimodal image training dataset as training samples; The training samples are input into the segmentation network to obtain the predicted segmentation probability heatmap and cancer type probability distribution; Calculate the segmentation loss between the predicted segmentation probability heatmap and the actual pixel-level segmentation annotation; Calculate the classification loss between the predicted cancer type probability distribution and the actual cancer type label; The parameters of the segmentation network are updated using the backpropagation algorithm, combining the segmentation loss and the classification loss, until the network converges.

[0012] As a further aspect of the present invention, the step of generating the final three-dimensional cancer lesion segmentation and recognition result based on the pixel-level segmentation mask and the cancer type probability includes: Receive pixel-level segmentation masks and cancer type probabilities for all candidate regions from the segmentation network; Non-maximum suppression processing is applied to the pixel-level segmentation mask to remove redundant segmentation masks that are highly overlapping in three-dimensional space and have a low probability of cancer type. The pixel-level segmentation mask retained after non-maximum suppression is mapped back to the global three-dimensional coordinate system of the original multimodal medical image set; In the global three-dimensional coordinate system, adjacent pixel-level segmentation masks that belong to the same predicted cancer type are merged to form a complete three-dimensional cancer lesion segmentation body. Each 3D cancer lesion segment is labeled with its corresponding predicted cancer type and probability. All lesion segments and their labeled information are combined to form the final 3D cancer lesion segmentation and recognition result.

[0013] As a further aspect of the present invention, the improved encoder-decoder network optimizes the standard encoder-decoder structure based on the channel characteristics and spatial dependencies of different modal images, including: After each downsampling stage of the encoder, a channel feature weighting module is introduced. This module learns the weights of different image modal channels and adaptively weights the feature channels of each modality before feature fusion to highlight the feature contributions of important modalities. Before each upsampling stage of the decoder, a spatial dependency enhancement module is introduced. This module models long-range spatial dependencies by calculating the self-attention of the feature map in the spatial dimension and injects the enhanced spatial information into the upsampled features. On the skip connection path corresponding to the encoder and decoder, a cross-modal feature calibration connection is introduced. This connection dynamically selects and fuses the most relevant multimodal features from the corresponding layer of the encoder based on the context information requirements of the current layer of the decoder.

[0014] As a further aspect of the present invention, the present invention also includes an automatic cancer image segmentation and recognition system that integrates multimodal images. The system includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to run the programs, instructions or code in the memory to implement the automatic cancer image segmentation and recognition method that integrates multimodal images as described above.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Based on the channel characteristics and spatial dependencies of computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) images, the standard encoder-decoder structure is optimized. This optimized network extracts and fuses features from spatially aligned multimodal image data blocks, accurately capturing the unique features of different modalities, reducing intermodal feature interference, and achieving deep fusion of cross-modal features. This results in a more representative fused multimodal deep feature map. Compared to the feature extraction and fusion methods of conventional standard encoder-decoder networks, this approach effectively avoids feature loss or redundancy caused by differences in modal characteristics, allowing feature representation to better reflect the actual feature distribution of cancer lesions.

[0016] By performing multi-level context-aware processing based on the fusion of multimodal deep feature maps, deep semantic information in the feature maps can be mined, strengthening the correlation between features in different regions. The generated enhanced semantic feature maps can more clearly present the boundary differences between lesion areas and normal tissues. A set of candidate 3D bounding boxes for suspected cancer areas is generated through a region candidate generation network, and then a segmentation network is called to process the candidate regions, simultaneously outputting pixel-level segmentation masks and cancer type probabilities. There is no need to perform candidate region generation, segmentation, and type judgment steps separately, which simplifies the processing flow and improves the accuracy of suspected region localization. The segmentation results are more in line with the actual morphology of the lesion, and the type judgment is more in line with the true attributes of the lesion. Compared with the conventional step-by-step processing method of first locating, then segmenting, and then judging the type, it can effectively reduce the accumulation of errors in the processing steps. Attached Figure Description

[0017] Figure 1 This is a flowchart of the automatic segmentation and recognition method for cancer images based on the fusion of multimodal images described in this invention; Figure 2 A flowchart for generating spatially aligned multimodal image data blocks for 3D spatial registration and voxel alignment; Figure 3 A flowchart for generating enhanced semantic feature maps for multi-level context-aware processing. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1The overall implementation scheme of the automatic segmentation and recognition method for cancer images fused with multimodal images described in this invention strictly follows the process of the aforementioned technical solution. A multimodal medical image set, including computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) images, corresponding to the target case, is acquired. Three-dimensional spatial registration and voxel alignment are performed on this multimodal medical image set to generate a spatially perfectly aligned multimodal image data block. An improved encoder-decoder network is used to extract and fuse deep features from this spatially aligned multimodal image data block. This network is optimized based on the channel characteristics and spatial dependencies of different modalities of the images, outputting a fused multimodal depth feature map. Multi-level context-aware processing is performed on this fused multimodal depth feature map to capture multi-scale information and enhance semantic expression, generating an enhanced semantic feature map. This enhanced semantic feature map is input into a region candidate generation network, which is responsible for generating a series of three-dimensional bounding boxes containing suspected cancer regions, forming a candidate three-dimensional bounding box set. Another segmentation network is then invoked to process each image region within the candidate 3D bounding box set. The network outputs a pixel-level segmentation mask for each candidate region and the probability that the region belongs to a specific cancer type. Based on the pixel-level segmentation masks and cancer type probabilities returned by all candidate regions, fusion and post-processing are performed to generate the final 3D cancer lesion segmentation and recognition result containing information on 3D spatial location, morphology, and predicted type.

[0021] In one embodiment of the present invention, in the three-dimensional spatial registration and voxel alignment steps, see [reference needed]. Figure 2 From the acquired multimodal medical image set, computed tomography (CT) images are selected as reference images for spatial registration. Rigid registration calculations based on mutual information are performed on the selected reference images for both magnetic resonance imaging (MRI) and positron emission tomography (PET) images. This calculation process yields a spatial transformation matrix for each image to be registered. Using the calculated spatial transformation matrix, three-dimensional spatial resampling and interpolation operations are performed on the MRI and PET images. This operation ensures that the position of each voxel in the resampled image in the physical space coordinate system is precisely aligned with the corresponding voxel position in the reference image. The CT, MRI, and PET images, after the spatial alignment steps described above, are stacked along the channel dimension to form an integrated multi-channel three-dimensional data block, i.e., a spatially aligned multimodal image data block.

[0022] In practice, after obtaining the multimodal medical image set corresponding to the target case, which includes at least computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, spatial alignment needs to be performed for subsequent fusion analysis. In some embodiments, the CT images in the multimodal medical image set are selected as the reference image for spatial registration. CT images are often used as registration references due to their good spatial resolution and geometric fidelity. In practice, rigid registration calculations based on mutual information are performed on the MRI images and PET images and the reference image, respectively. This calculation aims to find a spatial transformation that maximizes the mutual information between the image to be registered and the reference image. Mutual information is understood to be an indicator of the statistical dependence between two images, and its maximization process is solved through optimization algorithms. For each MRI image and PET image to be registered, the above calculation outputs a corresponding spatial transformation matrix, which defines the rotation, translation, and scaling parameters. Using the solved spatial transformation matrix, three-dimensional spatial resampling and interpolation operations are performed on the original magnetic resonance imaging (MRI) images and positron emission tomography (PET) images. Optionally, the resampling process employs a trilinear interpolation method to achieve a balance between computational efficiency and accuracy. Through resampling and interpolation, the position of each voxel in the MRI and PET images in the physical spatial coordinate system is precisely aligned with the corresponding voxel position in the computed tomography (CT) image, which serves as a reference image. In some embodiments, the aligned physical spatial coordinate system can be a world coordinate system established based on patient anatomical landmarks or the imaging equipment coordinate system. It is understood that voxel alignment eliminates geometric inconsistencies caused by differences in patient positioning or imaging equipment. In a specific implementation, the spatially aligned CT, MRI, and PET images are stacked along the channel dimension. The stacking operation generates a multi-channel three-dimensional data array, namely a spatially aligned multimodal image data block, in which different channel values ​​under the same spatial location index correspond to image information of different modalities.

[0023] In one embodiment of the present invention, when using an improved encoder-decoder network for feature extraction and fusion, the network structure includes a multi-branch encoder and a single-branch decoder. The multi-branch encoder configures an independent encoder branch for each of the input computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images. Spatially aligned multimodal image data blocks are split into independent image channels according to image modality and input into their respective independent encoder branches for depth feature extraction. Each branch outputs a depth feature sequence corresponding to its modality. At a specific level of the multi-branch encoder, a cross-modal feature interaction module is provided. This module receives feature maps from the same level from different encoder branches, calculates the feature correlation between different modal feature maps using a cross-modal attention mechanism, and performs weighted fusion of features based on this correlation to generate an interactive fused feature map. The final depth feature sequences output by each encoder branch, along with the interactive fused feature map generated at a specific level, are jointly input to the single-branch decoder. The decoder gradually increases the spatial resolution of the feature map through upsampling operations and integrates the feature information from each layer of the encoder through skip connections, generating a fused multimodal depth feature map at the final output layer of the decoder. The improved encoder-decoder network is optimized based on the channel characteristics and spatial dependencies of different image modalities. Optimization measures include introducing a channel characteristic weighting module after each downsampling stage of the encoder. This module learns a set of weights to adaptively weight feature channels from different image modalities, thus highlighting the feature contributions of important modalities before feature fusion. A spatial dependency enhancement module is introduced before each upsampling stage of the decoder. This module models long-range spatial dependencies between feature points by calculating self-attention in the spatial dimension of the feature map and injects enhanced spatial context information into the features to be upsampled. Furthermore, a cross-modal feature calibration connection is introduced on the skip connection paths corresponding to the encoder and decoder. This connection dynamically selects and fuses the most relevant multimodal features from the corresponding layer of the encoder based on the specific contextual information requirements of the current layer of the decoder.

[0024] In this implementation, spatially aligned multimodal image data blocks serve as network input. The improved encoder-decoder network comprises a multi-branch encoder and a single-branch decoder. The multi-branch encoder assigns a separate encoder branch to each of the computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) images. Specifically, the spatially aligned multimodal image data blocks are split into independent image channels according to their original image modalities. The CT, MRI, and PET channels are then fed into their respective independent encoder branches for depth feature extraction. Each independent encoder branch progressively downsamples the data through operations such as convolution, normalization, and activation functions, outputting a depth feature sequence for the corresponding modality. In some embodiments, the independent encoder branches can share some shallow network structures to extract general features while remaining independent in their deep network structures to capture modality-specific features. This independent encoder branch design allows the network to adaptively learn unique representations for different image modalities.

[0025] At a specific level of the multi-branch encoder, a cross-modal feature interaction module is set up. This module receives feature maps from the same level of the computed tomography (CT) encoder branch, magnetic resonance imaging (MRI) encoder branch, and positron emission tomography (PET) encoder branch as input. The module calculates the feature correlation between different modal feature maps using a cross-modal attention mechanism. This mechanism first linearly projects the feature maps from different branches into query vectors, key vectors, and value vectors. It then calculates the similarity between the query vector and the key vector to obtain attention weights, and uses these attention weights to perform a weighted summation of the value vectors, thereby achieving feature fusion. Optionally, the attention weight calculation method used by the cross-modal feature interaction module can be expressed as follows: in: Represents the query vector matrix. Represents the key vector matrix, Represents a value vector matrix. The dimension of the key vector is represented by , and softmax represents the normalized exponential function. The cross-modal feature interaction module performs weighted fusion of features from different modalities based on the calculated attention weights, generating a fused feature map after interaction. This fused feature map contains both its own modal information and complementary information from other modalities. In specific implementations, the deep feature sequences output by each independent encoder branch and the fused feature map generated by the cross-modal feature interaction module are input together into a single-branch decoder. The single-branch decoder gradually restores the spatial resolution of the feature map through upsampling operations and integrates feature information from corresponding levels in the multi-branch encoder through skip connections, generating a fused multimodal deep feature map containing rich multimodal information at the final output layer of the decoder. In some embodiments, the features integrated by skip connections can come from the output of independent encoder branches or from the output of the cross-modal feature interaction module.

[0026] The improved encoder-decoder network is optimized based on the channel characteristics and spatial dependencies of different image modalities. Optimization measures include introducing a channel feature weighting module after each downsampling stage of the encoder. This module receives the multimodal features output from the current downsampling stage and learns a set of weight vectors corresponding to the channel dimension through global average pooling and fully connected layers. These weight vectors are used to adaptively weight the feature channels from different image modalities. Before feature fusion, the module uses the learned weights to scale the feature channels of different modalities channel-by-channel, thereby highlighting the feature contributions of important modalities and suppressing the feature responses of noisy or redundant modalities. A spatial dependency enhancement module is introduced before each upsampling stage of the decoder. This module models long-range spatial dependencies by calculating the self-attention of the feature map in the spatial dimension. It models the spatial positional relationships of the input feature map, generating a spatial attention map that reflects the correlation strength between different spatial positions in the feature map. The module multiplies the spatial attention map with the original input feature map, injecting the enhanced spatial context information into the features to be upsampled. Optionally, a cross-modal feature calibration connection is introduced on the skip connection paths corresponding to the encoder and decoder. This connection dynamically selects and fuses the most relevant multimodal features from the corresponding layer of the encoder based on the contextual information requirements of the current layer of the decoder. The cross-modal feature calibration connection can be a lightweight attention network that uses the features of the current layer of the decoder as the query and the features of all available modalities in the corresponding layer of the encoder as the keys and values. It calculates attention weights and performs a weighted sum to achieve dynamic feature selection and fusion. It can be understood that the channel feature weighting module, the spatial dependency enhancement module, and the cross-modal feature calibration connection work together to enable the improved encoder-decoder network to more effectively fuse multimodal information and capture key features.

[0027] In one embodiment of the present invention, when performing multi-level context-aware processing on the fused multimodal deep feature map, refer to... Figure 3 A spatial pyramid pooling structure is constructed, applying multiple pooling windows of different scales in parallel on the input fused multimodal deep feature map to extract multi-scale contextual features with different receptive fields. Simultaneously, dilated convolutions with different dilation rates are used to construct multiple parallel convolutional paths on the fused multimodal deep feature map to capture feature information covering multiple spatial contexts. The multi-scale contextual features extracted from the spatial pyramid pooling structure, along with the multi-range contextual information captured from the dilated convolutional paths, are concatenated with the original fused multimodal deep feature map. The resulting combined feature map is then processed by channel attention and spatial attention mechanisms. The channel attention mechanism recalibrates the importance weights of each channel in the feature map, while the spatial attention mechanism re-evaluates the importance of each spatial location in the feature map. After recalibration, an enhanced semantic feature map is generated. When the enhanced semantic feature map is input into the region candidate generation network to generate a set of candidate 3D bounding boxes, multiple 3D anchor boxes with different scales and aspect ratios are first preset in the network. These anchor boxes are densely sliding and covering the 3D space corresponding to the enhanced semantic feature map. For each sliding anchor box, features of the corresponding region are extracted from the enhanced semantic feature map. These features are then input into two parallel fully connected layers. One fully connected layer predicts the probability score of the anchor box containing a cancer target, while the other predicts the position and size correction of the anchor box relative to the potential real cancer region bounding box. Based on the probability scores of cancer targets predicted by the fully connected layers, all sliding anchor boxes are sorted, and the top few with the highest probability scores are selected as initial candidate boxes. Using the bounding box position correction predicted by the other fully connected layer, the position coordinates and size parameters of these initial candidate boxes are adjusted and optimized, resulting in a set of candidate 3D bounding boxes with more accurate positions and sizes.

[0028] In practical implementation, when performing multi-level context-aware processing on the fused multimodal deep feature map, a spatial pyramid pooling structure is constructed. This structure applies multiple pooling windows of different scales in parallel on the fused multimodal deep feature map, such as 1x1x1, 2x2x2, and 4x4x4 windows, to extract multi-scale contextual features with different spatial ranges. Using dilated convolutions with different dilation rates, multiple parallel convolutional paths are constructed on the fused multimodal deep feature map. In some embodiments, three parallel convolutional paths can be set with dilation rates of 1, 2, and 4, respectively, to capture multi-range contextual information ranging from local details to broader regions. The multi-scale contextual features extracted from the spatial pyramid pooling structure, along with the multi-range contextual information captured from the dilated convolutional paths, and the original fused multimodal deep feature map, are concatenated along the channel dimension to form a combined feature map containing richer contextual information. Optionally, the combined feature map obtained after splicing is processed by channel attention and spatial attention mechanisms in turn. The channel attention mechanism recalibrates the importance weight of each channel in the combined feature map by modeling the relationship between channels, while the spatial attention mechanism re-evaluates the importance of each spatial position in the combined feature map by modeling the spatial position relationship. After recalibration, an enhanced semantic feature map is generated.

[0029] It is understandable that multi-level context-aware processing aims to aggregate information at different scales to enhance feature representation. The enhanced semantic feature map is input into a region candidate generation network to generate a set of candidate 3D bounding boxes. In practice, multiple 3D anchor boxes with different scales and aspect ratios are first pre-set in the region candidate generation network. The size design of the anchor boxes can be based on the common size distributions of several cancer lesions in the training data, as shown in Table 1. Table 1: A possible configuration of anchor point frame dimensions and aspect ratios In some embodiments, these pre-defined anchor boxes are densely slid across the 3D space corresponding to the enhanced semantic feature map, with the sliding step size determined based on the downsampling rate of the feature map. For each slid anchor box location, features of the corresponding region are extracted from the enhanced semantic feature map, and these region features are input into two parallel fully connected layers. One fully connected layer predicts the probability score of containing a cancer target within the anchor box, and the other fully connected layer predicts the position and size correction of the anchor box relative to the bounding box of the potential real cancer region. The bounding box position correction typically includes a center coordinate offset and a scale scaling. Optionally, it is assumed that the center coordinates of the anchor box are... Width, height, and depth are The predicted correction amount is The adjusted candidate box coordinates can be calculated as follows: candidate box center Candidate box width The calculation methods for the other dimensions are similar. This represents the normalized offset of the center coordinates. The logarithm of the scaling factor.

[0030] Based on the probability scores of cancer targets predicted by the first fully connected layer, all sliding anchor boxes are sorted, and the top N anchor boxes with the highest probability scores are selected as initial candidate boxes, where N is a preset positive integer. Using the bounding box position corrections predicted by the second fully connected layer, the position coordinates and size parameters of these initial candidate boxes are adjusted and optimized. The adjustment process applies the aforementioned formula, resulting in a set of candidate 3D bounding boxes with more accurate positions and sizes. In essence, the region candidate generation network achieves rapid and dense proposals for potential lesion regions in 3D space through preset anchor boxes and regression correction.

[0031] In one embodiment of the present invention, when the segmentation network processes the image regions within the candidate 3D bounding box set, a corresponding 3D image region block is cropped from the original spatially aligned multimodal image data block based on the 3D coordinates of each candidate 3D bounding box. Each 3D image region block is input into the segmentation network, which employs an encoder-decoder structure. The encoder part downsamples the input 3D image region block multiple times to extract high-level semantic features, while the decoder part gradually restores the spatial resolution of the feature map to the size of the original input region block through upsampling and skip connections. In the final output layer of the decoder, the network outputs the probability value of each voxel belonging to a cancer lesion through a segmentation head, forming a probability heatmap. Simultaneously, it outputs the probability distribution of the entire 3D image region block belonging to various specific cancer types through a classification head. Thresholding is applied to the probability heatmap output by the segmentation head, marking voxels with probability values ​​higher than a preset threshold as cancer lesions, thereby generating a pixel-level segmentation mask for the candidate region. Simultaneously, the type with the highest probability in the probability distribution output by the classification head is recorded as the predicted cancer type for the candidate region. The training process of this segmentation network involves acquiring a multimodal image training dataset with pixel-level segmentation annotations and cancer type annotations. Three-dimensional image regions containing cancer lesions are extracted from the training dataset as training samples. These training samples are input into the segmentation network to obtain a predicted segmentation probability heatmap and a cancer type probability distribution. The segmentation loss between the predicted segmentation probability heatmap and the ground truth pixel-level segmentation annotations is calculated. The classification loss between the predicted cancer type probability distribution and the ground truth cancer type annotations is calculated. Combining the segmentation and classification losses, all parameters of the segmentation network are updated using backpropagation. This process is iterated until the network training converges.

[0032] In practice, from the original spatially aligned multimodal image data blocks, based on the 3D coordinate parameters of each candidate 3D bounding box, the corresponding 3D image region blocks are cropped using a 3D array slicing operation. In practice, assuming the coordinates of a candidate 3D bounding box are... Then, all voxels indexed within the three-dimensional boundary are extracted from the multimodal image data block to form a fixed-size three-dimensional image region block containing multimodal channel information. In some embodiments, to ensure consistent input size, the extracted three-dimensional image region block is resampled to a fixed spatial size preset by the network, such as 128x128x128 voxels, using an interpolation method.

[0033] Each cropped and resized 3D image region is input into the segmentation network, which employs an encoder-decoder structure. The encoder part of the segmentation network downsamples the input 3D image region multiple times to extract high-level semantic features. Downsampling operations are typically implemented using convolutional or pooling layers with a stride greater than 1. The decoder part of the segmentation network then gradually restores the spatial resolution of the feature map through upsampling operations. Upsampling operations can be implemented using transposed convolution or interpolation methods. The decoder integrates feature information from corresponding layers of the encoder through skip connections to supplement spatial details, restoring the feature map to the resolution of the original input region. In the final output layer of the decoder, the segmentation network outputs results in parallel through a segmentation head and a classification head. The segmentation head outputs the probability of each voxel belonging to a cancer lesion through a 1x1x1 convolutional layer and an activation function, forming a 3D probability heatmap. The classification head outputs the probability distribution of the entire 3D image region belonging to various specific cancer types through a global pooling layer and a fully connected layer. Optionally, the activation function can be a sigmoid function for the segmentation head to output voxel probabilities, and a softmax function for the classification head to output type probabilities.

[0034] Thresholding is applied to the 3D probability heatmap output by the segmentation head. Voxels with probability values ​​higher than a preset threshold (e.g., 0.5) are marked as cancer lesions, thereby generating a binarized pixel-level segmentation mask for the candidate region. Simultaneously, the type with the highest probability in the probability distribution output by the classification head is recorded as the predicted cancer type for the candidate region. In essence, the segmentation network achieves precise pixel-level localization and overall type determination of lesions within the candidate region. The training process of the segmentation network involves acquiring a multimodal image training dataset with pixel-level segmentation annotations and cancer type annotations. The training dataset contains a large number of 3D medical image samples already annotated by experts. In specific implementation, 3D image region blocks containing cancer lesions are extracted from the multimodal image training dataset as training samples. The extraction process can be based on real lesion annotation bounding boxes, as shown in Table 2. Table 2: Composition of the Training Dataset The training samples are input into the segmentation network to obtain the predicted segmentation probability heatmap and the cancer type probability distribution. The segmentation loss between the predicted segmentation probability heatmap and the actual pixel-level segmentation annotations is calculated. Segmentation loss measures the error in pixel-level prediction. In some embodiments, segmentation loss... A combination of Dice loss and binary cross-entropy loss can be used, and its calculation method can be expressed as follows: in: Indicates the predicted first The probability that an individual element is a lesion. Representing the true first Binary labeling (0 or 1) for individual elements. It is a hyperparameter that balances the weights of two items. This represents the summation over all voxels. The classification loss is calculated between the predicted cancer type probability distribution and the true cancer type labeling. Classification loss measures the error in type prediction. The cross-entropy loss function is typically used, combined with segmentation loss. With classification loss By updating all learnable parameters of the segmentation network through backpropagation, it can be understood that the total loss... ,in These are the weighting coefficients that balance the two losses. The process of forward propagation, loss calculation, backpropagation, and parameter update is performed iteratively until the performance of the segmentation network converges on the validation set. Optionally, the optimization algorithm can be Adam or stochastic gradient descent.

[0035] In one embodiment of the present invention, when generating the final three-dimensional cancer lesion segmentation and recognition result based on pixel-level segmentation masks and cancer type probabilities, the pixel-level segmentation masks and corresponding cancer type probabilities output from the segmentation network after processing all candidate regions are received. Non-maximum suppression (NMS) is applied to these pixel-level segmentation masks. This process calculates the overlap between different segmentation masks in three-dimensional space and removes redundant segmentation masks that highly overlap with masks with higher cancer type probabilities. The pixel-level segmentation masks retained after NMS are mapped back to the global three-dimensional coordinate system of the original multimodal medical image set based on the coordinate information of their corresponding candidate three-dimensional bounding boxes. In the global three-dimensional coordinate system, spatially adjacent pixel-level segmentation masks with the same predicted cancer type are merged to form several complete and connected three-dimensional cancer lesion segments. Each three-dimensional cancer lesion segment is labeled with its corresponding predicted cancer type and probability value. All independent three-dimensional cancer lesion segments and their labeled information are combined to form the final three-dimensional cancer lesion segmentation and recognition result.

[0036] In a specific implementation, the system receives pixel-level segmentation masks and corresponding cancer type probabilities output from the segmentation network after processing all candidate regions. Each candidate region outputs a pixel-level segmentation mask and a predicted cancer type and its probability. Non-maximum suppression (NMS) is applied to these pixel-level segmentation masks, aiming to remove highly overlapping redundant segmentation masks in three-dimensional space. In some embodiments, NMS calculates the 3D intersection-union ratio (IUU) between every two pixel-level segmentation masks, defined as the ratio of the intersection to the union of the 3D volumes of the two segmentation masks. The IUU measures the degree of overlap between two 3D regions. When the IUU of two pixel-level segmentation masks is higher than a preset overlap threshold, and their predicted cancer types are the same, NMS compares the cancer type probabilities corresponding to these two pixel-level segmentation masks. NMS retains the pixel-level segmentation mask with the higher cancer type probability and suppresses the pixel-level segmentation mask with the lower cancer type probability. Optionally, the overlap threshold used to determine whether to suppress the mask can be set to 0.5. This comparison and suppression process is performed iteratively on all pixel-level segmentation mask pairs until no redundant masks need to be suppressed. The 3D intersection-union ratio is then calculated. The process can be described by a formula: in: and Let represent the sets of three-dimensional voxels occupied by the m-th and n-th pixel-level segmentation masks, respectively. This represents the volume (number of voxels) of the intersection of two sets. This represents the volume (number of voxels) of the union of two sets.

[0037] The pixel-level segmentation masks, preserved after non-maximum suppression, are mapped back to the global 3D coordinate system of the original multimodal medical image set. The mapping process is based on the coordinate information of the candidate 3D bounding boxes that generated each pixel-level segmentation mask. The local coordinates of the pixel-level segmentation mask need to be added with the starting coordinate offset of its corresponding candidate 3D bounding box to transform it to the global coordinate system. In the global 3D coordinate system, spatially adjacent pixel-level segmentation masks with the same predicted cancer type are merged. In some embodiments, the merging operation can be implemented through 3D connectivity component analysis, treating pixel-level segmentation masks with a spatial distance less than a preset distance threshold and the same predicted cancer type label as belonging to the same lesion entity, and merging their voxel sets. After merging, several complete and connected 3D cancer lesion segments are formed, each representing an independent, physically continuous suspected cancer region. Each 3D cancer lesion segment is labeled with its corresponding predicted cancer type and its probability value. Optionally, the probability value can be the average or maximum value of the cancer type probabilities of all pixel-level segmentation masks constituting that segment. All independent 3D cancer lesion segments and their annotation information are combined to form the final 3D cancer lesion segmentation and recognition result. This final result includes the lesion's precise morphology and spatial location in 3D space, as well as the predicted cancer type and confidence level.

[0038] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An automatic segmentation and recognition method for cancer images based on multimodal imaging, characterized in that, The method includes: Obtain a multimodal medical image set corresponding to the target case, wherein the multimodal medical image set includes at least computed tomography images, magnetic resonance imaging images, and positron emission tomography images; Perform three-dimensional spatial registration and voxel alignment on the multimodal medical image set to generate spatially aligned multimodal image data blocks; An improved encoder-decoder network is used to extract and fuse features from the spatially aligned multimodal image data blocks to generate a fused multimodal depth feature map. The improved encoder-decoder network optimizes the standard encoder-decoder structure based on the channel characteristics and spatial dependencies of different modal images. Perform multi-level context-aware processing on the fused multimodal deep feature map to generate an enhanced semantic feature map; The enhanced semantic feature map is input into the region candidate generation network to generate a set of candidate 3D bounding boxes containing suspected cancer regions. The segmentation network is invoked to process the image regions within the candidate 3D bounding box set, and the pixel-level segmentation mask and cancer type probability corresponding to each candidate region are output. Based on the pixel-level segmentation mask and the cancer type probability, the final three-dimensional cancer lesion segmentation and recognition result is generated.

2. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 1, characterized in that, The step of performing three-dimensional spatial registration and voxel alignment on the multimodal medical image set to generate spatially aligned multimodal image data blocks includes: The computed tomography images in the multimodal medical image set are selected as the reference reference images for spatial registration; Rigid registration calculations based on mutual information were performed on the magnetic resonance imaging images and positron emission tomography images and the reference reference image respectively to obtain their respective spatial transformation matrices; Using the spatial transformation matrix, three-dimensional spatial resampling and interpolation are performed on magnetic resonance imaging images and positron emission tomography images, so that each voxel is physically aligned with the corresponding voxel of the reference image. Spatially aligned computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images are stacked along the channel dimension to form the spatially aligned multimodal image data block.

3. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 1, characterized in that, The improved encoder-decoder network is used to extract and fuse features from the spatially aligned multimodal image data blocks to generate a fused multimodal depth feature map, including: The improved encoder-decoder network includes a multi-branch encoder and a single-branch decoder, wherein the multi-branch encoder is configured with an independent encoder branch for each modal image; The spatially aligned multimodal image data block is split into independent image channels according to image modality, and each channel is input into the corresponding encoder branch for depth feature extraction to obtain the depth feature sequence of each modality. A cross-modal feature interaction module is set at a specific level of a multi-branch encoder. The cross-modal feature interaction module receives feature maps of the same level from different encoder branches, calculates the feature correlation between modes through a cross-modal attention mechanism, performs feature weighted fusion, and generates an interactive fused feature map. The final output depth feature sequences of each encoder branch and the fused feature map after interaction are input into the single-branch decoder. The decoder gradually recovers the spatial resolution through upsampling and skip connections, and outputs the fused multimodal depth feature map containing rich multimodal information in the final layer.

4. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 1, characterized in that, The step of performing multi-level context-aware processing on the fused multimodal deep feature map to generate an enhanced semantic feature map includes: A spatial pyramid pooling structure is constructed, and pooling windows of different scales are applied to the fused multimodal deep feature map to extract multi-scale contextual features; By utilizing dilated convolution, parallel convolutional paths with different receptive fields are constructed on the fused multimodal deep feature map to capture multi-range contextual information; The extracted multi-scale context features and the captured multi-range context information are then concatenated with the original fused multimodal deep feature map. The concatenated feature map is processed by channel attention and spatial attention mechanisms to recalibrate the importance of different channels and spatial positions in the feature map, thereby generating the enhanced semantic feature map.

5. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 1, characterized in that, The step of inputting the enhanced semantic feature map into a region candidate generation network to generate a set of candidate 3D bounding boxes containing suspected cancer regions includes: In the region candidate generation network, multiple anchor boxes of different scales and aspect ratios are preset, and the anchor boxes are densely slid in the three-dimensional space corresponding to the enhanced semantic feature map. For each anchor box location, features of the corresponding region are extracted from the enhanced semantic feature map and input into two parallel fully connected layers. One fully connected layer is used to predict the probability that the anchor box contains a cancer target, and the other fully connected layer is used to predict the positional correction of the anchor box relative to the true cancer region bounding box. All anchor boxes are sorted according to the predicted probability of containing cancer targets, and the top few anchor boxes with the highest probability are selected as initial candidate boxes. The position and size of the initial candidate boxes are adjusted using the predicted bounding box position correction amount to obtain a set of candidate 3D bounding boxes with more accurate positions.

6. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 1, characterized in that, The segmentation network is invoked to process the image regions within the candidate 3D bounding box set, outputting a pixel-level segmentation mask and cancer type probability for each candidate region, including: From the original spatially aligned multimodal image data blocks, the corresponding three-dimensional image region blocks are cropped according to the coordinates of the candidate three-dimensional bounding boxes; Each 3D image region block is input into the segmentation network, which adopts an encoder-decoder structure. The encoder downsamples the input region block to extract high-level semantic features, while the decoder gradually restores the resolution of the original region block through upsampling and skip connections. In the final output layer of the decoder, a segmentation head outputs the probability that each voxel belongs to a cancer lesion, forming a probability heatmap, and a classification head outputs the probability distribution of the entire region block belonging to a specific cancer type. Thresholding is applied to the probability heatmap, and voxels with a probability higher than a preset threshold are marked as cancer lesions. The pixel-level segmentation mask is generated, and the type with the highest probability in the probability distribution of the specific cancer type is recorded as the predicted cancer type of the candidate region.

7. The automatic segmentation and recognition method for cancer images based on fused multimodal images according to claim 6, characterized in that, The training process of the segmentation network includes: Obtain a multimodal image training dataset with pixel-level segmentation annotations and cancer type annotations; Three-dimensional image regions containing cancer lesions are extracted from the multimodal image training dataset as training samples; The training samples are input into the segmentation network to obtain the predicted segmentation probability heatmap and cancer type probability distribution; Calculate the segmentation loss between the predicted segmentation probability heatmap and the actual pixel-level segmentation annotation; Calculate the classification loss between the predicted cancer type probability distribution and the actual cancer type label; The parameters of the segmentation network are updated using the backpropagation algorithm, combining the segmentation loss and the classification loss, until the network converges.

8. The automatic segmentation and recognition method for cancer images based on the fusion of multimodal images according to claim 1, characterized in that, The step of generating the final three-dimensional cancer lesion segmentation and recognition result based on the pixel-level segmentation mask and cancer type probability includes: Receive pixel-level segmentation masks and cancer type probabilities for all candidate regions from the segmentation network; Non-maximum suppression processing is applied to the pixel-level segmentation mask to remove redundant segmentation masks that are highly overlapping in three-dimensional space and have a low probability of cancer type. The pixel-level segmentation mask retained after non-maximum suppression is mapped back to the global three-dimensional coordinate system of the original multimodal medical image set; In the global three-dimensional coordinate system, adjacent pixel-level segmentation masks that belong to the same predicted cancer type are merged to form a complete three-dimensional cancer lesion segmentation body. Each 3D cancer lesion segment is labeled with its corresponding predicted cancer type and probability. All lesion segments and their labeled information are combined to form the final 3D cancer lesion segmentation and recognition result.

9. The automatic segmentation and recognition method for cancer images based on the fusion of multimodal images according to claim 1, characterized in that, The improved encoder-decoder network optimizes the standard encoder-decoder structure based on the channel characteristics and spatial dependencies of different modal images, including: After each downsampling stage of the encoder, a channel feature weighting module is introduced. This module learns the weights of different image modal channels and adaptively weights the feature channels of each modality before feature fusion to highlight the feature contributions of important modalities. Before each upsampling stage of the decoder, a spatial dependency enhancement module is introduced. This module models long-range spatial dependencies by calculating the self-attention of the feature map in the spatial dimension and injects the enhanced spatial information into the upsampled features. On the skip connection path corresponding to the encoder and decoder, a cross-modal feature calibration connection is introduced. This connection dynamically selects and fuses the most relevant multimodal features from the corresponding layer of the encoder based on the context information requirements of the current layer of the decoder.

10. An automatic cancer image segmentation and recognition system integrating multimodal imaging, characterized in that, The system includes a processor and a memory connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to run the programs, instructions, or code in the memory to implement the automatic segmentation and recognition method for cancer images based on the fusion of multimodal images as described in any one of claims 1-9.

Images