A method and system for identifying and segmenting the boundary of a chest tumor based on multi-modal images

Abstract

The application discloses a chest tumor boundary recognition and segmentation method and system based on multi-modal images, relates to the field of computer vision and image processing, and comprises the following steps: CT, MRI and PET multi-modal chest images are preprocessed and spatially registered; a double-branch attention U-Net network is constructed, multi-modal fusion features are obtained through inter-modal feature weighted fusion, and a self-adaptive threshold algorithm is used for tumor rough segmentation; local features of each mode boundary are extracted for enhanced fusion, and a fine segmentation result is output in combination with a boundary refinement network; and through boundary optimization and index evaluation, the optimal segmentation is obtained through dynamic adjustment. The application has the advantages that: by fusing multi-modal chest image information, combining the attention mechanism, the rough-fine progressive segmentation strategy and dynamic evaluation and optimization, the limitations of single mode are effectively made up, and the accuracy and robustness of tumor boundary recognition are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing, and in particular to a method and system for chest tumor boundary recognition and segmentation based on multimodal images. Background Technology

[0002] Chest tumor boundary identification and segmentation is a crucial task in medical image analysis, aiming to accurately locate the contours of chest tumors such as those in the lungs and breasts, providing key information for clinical diagnosis, treatment planning, and prognostic assessment. With the development of medical imaging technology and the widespread use of imaging modalities such as CT, MRI, and X-rays, the demand for early detection and precise segmentation of tumors is increasing. Traditional methods rely on manual delineation, which is inefficient and easily influenced by subjective factors.

[0003] Current methods for segmenting thoracic tumor boundaries rely on single-modal images, resulting in limited information dimensionality and an inability to fully utilize complementary multimodal information. While attempts at multimodal fusion exist, insufficient registration accuracy remains a problem; rigid registration alone cannot correct local nonlinear deformations, easily leading to fusion artifacts. Feature extraction and fusion methods are relatively coarse, lacking precise weighted fusion guided by cross-modal attention mechanisms, making it difficult to effectively mine complementary features between modalities. Segmentation strategies often lack progressive optimization design, either resulting in severe noise interference through direct fine segmentation or insufficient boundary optimization after coarse segmentation, easily leading to boundary fragmentation and discontinuities. Furthermore, existing methods have rigid evaluation mechanisms, relying heavily on single metrics and lacking feedback mechanisms for dynamically adjusting feature fusion weights. When faced with complex scenarios such as variable tumor morphology and blurred boundaries, both robustness and segmentation accuracy are significantly insufficient. Summary of the Invention

[0004] To improve existing methods and systems, this paper provides a method and system for chest tumor boundary recognition and segmentation based on multimodal images. This method effectively overcomes the limitations of single-modal imaging by fusing multimodal chest image information, combining attention mechanisms with a coarse-to-fine progressive segmentation strategy and dynamic evaluation optimization, and significantly improves the accuracy and robustness of tumor boundary recognition.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Methods for identifying and segmenting chest tumor boundaries based on multimodal images include: Acquire multimodal chest imaging data, including computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, and perform preprocessing operations on the chest imaging data of each modality; Using the preprocessed CT images as the reference images, the preprocessed images of other modalities are spatially registered with the reference images to obtain a spatially aligned multimodal chest image set. A dual-branch feature extraction network based on an attention mechanism is constructed. Each branch adopts an improved U-Net network structure. In each downsampling stage of the encoder, complementary features are fused through intermodal correlation weights. The decoder recovers and fuses the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Based on the preliminary fusion features of multimodal data, an adaptive threshold segmentation algorithm is used for coarse segmentation of the tumor region. By optimizing the segmentation boundary, a continuous and complete candidate tumor boundary contour is obtained. Based on the local images of each modality boundary within the candidate tumor boundary contour, features are extracted from the local image blocks of each modality boundary. The local image features of each modality boundary are then fused using a weighted average fusion algorithm to obtain enhanced multimodal boundary features.

[0006] The enhanced multimodal boundary features are concatenated with the preliminary multimodal fusion features. A boundary refinement segmentation network with an encoder-decoder structure is used to output a probability map of the tumor boundary. The tumor is then identified by a threshold to obtain the preliminary fine segmentation boundary. The initial fine segmentation boundary is smoothed and adjusted. The Dice similarity coefficient, crossover ratio, and average distance of the optimized segmentation boundary are calculated. By comparing with the preset threshold, the final chest tumor boundary segmentation result is output or the feature fusion weights are adjusted to re-fuse boundary features and perform fine segmentation.

[0007] Preferably, the acquisition of multimodal chest imaging data includes computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, and the preprocessing of each modality of chest imaging data specifically includes: Preprocessing operations were performed on the chest image data for each modality. These preprocessing operations included noise suppression, grayscale normalization, and image resolution unification. Noise suppression employs an adaptive median filtering algorithm, which dynamically adjusts the size of the filtering window based on the noise density in different regions of the image to remove noise. Gray-level normalization maps the gray values ​​of each modality image to the [0, 255] range; Image resolution is unified by adjusting the images of each modality to the same spatial resolution using bilinear interpolation.

[0008] Preferably, the step of using the preprocessed CT image as the reference image and spatially registering the preprocessed images of other modalities with the reference image to obtain a spatially aligned multimodal chest image set specifically includes: The preprocessed CT image is selected as the reference image, and the spatial coordinate system parameters of the reference image are extracted. A rigid registration method based on normalized mutual information is adopted to calculate the mutual information value between the image to be registered and the reference image. An optimization function with the goal of maximizing mutual information is constructed. By iteratively adjusting the translation, rotation and scaling parameters, preliminary rigid registration results are obtained. The thin-plate spline interpolation algorithm is used to perform elastic registration on the preliminary rigid registration results, and the residual local nonlinear deformation in the preliminary rigid registration results is corrected to obtain a spatially aligned multimodal chest image set.

[0009] Preferably, the construction of the attention-based dual-branch feature extraction network, where each branch adopts an improved U-Net network structure, involves fusing complementary features through intermodal correlation weights at each downsampling stage of the encoder, and the decoder recovers and fuses the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Specifically, this includes: A dual-branch feature extraction network based on an attention mechanism is constructed. An improved U-Net structure is used within each branch, and the spatially aligned images of each modality are input into the corresponding feature extraction branch. Each branch encoder performs multi-scale feature extraction through 4 levels of downsampling. Each level of downsampling uses 2 layers of convolution to extract hierarchical features from low-dimensional texture to high-dimensional semantics. After each downsampling stage, a cross-modal attention module is embedded to calculate the correlation weights between the current modality and other modal features. Complementary features are then fused according to the weights to generate an enhanced feature map. The decoder recovers the feature map resolution through four levels of upsampling. Each level of upsampling is combined with the enhanced feature map of the encoder level for skip connections to obtain a preliminary multimodal fused feature map.

[0010] Preferably, the step of using an adaptive threshold segmentation algorithm to coarsely segment the tumor region based on preliminary multimodal fusion features, and obtaining continuous and complete tumor candidate boundary contours through optimization of the segmentation boundaries, specifically includes: Based on the preliminary fusion features of multimodal features, all gray levels of the feature map are traversed, the inter-class variance of the foreground and background regions corresponding to each gray level is calculated, and the gray level with the largest inter-class variance is selected as the final segmentation threshold. The multimodal preliminary fusion feature map is binarized. Pixels with gray values ​​greater than the segmentation threshold are marked as foreground pixels, and pixels with gray values ​​less than or equal to the segmentation threshold are marked as background pixels, thus obtaining a coarse segmentation binary map containing the tumor candidate region and the background. Edges are extracted from the coarsely segmented binary image using the Canny edge detection algorithm to obtain candidate tumor boundaries. Noise from the candidate boundaries is removed by morphological dilation and erosion operations to obtain continuous and complete candidate tumor boundary contours.

[0011] Preferably, the step of extracting features from the local boundary images of each modality within the tumor candidate boundary contour range, and fusing the local boundary image features of each modality using a weighted average fusion algorithm to obtain enhanced multimodal boundary features specifically includes: Based on the candidate tumor boundary contour, a set of local boundary images is constructed on each spatially aligned modal image. Features are extracted from the boundary local images of each modality. The gray-level gradient features of the boundary are extracted from the CT image blocks, the texture features are extracted from the MRI image blocks, and the metabolic activity distribution features are extracted from the PET image blocks. Shape features are extracted from all modal images to form a multi-dimensional modality-specific boundary feature set; A feature fusion matrix is ​​constructed, and a weighted average fusion strategy is used to fuse the modality-specific boundary features to obtain the initial fused features; The initial fused features are batch normalized to obtain enhanced multimodal boundary features.

[0012] Preferably, the step of concatenating the enhanced multimodal boundary features with the preliminary multimodal fusion features, and outputting a probability map of the tumor boundary through a boundary refinement segmentation network with an encoder-decoder structure, and determining the tumor based on a threshold to obtain the preliminary fine segmentation boundary specifically includes: A boundary refinement segmentation network with an encoder-decoder architecture is constructed. Multimodal boundary features and preliminary fusion features are fused by channel splicing to form an input that combines global semantics and local boundary information. The encoder captures subtle differences and spatial correlations in tumor boundaries through 4 levels of downsampling and ReLU activation; The decoder upsamples step by step and combines deep and shallow features with skip connections. It optimizes the feature representation through convolution, gradually restores the resolution, and outputs a boundary probability map with the same size as the original image. The pixel values ​​of the probability map are mapped to the [0,1] interval by using the sigmoid activation function, and the preliminary fine segmentation boundary is obtained by using a threshold.

[0013] Preferably, the step of smoothing the initial fine segmentation boundary, calculating the Dice similarity coefficient, intersection-over-union ratio, and average boundary distance of the optimized segmentation boundary, and comparing it with a preset threshold to output the final chest tumor boundary segmentation result or adjusting the feature fusion weights to re-fuse boundary features and perform fine segmentation specifically includes: Using the preliminary fine segmentation boundary as the initial contour, an active contour model is constructed. The initial contour is input into the active contour model, and the contour curve parameters are iteratively adjusted. During the iteration process, the contour curve moves along the direction of the gray-level gradient decrease at the tumor edge until the contour position is stable, and the smoothed and optimized tumor boundary is obtained. The Dice similarity coefficient, crossover ratio, and average distance between the optimized boundary and the gold standard are calculated. By comparing with the threshold, if all criteria are met, the final chest tumor boundary segmentation result is output. If the criteria are not met, the feature fusion weights are adjusted and boundary feature fusion and fine segmentation are performed again.

[0014] Furthermore, a chest tumor boundary recognition and segmentation system based on multimodal imagery is proposed, including: Data acquisition and preprocessing module: Acquires multimodal chest images from CT, MRI, and PET, and performs noise suppression, grayscale normalization, and resolution unification preprocessing; Multimodal registration module: Using CT images as a reference, spatial alignment of multimodal images is achieved through rigid registration and flexible registration; Dual-branch feature extraction module: adopts an improved U-Net structure, combines cross-modal attention mechanism to fuse multimodal features, and generates a preliminary fused feature map; Tumor coarse segmentation module: Based on adaptive threshold segmentation and morphological processing, it extracts continuous and complete candidate tumor boundary contours; Boundary feature enhancement module: Extracts local features of each modality boundary and generates enhanced multimodal boundary features through weighted fusion; Boundary Refinement Segmentation Module: The encoder-decoder network fuses global and local features, outputs a probability map, and generates preliminary fine segmentation boundaries. Post-processing and evaluation module: performs smoothing optimization on the segmentation results, calculates evaluation indicators, dynamically adjusts parameters, or outputs the final boundary segmentation results; Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0015] Compared with the prior art, the advantages of the present invention are: By fusing multimodal imaging data from CT, MRI, and PET, the limitations of single-modal imaging information are overcome. Adaptive preprocessing and rigid-flexible combined registration ensure data consistency. An improved U-Net network with an attention mechanism and dual branches achieves efficient cross-modal feature fusion. A progressive strategy from coarse to fine segmentation is employed, first obtaining complete candidate boundaries through adaptive thresholding and morphological processing, then enhancing local boundary features and refining the network with an encoder-decoder to improve boundary accuracy. Subsequent smoothing optimization and multi-index dynamic evaluation mechanisms allow for flexible parameter adjustment to ensure segmentation stability, effectively improving the completeness, accuracy, and robustness of tumor boundary identification, providing reliable support for clinical tumor diagnosis and treatment. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method proposed in this invention; Figure 2This is a schematic diagram of the data acquisition and preprocessing proposed in this invention; Figure 3 This is a schematic diagram of the multimodal image spatial registration proposed in this invention; Figure 4 This is a schematic diagram of the cross-modal feature collaborative extraction proposed in this invention; Figure 5 This is a schematic diagram of the coarse segmentation of the tumor region and the extraction of candidate boundaries proposed in this invention; Figure 6 This is a schematic diagram of the multimodal boundary feature fusion enhancement proposed in this invention; Figure 7 This is a schematic diagram of the fine segmentation of tumor boundaries proposed in this invention; Figure 8 This is a schematic diagram illustrating the optimized segmentation results proposed in this invention. Detailed Implementation

[0017] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0018] A multimodal image-based system for identifying and segmenting thoracic tumor boundaries includes: Data acquisition and preprocessing module: Acquires multimodal chest images from CT, MRI, and PET, and performs noise suppression, grayscale normalization, and resolution unification preprocessing; Multimodal registration module: Using CT images as a reference, spatial alignment of multimodal images is achieved through rigid registration and flexible registration; Dual-branch feature extraction module: adopts an improved U-Net structure, combines cross-modal attention mechanism to fuse multimodal features, and generates a preliminary fused feature map; Tumor coarse segmentation module: Based on adaptive threshold segmentation and morphological processing, it extracts continuous and complete candidate tumor boundary contours; Boundary feature enhancement module: Extracts local features of each modality boundary and generates enhanced multimodal boundary features through weighted fusion; Boundary Refinement Segmentation Module: The encoder-decoder network fuses global and local features, outputs a probability map, and generates preliminary fine segmentation boundaries. Post-processing and evaluation module: performs smoothing optimization on the segmentation results, calculates evaluation indicators, dynamically adjusts parameters, or outputs the final boundary segmentation results; Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

[0019] See Figure 1 As shown, a method for chest tumor boundary recognition and segmentation based on multimodal images includes: Step 1: Acquire multimodal chest imaging data, including computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, and perform preprocessing operations on the chest imaging data of each modality; Step 2: Using the preprocessed CT image as the reference image, spatially register the preprocessed images of other modalities with the reference image to obtain a spatially aligned multimodal chest image set; Step 3: Construct a dual-branch feature extraction network based on the attention mechanism. Each branch adopts an improved U-Net network structure. In each downsampling stage of the encoder, complementary features are fused through intermodal correlation weights. The decoder recovers and fuses the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Step 4: Based on the preliminary fusion features of multimodal data, an adaptive threshold segmentation algorithm is used to coarsely segment the tumor region. By optimizing the segmentation boundary, a continuous and complete candidate tumor boundary contour is obtained. Step 5: Based on the local images of each modality boundary within the tumor candidate boundary contour range, feature extraction is performed on the local image blocks of each modality boundary. The local image features of each modality boundary are fused using a weighted average fusion algorithm to obtain enhanced multimodal boundary features. Step 6: The enhanced multimodal boundary features are concatenated with the preliminary multimodal fusion features. A boundary refinement segmentation network with an encoder-decoder structure is used to output a probability map of the tumor boundary. The tumor is then identified by a threshold to obtain the preliminary fine segmentation boundary. Step 7: Smoothly adjust the preliminary fine segmentation boundary, calculate the Dice similarity coefficient, crossover ratio, and average boundary distance of the optimized segmentation boundary, compare it with the preset threshold, and output the final chest tumor boundary segmentation result or adjust the feature fusion weight to perform boundary feature fusion and fine segmentation again.

[0020] See Figure 2 As shown, multimodal chest imaging data were acquired, including computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images. Preprocessing of the chest imaging data for each modality specifically included: Preprocessing operations were performed on the chest image data for each modality. These preprocessing operations included noise suppression, grayscale normalization, and image resolution unification. Noise suppression employs an adaptive median filtering algorithm, which dynamically adjusts the size of the filtering window based on the noise density in different regions of the image to remove noise. Gray-level normalization maps the gray values ​​of each modality image to the [0, 255] range; Image resolution is unified by adjusting the images of each modality to the same spatial resolution using bilinear interpolation.

[0021] See Figure 3 As shown, using the preprocessed CT image as the reference image, the preprocessed images of other modalities are spatially registered with the reference image to obtain a spatially aligned multimodal chest image set, specifically including: The preprocessed CT image is selected as the reference image, and the spatial coordinate system parameters of the reference image are extracted. A rigid registration method based on normalized mutual information is adopted to calculate the mutual information value between the image to be registered and the reference image. An optimization function with the goal of maximizing mutual information is constructed. By iteratively adjusting the translation, rotation and scaling parameters, preliminary rigid registration results are obtained. The thin-plate spline interpolation algorithm is used to perform elastic registration on the preliminary rigid registration results, and the residual local nonlinear deformation in the preliminary rigid registration results is corrected to obtain a spatially aligned multimodal chest image set.

[0022] Specifically, a rigid registration method based on normalized mutual information is used to correct rigid deformations (translation, rotation, scaling) of cross-modal images. First, the overlapping area between the reference CT image and the image to be registered is selected as the region of interest for registration, and the normalized mutual information value of the two images within this area is calculated. With the goal of maximizing the normalized mutual information, the translation (x, y, z axes), rotation (pitch, yaw, roll) and scaling parameters are initialized, and the parameters are iteratively adjusted using the gradient descent method. After each iteration, the normalized mutual information value is recalculated. When the change in mutual information value after three consecutive iterations is less than 0.001, the iteration is stopped, and the rigid registration parameters and the preliminary registered image are obtained. To address residual local nonlinear deformations after rigid registration, such as lung tissue deformation caused by respiratory motion and soft tissue traction deformation, a thin-plate spline interpolation algorithm is used for elastic registration. First, 50-80 feature control points are uniformly selected on the core anatomical structure contours of the baseline CT image and the preliminary registered image, and the correspondence between the control points is calculated using the least squares method. Based on the control points, a thin-plate spline interpolation function is constructed to generate a globally smooth deformation field, which is then used to perform nonlinear deformation mapping on each pixel of the preliminary registered image. During the mapping process, the mean square error between the two registered images is minimized as a supplementary objective, and the deformation field parameters are iteratively optimized until the mean square error is less than a preset threshold, resulting in a fully spatially aligned multimodal image.

[0023] See Figure 4 As shown, a dual-branch feature extraction network based on an attention mechanism is constructed. Each branch adopts an improved U-Net network structure. In each downsampling stage of the encoder, complementary features are fused through intermodal correlation weights. The decoder recovers and fuses the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Specifically, this includes: A dual-branch feature extraction network based on an attention mechanism is constructed. An improved U-Net structure is used within each branch, and the spatially aligned images of each modality are input into the corresponding feature extraction branch. Each branch encoder performs multi-scale feature extraction through 4 levels of downsampling. Each level of downsampling uses 2 layers of convolution to extract hierarchical features from low-dimensional texture to high-dimensional semantics. After each downsampling stage, a cross-modal attention module is embedded to calculate the correlation weights between the current modality and other modal features. Complementary features are then fused according to the weights to generate an enhanced feature map. The decoder recovers the feature map resolution through four levels of upsampling. Each level of upsampling is combined with the enhanced feature map of the encoder level for skip connections to obtain a preliminary multimodal fused feature map.

[0024] Specifically, a dual-branch feature extraction network based on a cross-modal attention mechanism is constructed. The two branches are adapted to the feature characteristics of different modalities of images, such as the CT branch focusing on density feature capture and the MRI / PET branch focusing on soft tissue contrast / metabolic activity feature capture. The basic network framework adopts an improved U-Net structure, adding a cross-modal feature interaction layer between the encoder and decoder, and optimizing the encoder downsampling module—each downsampling layer consists of two 3×3 convolutional kernels connected in series, combined with batch normalization and ReLU activation function to enhance feature representation capability. When initializing the network parameters, the convolutional layer weights are initialized using a He normal distribution. Multimodal images are input into their respective branches, and the encoder is activated to perform multi-scale downsampling to extract features. The feature extraction is divided into four levels according to the image resolution: Level 1 extracts shallow texture features, corresponding to 1 / 2 resolution of the original image; Level 2 extracts mid-level edge features, corresponding to 1 / 4 resolution of the original image; Level 3 extracts deep structure features, corresponding to 1 / 8 resolution of the original image; Level 4 extracts global semantic features, corresponding to 1 / 16 resolution of the original image. After extraction at each level, key features are retained through a max pooling layer. A cross-modal attention module is embedded at each feature extraction level to achieve collaborative enhancement of dual-branch features. The module first calculates the correlation matrix of the feature maps at the same level of the two branches to locate complementary feature regions. Based on the correlation matrix, dynamic weights are assigned, with high weights given to highly complementary features and low weights given to redundant features. The weighted dual-branch features are then fused pixel by pixel to generate an enhanced feature map. After fusion, dimensionality reduction is performed using a 1×1 convolution kernel to eliminate feature dimensionality redundancy. The decoder uses transposed convolution to achieve feature upsampling. Each upsampling layer is spliced ​​with the enhanced feature map of the corresponding layer of the encoder through skip connections to supplement edge detail information; finally, the decoder outputs a multimodal fusion feature map.

[0025] See Figure 5As shown, based on preliminary multimodal fusion features, an adaptive threshold segmentation algorithm is used for coarse segmentation of the tumor region. By optimizing the segmentation boundaries, continuous and complete tumor candidate boundary contours are obtained, specifically including: Based on the preliminary fusion features of multimodal features, all gray levels of the feature map are traversed, the inter-class variance of the foreground and background regions corresponding to each gray level is calculated, and the gray level with the largest inter-class variance is selected as the final segmentation threshold. The multimodal preliminary fusion feature map is binarized. Pixels with gray values ​​greater than the segmentation threshold are marked as foreground pixels, and pixels with gray values ​​less than or equal to the segmentation threshold are marked as background pixels, thus obtaining a coarse segmentation binary map containing the tumor candidate region and the background. Edges are extracted from the coarsely segmented binary image using the Canny edge detection algorithm to obtain candidate tumor boundaries. Noise from the candidate boundaries is removed by morphological dilation and erosion operations to obtain continuous and complete candidate tumor boundary contours.

[0026] Specifically, using an enhanced multimodal fusion feature map as input, an adaptive threshold segmentation algorithm is employed to coarsely extract the tumor region. First, the grayscale distribution histogram of the fusion feature map is statistically analyzed. The grayscale clustering characteristics are analyzed using the maximum inter-class variance method, and an initial threshold is automatically used to divide the tumor foreground and background. Considering the ambiguity of the grayscale transition zone at the tumor edge, a dynamic threshold adjustment mechanism is introduced. The mean grayscale value of the region after the initial threshold segmentation is calculated. If the mean grayscale value of the foreground region deviates from the preset tumor feature grayscale range by more than 10%, the threshold is iteratively adjusted until the deviation meets the requirements. Finally, the feature map is converted into a black-and-white binary image through binarization, with the white area representing the initially coarsely segmented tumor region and the black area representing the background region. The formula for maximum between-class variance is:

[0027] in, For This represents the inter-class variance between the foreground and background after threshold segmentation. A larger value indicates a more significant difference in grayscale between the foreground (tumor region) and the background. For When the threshold is set, it represents the proportion of background pixels in the entire feature map. For When the threshold is set, the proportion of pixels in the foreground region (tumor candidate region) to the entire feature map satisfies the following condition. , For When the threshold is reached, the average gray value of the background region. For When the threshold is reached, the average gray value of the foreground region. This represents the global average grayscale value of the entire fused feature map; An improved Canny edge detection algorithm was used to extract candidate boundaries from the coarsely segmented binary image. First, the binary image was smoothed by Gaussian filtering, and the size of the filter kernel was dynamically adjusted according to the area of ​​the tumor region. Then, the gradient amplitude and direction of the image were calculated to capture the gray-scale change information of the edges. A dual threshold method was used to screen edge points. The high threshold was used to extract high-confidence edges, and the low threshold was used to connect the broken parts between high-confidence edges. The difference between the two thresholds was adaptively set according to the mean gradient of the image to ensure the continuity and accuracy of edge extraction. Finally, a discrete set of tumor candidate boundary points was obtained.

[0028] See Figure 6 As shown, based on the local images of each modality boundary within the tumor candidate boundary contour range, features are extracted from the local image blocks of each modality boundary. A weighted average fusion algorithm is then used to fuse the local image features of each modality boundary to obtain enhanced multimodal boundary features. Specifically, this includes: Based on the candidate tumor boundary contour, a set of local boundary images is constructed on each spatially aligned modal image. Features are extracted from the boundary local images of each modality. The gray-level gradient features of the boundary are extracted from the CT image blocks, the texture features are extracted from the MRI image blocks, and the metabolic activity distribution features are extracted from the PET image blocks. Shape features are extracted from all modal images to form a multi-dimensional modality-specific boundary feature set; A feature fusion matrix is ​​constructed, and a weighted average fusion strategy is used to fuse the modality-specific boundary features to obtain the initial fused features; The initial fused features are batch normalized to obtain enhanced multimodal boundary features.

[0029] Specifically, based on the optimized tumor candidate boundary contour, local boundary regions are precisely cropped from spatially aligned images of each modality to construct a set of boundary local image blocks. First, the candidate boundary contour is extended 5-8 pixels inwards and outwards to form a boundary region of interest (ROI), ensuring coverage of the tumor edge transition zone and adjacent normal tissue. A pixel coordinate mapping method is used to synchronously map the coordinates of the boundary ROI in the baseline CT image to other modalities of MRI and PET, ensuring complete spatial matching of the local boundary image blocks in each modality. The cropped local boundary image blocks of each modality undergo edge smoothing preprocessing, and Gaussian filtering is used to eliminate local noise. Based on the characteristics of different image modalities, modality-specific boundary features of local image blocks are extracted layer by layer to form a multi-dimensional feature set. For CT image blocks, gray-level gradient features, edge direction features, and calcification point distribution features are extracted. For MRI image blocks, T1 / T2 weighted signal contrast features and edge texture features are extracted. For PET image blocks, metabolic activity gradient features and high-metabolic region contour features are extracted. A dynamic weighted fusion model based on an attention mechanism is constructed to optimize the fusion of modal boundary-specific features. Initial weights for each modality feature are preset using a training sample set, and the correlation coefficients between different modal features are calculated. The weights of redundant features with high correlation are reduced, while the weights of complementary features are increased. A segmentation accuracy feedback mechanism is introduced during the fusion process, using the boundary segmentation accuracy as the evaluation index to dynamically adjust the weight allocation ratio until the segmentation accuracy corresponding to the fused features reaches the optimal level. After fusion, feature dimensionality reduction and enhancement are performed using a 1×1 convolutional kernel to strengthen the expressive power of core boundary features and suppress invalid redundant information, resulting in enhanced multimodal boundary features.

[0030] See Figure 7 As shown, enhanced multimodal boundary features are concatenated with preliminary multimodal fusion features. A boundary refinement segmentation network with an encoder-decoder structure is used to output a probability map of the tumor boundary. A threshold is then used to determine if the tumor is a tumor, thus obtaining a preliminary fine segmentation boundary. Specifically, this includes: A boundary refinement segmentation network with an encoder-decoder architecture is constructed. Multimodal boundary features and preliminary fusion features are fused by channel splicing to form an input that combines global semantics and local boundary information. The encoder captures subtle differences and spatial correlations in tumor boundaries through 4 levels of downsampling and ReLU activation; The decoder upsamples step by step and combines deep and shallow features with skip connections. It optimizes the feature representation through convolution, gradually restores the resolution, and outputs a boundary probability map with the same size as the original image. The pixel values ​​of the probability map are mapped to the [0,1] interval by using the sigmoid activation function, and the preliminary fine segmentation boundary is obtained by using a threshold.

[0031] Specifically, a boundary refinement segmentation network with an encoder-decoder architecture is constructed. The encoder adopts a deep residual network structure, which performs deep mining of the input fused features through downsampling at four levels. Each downsampling level adds an edge attention module to accurately capture the gray-level gradient and morphological change information of the tumor boundary. The decoder uses transposed convolution to achieve feature upsampling. The features of each upsampling level and the corresponding level of the encoder are concatenated through skip connections to supplement the boundary details lost during downsampling. The network output layer uses the sigmoid activation function to output a probability map of the tumor boundary. The probability value represents the confidence that the corresponding pixel is the tumor boundary. During training, the weighted sum of cross-entropy loss and Dice loss is used as the loss function. An adaptive momentum optimization algorithm is used to iteratively update the network parameters until the loss function value converges and the segmentation accuracy of the validation set is stable. Multiple rounds of precise calibration are performed on the boundary probability map output by the network. Adaptive threshold segmentation is used to convert the probability map into a binary boundary image. The threshold is dynamically adjusted according to the distribution characteristics of the probability map to ensure that high-confidence boundary points are completely preserved. An active contour model is used to smooth and optimize the binary boundary image. The contour curve is adjusted iteratively by minimizing the energy function to eliminate jagged artifacts at the boundary and make the contour more closely match the actual edge morphology of the tumor. Anatomical constraints from the benchmark CT image are introduced. If there is an unreasonable overlap between the optimized boundary and the surrounding key anatomical structures, the overlapping area is corrected by local contour fine-tuning.

[0032] See Figure 8 As shown, the initial fine segmentation boundary is smoothed, and the Dice similarity coefficient, intersection-over-union ratio, and average boundary distance of the optimized segmentation boundary are calculated. By comparing with a preset threshold, the final chest tumor boundary segmentation result is output, or the feature fusion weights are adjusted to re-fuse boundary features and perform fine segmentation. Specifically, this includes: Using the preliminary fine segmentation boundary as the initial contour, an active contour model is constructed. The initial contour is input into the active contour model, and the contour curve parameters are iteratively adjusted. During the iteration process, the contour curve moves along the direction of the gray-level gradient decrease at the tumor edge until the contour position is stable, and the smoothed and optimized tumor boundary is obtained. The Dice similarity coefficient, crossover ratio, and average distance between the optimized boundary and the gold standard are calculated. By comparing with the threshold, if all criteria are met, the final chest tumor boundary segmentation result is output. If the criteria are not met, the feature fusion weights are adjusted and boundary feature fusion and fine segmentation are performed again.

[0033] Specifically, the calibrated boundary contours undergo false boundary identification and removal. By calculating the curvature, length, and continuity characteristics of the boundary segments, false boundaries that do not conform to the morphological patterns of tumor boundaries are screened out. Morphological dilation operations are used to fill the tiny gaps in the boundary contours. The size of the dilated structural elements is dynamically adjusted according to the gap width to avoid excessive dilation that could lead to boundary shift. Erosion operations are then performed on the filled boundaries to restore their original positions and ensure boundary accuracy. Simultaneously, connected component analysis is used to retain the largest connected boundary regions and remove isolated, small boundary noise points.

[0034] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0035] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0036] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for chest tumor boundary recognition and segmentation based on multimodal imaging, characterized in that, include: Acquire multimodal chest imaging data, including computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, and perform preprocessing operations on the chest imaging data of each modality; Using the preprocessed CT images as the reference images, the preprocessed images of other modalities are spatially registered with the reference images to obtain a spatially aligned multimodal chest image set. A dual-branch feature extraction network based on an attention mechanism is constructed. Each branch adopts an improved U-Net network structure. In each downsampling stage of the encoder, complementary features are fused through intermodal correlation weights. The decoder recovers and fuses the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Based on the preliminary fusion features of multimodal data, an adaptive threshold segmentation algorithm is used for coarse segmentation of the tumor region. By optimizing the segmentation boundary, a continuous and complete candidate tumor boundary contour is obtained. Based on the local images of each modality boundary within the candidate tumor boundary contour, features are extracted from the local image blocks of each modality boundary. The features of each modality boundary local image are then fused using a weighted average fusion algorithm to obtain enhanced multimodal boundary features. The enhanced multimodal boundary features are concatenated with the preliminary multimodal fusion features. A boundary refinement segmentation network with an encoder-decoder structure is used to output a probability map of the tumor boundary. The tumor is then identified by a threshold to obtain the preliminary fine segmentation boundary. The initial fine segmentation boundary is smoothed and adjusted. The Dice similarity coefficient, crossover ratio, and average distance of the optimized segmentation boundary are calculated. By comparing with the preset threshold, the final chest tumor boundary segmentation result is output or the feature fusion weights are adjusted to re-fuse boundary features and perform fine segmentation.

2. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The acquisition of multimodal chest imaging data, including computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography (PET) images, and the preprocessing of each modality of chest imaging data specifically include: Preprocessing operations were performed on the chest image data for each modality. These preprocessing operations included noise suppression, grayscale normalization, and image resolution unification. Noise suppression employs an adaptive median filtering algorithm, which dynamically adjusts the size of the filtering window based on the noise density in different regions of the image to remove noise. Gray-level normalization maps the gray values ​​of each modality image to the [0, 255] range; Image resolution is unified by adjusting the images of each modality to the same spatial resolution using bilinear interpolation.

3. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The step of using the preprocessed CT image as the reference image and spatially registering the preprocessed images of other modalities with the reference image to obtain a spatially aligned multimodal chest image set specifically includes: The preprocessed CT image is selected as the reference image, and the spatial coordinate system parameters of the reference image are extracted. A rigid registration method based on normalized mutual information is adopted to calculate the mutual information value between the image to be registered and the reference image. An optimization function with the goal of maximizing mutual information is constructed. By iteratively adjusting the translation, rotation and scaling parameters, preliminary rigid registration results are obtained. The thin-plate spline interpolation algorithm is used to perform elastic registration on the preliminary rigid registration results, and the residual local nonlinear deformation in the preliminary rigid registration results is corrected to obtain a spatially aligned multimodal chest image set.

4. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The construction of a dual-branch feature extraction network based on an attention mechanism, with each branch employing an improved U-Net network structure, involves fusing complementary features through intermodal correlation weights at each downsampling stage of the encoder, and the decoder recovering and fusing the enhanced multi-scale features through upsampling to obtain a preliminary multimodal fused feature map. Specifically, this includes: A dual-branch feature extraction network based on an attention mechanism is constructed. An improved U-Net structure is used within each branch, and the spatially aligned images of each modality are input into the corresponding feature extraction branch. Each branch encoder performs multi-scale feature extraction through 4 levels of downsampling. Each level of downsampling uses 2 layers of convolution to extract hierarchical features from low-dimensional texture to high-dimensional semantics. After each downsampling stage, a cross-modal attention module is embedded to calculate the correlation weights between the current modality and other modal features. Complementary features are then fused according to the weights to generate an enhanced feature map. The decoder recovers the feature map resolution through four levels of upsampling. Each level of upsampling is combined with the enhanced feature map of the encoder level for skip connections to obtain a preliminary multimodal fused feature map.

5. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The process of using an adaptive threshold segmentation algorithm for coarse segmentation of the tumor region based on preliminary multimodal fusion features, and obtaining continuous and complete tumor candidate boundary contours through optimization of the segmentation boundaries, specifically includes: Based on the preliminary fusion features of multimodal features, all gray levels of the feature map are traversed, the inter-class variance of the foreground and background regions corresponding to each gray level is calculated, and the gray level with the largest inter-class variance is selected as the final segmentation threshold. The multimodal preliminary fusion feature map is binarized. Pixels with gray values ​​greater than the segmentation threshold are marked as foreground pixels, and pixels with gray values ​​less than or equal to the segmentation threshold are marked as background pixels, thus obtaining a coarse segmentation binary map containing the tumor candidate region and the background. Edges are extracted from the coarsely segmented binary image using the Canny edge detection algorithm to obtain candidate tumor boundaries. Noise from the candidate boundaries is removed by morphological dilation and erosion operations to obtain continuous and complete candidate tumor boundary contours.

6. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The process involves extracting features from local boundary images of each modality within the tumor candidate boundary contour range, fusing the features of each modality's local boundary image using a weighted average fusion algorithm, and obtaining enhanced multimodal boundary features. Specifically, this includes: Based on the candidate tumor boundary contour, a set of local boundary images is constructed on each spatially aligned modal image. Features are extracted from the boundary local images of each modality. The gray-level gradient features of the boundary are extracted from the CT image blocks, the texture features are extracted from the MRI image blocks, and the metabolic activity distribution features are extracted from the PET image blocks. Shape features are extracted from all modal images to form a multi-dimensional modality-specific boundary feature set; A feature fusion matrix is ​​constructed, and a weighted average fusion strategy is used to fuse the modality-specific boundary features to obtain the initial fused features; The initial fused features are batch normalized to obtain enhanced multimodal boundary features.

7. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The process of concatenating enhanced multimodal boundary features with preliminary multimodal fusion features, using an encoder-decoder structured boundary refinement segmentation network to output a probability map of the tumor boundary, and obtaining preliminary fine segmentation boundaries by determining tumors through thresholding specifically includes: A boundary refinement segmentation network with an encoder-decoder architecture is constructed. Multimodal boundary features and preliminary fusion features are fused by channel splicing to form an input that combines global semantics and local boundary information. The encoder captures subtle differences and spatial correlations in tumor boundaries through 4 levels of downsampling and ReLU activation; The decoder upsamples step by step and combines deep and shallow features with skip connections. It optimizes the feature representation through convolution, gradually restores the resolution, and outputs a boundary probability map with the same size as the original image. The pixel values ​​of the probability map are mapped to the [0,1] interval by using the sigmoid activation function, and the preliminary fine segmentation boundary is obtained by using a threshold.

8. The method for chest tumor boundary recognition and segmentation based on multimodal images according to claim 1, characterized in that, The process of smoothing the initial fine segmentation boundary, calculating the Dice similarity coefficient, intersection-over-union ratio, and average boundary distance of the optimized segmentation boundary, comparing it with a preset threshold, and outputting the final chest tumor boundary segmentation result or adjusting the feature fusion weights to re-fuse boundary features and perform fine segmentation specifically includes: Using the preliminary fine segmentation boundary as the initial contour, an active contour model is constructed. The initial contour is input into the active contour model, and the contour curve parameters are iteratively adjusted. During the iteration process, the contour curve moves along the direction of the gray-level gradient decrease at the tumor edge until the contour position is stable, and the smoothed and optimized tumor boundary is obtained. The Dice similarity coefficient, crossover ratio, and average distance between the optimized boundary and the gold standard are calculated. By comparing with the threshold, if all criteria are met, the final chest tumor boundary segmentation result is output. If the criteria are not met, the feature fusion weights are adjusted and boundary feature fusion and fine segmentation are performed again.

9. A chest tumor boundary recognition and segmentation system based on multimodal images, used to implement the chest tumor boundary recognition and segmentation method based on multimodal images as described in any one of claims 1-8, characterized in that, include: Data acquisition and preprocessing module: Acquires multimodal chest images from CT, MRI, and PET, and performs noise suppression, grayscale normalization, and resolution unification preprocessing; Multimodal registration module: Using CT images as a reference, spatial alignment of multimodal images is achieved through rigid registration and flexible registration; Dual-branch feature extraction module: adopts an improved U-Net structure, combines cross-modal attention mechanism to fuse multimodal features, and generates a preliminary fused feature map; Tumor coarse segmentation module: Based on adaptive threshold segmentation and morphological processing, it extracts continuous and complete candidate tumor boundary contours; Boundary feature enhancement module: Extracts local features of each modality boundary and generates enhanced multimodal boundary features through weighted fusion; Boundary Refinement Segmentation Module: The encoder-decoder network fuses global and local features, outputs a probability map, and generates preliminary fine segmentation boundaries. Post-processing and evaluation module: performs smoothing optimization on the segmentation results, calculates evaluation indicators, dynamically adjusts parameters, or outputs the final boundary segmentation results; Processor: The processor is used to handle the calculation process of each formula and the construction calculation process of each model.

Images