A medical image tumor segmentation method involving a cross-modal attention mechanism

Abstract

The application discloses a medical image tumor segmentation method related to a cross-modal attention mechanism, and comprises the following steps: obtaining a positron emission computed tomography (PET) image and a computed tomography (CT) image, and transversely scaling the PET image and the CT image into a same-resolution image data pair; inputting the image data pair into a trained cross-modal image segmentation model to generate a target segmentation image; wherein the cross-modal image segmentation model comprises an image feature extraction module, an image feature fusion module and a cross-modal semantic enhancement module, the cross-modal semantic enhancement module comprises an extended encoder group and a semantic fusion network based on an aggregated attention unit, and the cross-modal image segmentation model is trained through historical positron emission computed tomography (PET) images and computed tomography (CT) images. The medical image tumor segmentation method related to the cross-modal attention mechanism can realize accurate segmentation of cross-modal medical images.

Description

TECHNICAL FIELD

[0001] The application relates to a medical image tumor segmentation method involving a cross-modal attention mechanism and belongs to the technical field of cross-modal image segmentation. BACKGROUND

[0002] Diffuse large B-cell lymphoma is a type of lymphoma with high incidence. In medical images, especially positron emission tomography-Computed Tomography (PET-CT) images, they are usually the best choice for clinical analysis of lymphoma structures and have been successfully applied in computer-aided diagnosis and medical treatment.

[0003] PET is a cell-based molecular imaging technique that uses tumor tracers to provide full-spectrum tomographic imaging of the human body from the transverse, coronal and sagittal planes. It can dynamically show the treatment response by reflecting the metabolic status of organ tissues, and has high specificity and sensitivity. Compared with other medical images, PET images have strong spatial resolution for lesion areas, especially for malignant tumor tissues. However, due to the characteristics of PET technology and the influence of image reconstruction, the resolution and contrast of PET images are not high, and there are serious image artifacts, partial volume effects and halo effects. The above problems will cause image non-uniformity and distortion, resulting in a large error in the actual position and boundary of the lesion, thereby seriously affecting the accuracy and precision in the process of diagnosis and treatment. CT imaging uses the different absorption capacities of different tissues in the human body to X-rays to obtain information about the lesion and the surrounding structure. Since the CT device acquires images in a short time and is less affected by external factors, the positioning image generally does not distort and is close to the actual shape of the tumor, which is a widely used technology. Although CT images can better reflect the anatomy of organ tissues, accurately locate lesions and show lesion morphology, it is difficult to judge the position and boundary of the lesion when there is no significant difference in the density of the infiltrated tissue and the normal tissue. Combining PET images and CT images can provide more information about the patient's lesion and surrounding area, and the different image information has certain complementarity. Among them, the CT image can correct the attenuation of the PET image, solve the problem of unclear anatomical structure of the PET image, and also can perform positioning analysis on the PET image reflecting the physiological, metabolic and functional characteristics of the organ; and the qualitative and quantitative analysis of the PET image can provide valuable information on function and metabolism. Therefore, the multi-modal lymphoma segmentation combined with PET-CT has important value in radiosurgery and radiotherapy planning.

[0004] In the processing of lymphoma images, single-modal technology only focuses on single image feature extraction and processing, ignoring the complementarity of features between cross-modal images. The single modality limits the development of medical image segmentation technology. Combined with the rapid development of neural networks, cross-modal image processing technology has both the convenience of end-to-end and high accuracy. Learning-based methods, especially those based on convolutional neural networks (CNN), have rapidly developed in medical image analysis over the past decade. CNN was originally proposed to complete image-level classification tasks. The intuitive application of CNN to image segmentation is a pixel-level classification task, which classifies each pixel through a sliding window method (R-CNN). Later, U-Net was specifically proposed for biomedical image segmentation, which is currently the baseline network for various medical image segmentation tasks and the inspiration for many subsequent works such as attention-UNet, DenseUNet, etc. However, traditional single-modal convolutional neural networks are difficult to achieve significant improvements in segmentation accuracy due to the single nature of the features. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a medical image tumor segmentation method involving a cross-modal attention mechanism, which can achieve accurate segmentation of cross-modal medical images.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0007] A medical image tumor segmentation method involving a cross-modal attention mechanism, comprising the following steps:

[0008] Obtaining positron emission computed tomography (PET) images and computed tomography (CT) images, and scaling the PET images and CT images transversely to the same resolution of image data pairs;

[0009] Inputting the image data pairs into a trained cross-modal image segmentation model to generate a target segmentation image;

[0010] The cross-modal image segmentation model includes an image feature extraction module, an image feature fusion module, and a cross-modal semantic enhancement module, and the cross-modal semantic enhancement module includes a set of extended encoders and a semantic fusion network based on an aggregated attention unit.

[0011] The training method of the cross-modal image segmentation model includes:

[0012] acquire historical positron emission computed tomography (PET) images and computed tomography (CT) images, scale the PET images and the CT images transversely into image data pairs with the same resolution, the image data in each image data pair is labeled with tumor segmentation information, and then divide the image data pairs into a training set, a validation set and a test set;

[0013] extract features from the single-channel image data pairs in the training set through an image feature extraction module to obtain PET image features and CT image features in a single modality;

[0014] extract hierarchical features from the PET images and the CT images in the training set through an extended encoder respectively and obtain PET-CT multi-modal fusion feature blocks through an image feature fusion module to output attention feature maps in the PET modality;

[0015] input the attention feature maps in the PET modality into a semantic fusion network together with the CT image features through standard back propagation to perform first semantic fusion to obtain attention feature maps in a fusion modality;

[0016] input the attention feature maps in the fusion modality and the extracted PET-CT multi-modal fusion feature blocks into the semantic fusion network to perform up-sampling and second semantic fusion, recover the feature structure to obtain final attention fusion feature maps;

[0017] perform similarity calculation between the final attention fusion feature maps and the attention feature maps in the PET modality and perform sigmoid function operation to obtain a weight vector of each segmentation feature corresponding to the current image feature;

[0018] determine each segmentation feature based on the weight vector to obtain a target segmentation image most similar to the current image feature;

[0019] calculate a joint loss of the image features through the image feature extraction module, the joint loss includes a cross-entropy loss of comparing the target segmentation image with the tumor segmentation label information and a dice value loss, and the calculated joint loss is used to update parameters of the cross-modality image segmentation model, and when the parameters of the model converge, the optimal cross-modality image segmentation model and the parameters thereof are saved.

[0020] The image feature extraction module includes a convolutional neural network (CNN), the CNN includes a plurality of convolutional layers, and a pooling layer is connected behind each two convolutional layers, historical positron emission computed tomography (PET) images and computed tomography (CT) images are input into each layer of the CNN to obtain PET image features v′ i (p) and CT image features v′ i (c)wherein i represents the number of layers, i = 1, 2, 3, 4.

[0021] The extended encoder comprises a convolutional neural network (CNN) comprising a plurality of convolutional layers, one pooling layer being connected after each two convolutional layers, and the image feature fusion module comprises a feature fusion structure f p (.) composed of two convolutional layers and one pooling layer, the convolutional neural network (CNN) extracts PET modality information v'4 (p) and CT modality information v'4 (c) from the PET image and the CT image respectively, and inputs them to the feature fusion structure f p (.) for feature splicing and fusion to determine a fused feature layer, and the output obtained is a PET-CT multi-modal fusion feature block v'4 (pc) .

[0022] The attention aggregation unit comprises a skip connector f s (.) and a back propagator, the PET-CT multi-modal fusion feature block v'4 (pc) is input to the skip connector f s (.) and connected with the CT modality information v'4 (c) , multi-layer feature re-extraction and aggregation are performed, and finally a sigmoid layer is passed through to determine the output attention feature map, the back propagator uses the attention feature structure to back propagate to the feature extraction stage, performs feature resampling, optimizes the features using the attention feature map, and determines the aggregated attention map wherein is the image feature of the image data at the i-th down-sampling stage, and N is the total number of layers.

[0023] The joint loss is calculated by a cross-entropy loss function, and the joint loss is used to update the parameters of the cross-modality image segmentation model by using a gradient descent algorithm, and the cross-entropy loss is expressed as: wherein y represents the label of the tumor segmentation label information, the positive class is 1, and the negative class is 0, and y represents the probability that the output image is predicted as a positive class. The determination of whether the cross-modality image segmentation model is trained includes: determining a set similarity measure function dice coefficient, if the dice coefficient is within a dice coefficient threshold range, it is determined that the cross-modality image segmentation model is trained, wherein the expression of the Dice coefficient is expressed as: Dice(A, B) = 2*(A∩B)\(A∪B), wherein A is the number of tumor pixels of the network output image, B is the number of tumor pixels of the tumor segmentation label information, A∩B is the number of pixels of the tumor segmentation label information label which is a positive class and the network output image which is also predicted as a positive class, and A∪B is the total number of pixels of the tumor regions of the tumor segmentation label information and the network output image.

[0024] A medical image tumor segmentation device involving a cross-modal attention mechanism, comprising:

[0025] An image acquisition module for acquiring a positron emission computed tomography (PET) image and a computed tomography (CT) image;

[0026] An image scaling module for scaling the PET image and the CT image laterally into a same-resolution image data pair;

[0027] A cross-modal image segmentation model, the cross-modal image segmentation model comprising an image feature extraction module, an image feature fusion module, and a cross-modal semantic enhancement module, the cross-modal semantic enhancement module comprising a set of extended encoders and a semantic fusion network based on an aggregated attention unit;

[0028] The training method of the cross-modal image segmentation model comprises:

[0029] Acquiring historical positron emission computed tomography (PET) images and computed tomography (CT) images, scaling the PET images and the CT images laterally into a same-resolution image data pair, each image data in each set of image data pairs having tumor segmentation label information, and then dividing the image data pairs into a training set, a validation set, and a test set;

[0030] Extracting features from single-channel image data pairs in the training set through the image feature extraction module to obtain PET image features and CT image features in a single modality;

[0031] Extracting hierarchical features from the PET images and the CT images in the training set through the extended encoders respectively and fusing through the image feature fusion module to obtain PET-CT multi-modal fusion feature blocks, and outputting attention feature maps in the PET modality;

[0032] Inputting the attention feature maps in the PET modality through standard backpropagation together with the CT image features into the semantic fusion network for first semantic fusion to obtain attention feature maps in a fusion modality;

[0033] Inputting the attention feature maps in the fusion modality and the extracted PET-CT multi-modal fusion feature blocks into the semantic fusion network for up-sampling and second semantic fusion to recover the feature structure and obtain final attention fusion feature maps;

[0034] Calculating the similarity between the final attention fusion feature maps and the attention feature maps in the PET modality and performing a sigmoid function operation to obtain a weight vector of each segmentation feature corresponding to the current image feature;

[0035] Based on the weight vector, each segmentation feature is judged to obtain a target segmentation image most similar to the current image feature.

[0036] The image feature extraction module calculates the joint loss of image features, which includes the cross-entropy loss from comparing the target segmented image with the tumor segmentation label information, and the Dice value loss. The calculated joint loss is used to update the parameters of the cross-modal image segmentation model. When the model parameters converge, the optimal cross-modal image segmentation model and its parameters are saved.

[0037] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned medical image tumor segmentation method involving a cross-modal attention mechanism.

[0038] The beneficial effects of this invention are as follows: This invention provides a medical image tumor segmentation method involving a cross-modal attention mechanism, which can mine the semantic correlation between PET and CT modalities, construct a cross-modal common subspace, thereby bridging the gap between heterogeneous data. At the same time, it designs an attention interaction network, enabling the generative model to train data from both modalities simultaneously and improve the quality and accuracy of segmented images. It can realize cross-modal automatic segmentation of tumor medical images, effectively solving the problems of low efficiency and inconsistent annotation quality of manual segmentation, and improving the efficiency of medical image segmentation. Attached Figure Description

[0039] Figure 1 These are the PET images and CT images involved in this invention;

[0040] Figure 2 This is a flowchart of a medical image tumor segmentation method involving a cross-modal attention mechanism according to the present invention;

[0041] Figure 3 This is a schematic diagram of the cross-modal image segmentation model structure in this invention;

[0042] Figure 4 This is a schematic diagram of the attention network structure in this invention;

[0043] Figure 5 This is a comparison chart showing the performance of the medical image tumor segmentation method of the present invention with that of existing methods in generating images. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0045] like Figure 2 As shown, this invention discloses a medical image tumor segmentation method involving a cross-modal attention mechanism, comprising the following steps:

[0046] Step one, obtain historical positron emission computed tomography (PET) images and computed tomography (CT) images, specifically as Figure 1 As described above, the two modal images include images of high-grade tumor patients and images of low-grade tumor patients; the images of the two modalities are scaled transversely to the same resolution image data pairs, the PET image intensity is converted to SUV, and the CT image is converted to Hounsfield unit; the image pairs of the two modalities are divided into training set, validation set and test set. The image data in each group of image data pairs has tumor segmentation label information.

[0047] Step two, establish a cross-modality image segmentation model based on medical images, the structure of the cross-modality image segmentation model is shown in Fig. 3, including an image feature extraction module, an image feature fusion module, and a cross-modality semantic enhancement module based on an attention aggregation unit.

[0048] The image feature extraction module includes a convolutional neural network (CNN) for feature extraction of single-channel image data pairs in the training set, obtaining PET image features and CT image features of single modality. The image data feature extraction includes the following steps: down-sampling stage, any group of historical positron emission computed tomography (PET) images and computed tomography (CT) images in the training set, such as (2-1) image data P, C, are input to the layers of the convolutional neural network (CNN), and the image features v′ i (c) , v′ i (c) , i = 1, 2, 3, 4, the CNN includes multiple convolutional layers, and a pooling layer is connected after every two convolutional layers; the multi-level feature re-extraction neural network model is a two-input, two-output neural network model, and the output end includes PET and CT image features.

[0049] The cross-modality semantic enhancement module includes an extended encoder and a semantic fusion network based on an aggregation attention unit, and the extended encoder extracts hierarchical features of the PET image and the CT image in the training set. Specifically, the extended encoder includes a CNN neural network, and the features of the modalities are extracted by the CNN neural network structure in the down-sampling stage. Then the PET-CT multi-modality fusion feature block is obtained by fusion, wherein the image data is fused by the image feature fusion module, including the following steps: in the CNN network in the down-sampling stage, the two modal information v′4 (p) , v′4 (c) of PET and CT are extracted, and the features are spliced and fused through a feature fusion structure f p (.) to determine the fused feature layer, and the output is v′4 (pc)The fusion structure can be specifically composed of two layers of convolution layers connected with one layer of pooling layers.

[0050] The attention network structure in the application is shown in the schematic diagram as Figure 4 The specific process of outputting the attention feature map of the PET modality is as follows: in the up-sampling stage, the extracted modality feature is input into the skip connector f s (.), and the multi-layer feature v′ i (c) , i = 1, 2, 3, 4, is connected, the multi-layer feature is re-extracted and aggregated, and finally a sigmoid layer is passed to determine the output attention feature map M. The sigmoid layer normalizes the up-sampled feature H:

[0051] M = sigmoid (H)

[0052] The semantic fusion network based on the PET attention mechanism matches the CT feature and the extracted v (f) Each is used as a query vector to filter out the generated fusion feature belonging to the same category and forms a fusion feature-attention pair, and then the corresponding synthetic attention feature of v (f) is The specific process is as follows:

[0053] A multi-layer back propagator is constructed, the attention feature structure is back propagated to the feature extraction stage, the cross-modal feature is resampled, the feature is optimized by using the attention map, and the fused attention feature map is determined

[0054] Then, the PET attention map is input into the semantic fusion network based on the attention aggregation unit together with the CT image feature of the image feature extraction module through the standard back propagation to perform the first semantic fusion, and the attention feature map of the fusion modality is obtained; the attention map of the fusion modality is up-sampled together with the extracted PET-CT multi-modal fusion feature block to perform the second semantic fusion, the feature structure is restored to obtain the final attention fusion feature map; the similarity between the final attention fusion feature map and the PET attention map is calculated and the operation of the sigmoid function is performed to obtain the weight vector of each segmentation feature corresponding to the current image feature, and then the weight vector is used to judge each segmentation feature result to obtain the segmentation result most similar to the current image feature.

[0055] Step three, training the cross-modal medical image segmentation model on the training set. Any set of historical positron emission computed tomography (PET) images and computed tomography (CT) images in the training set is input into the cross-modal medical segmentation network model to determine the segmented tumor image, which also includes obtaining the tumor segmentation label information corresponding to the tumor mask.

[0056] The segmented tumor image using the cross-modal method and the tumor segmentation label information corresponding to the historical tumor image in the training set are compared, and the joint loss of the image features is calculated according to the image feature extraction module. The joint loss includes the cross-entropy loss of the cross-modal segmentation result compared with the true value image, and the loss of the dice value. The loss is calculated by the cross-entropy loss function, and the gradient descent algorithm is used to continue training the cross-modal medical segmentation neural network model. The calculated cross-entropy loss is used to update the parameters of the cross-modal image segmentation model.

[0057] The cross-entropy loss can be expressed as: Where y represents the label of the tumor segmentation label information, the positive class is 1, and the negative class is 0, represents the probability of the output image predicting the positive class.

[0058] After the cross-modal medical segmentation neural network model is continuously trained by the gradient descent algorithm, the set similarity measure function dice coefficient is determined. If the dice coefficient is within the threshold range of the dice coefficient, the cross-modal supervised learning medical segmentation neural network model is determined to be trained. The expression of the dice coefficient is expressed as: Dice(A, B) = 2*(A∩B) / (A∪B), where A is the number of tumor pixels of the network output image, B is the number of tumor pixels of the tumor segmentation label information, A∩B is the number of pixels of the tumor segmentation label information label that is positive class and the network output image that is also predicted to be positive class, and A∪B is the total number of pixels of the tumor region of the tumor segmentation label information and the network output image.

[0059] Step four, after the training is completed, the cross-modal medical images of the test set are input into the trained cross-modal image segmentation model, and the output is the target segmentation image.

[0060] The application also discloses a medical image tumor segmentation device related to a cross-modal attention mechanism, comprising:

[0061] An image acquisition module is used to acquire positron emission computed tomography (PET) images and computed tomography (CT) images.

[0062] An image scaling module is used to scale the PET images and CT images horizontally into image data pairs of the same resolution.

[0063] The cross-modal image segmentation model comprises an image feature extraction module, an image feature fusion module and a cross-modal semantic enhancement module, the cross-modal semantic enhancement module comprises a set of extended encoders and an aggregated attention unit-based semantic fusion network.

[0064] The training method of the cross-modal image segmentation model comprises:

[0065] Obtain historical positron emission computed tomography (PET) images and computed tomography (CT) images, scale the PET images and the CT images transversely into image data pairs with the same resolution, the image data in each image data pair is provided with tumor segmentation label information, and then divide the image data pairs into a training set, a validation set and a test set;

[0066] Extract features of the single-channel image data pairs in the training set through the image feature extraction module to obtain PET image features and CT image features in a single modality;

[0067] Extract features of the PET images and the CT images in the training set through the extended encoders respectively, fuse the PET-CT multi-modal fusion feature blocks through the image feature fusion module to obtain attention feature maps in the PET modality;

[0068] Input the attention feature maps in the PET modality into the semantic fusion network together with the CT image features through standard back propagation to perform first semantic fusion and obtain attention feature maps in a fusion modality;

[0069] Input the attention feature maps in the fusion modality and the extracted PET-CT multi-modal fusion feature blocks into the semantic fusion network to perform second semantic fusion through upsampling, restore the feature structure to obtain final attention fusion feature maps;

[0070] Calculate the similarity between the final attention fusion feature maps and the attention feature maps in the PET modality, and perform a sigmoid function operation to obtain a weight vector of each segmentation feature corresponding to the current image feature;

[0071] Determine each segmentation feature based on the weight vector to obtain a target segmentation image most similar to the current image feature;

[0072] Calculate the joint loss of the image features through the image feature extraction module, the joint loss comprises a cross-entropy loss obtained by comparing the target segmentation image with the tumor segmentation label information, and a dice value loss, use the calculated joint loss to update the parameters of the cross-modal image segmentation model, and save the optimal cross-modal image segmentation model and the parameters thereof when the parameters of the model converge.

[0073] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the medical image tumor segmentation method related to the cross-modal attention mechanism.

[0074] The method of the application is compared with existing cross-modal image generation methods.

[0075] (1) Simulation conditions

[0076] The application is simulated on an Intel(R) Core(TM) i3-2120 3.30 GHZ central processing unit, 6G memory and a windows 10 operating system by using Python software.

[0077] (2) Simulation content

[0078] Existing method one: a traditional UNet network is used to segment a single modality PET image.

[0079] Existing method two: a traditional UNet network is used to segment a single modality CT image.

[0080] Existing method three: an Attention-UNet added with an attention mechanism is used to segment a single modality PET image.

[0081] Existing method four: an Attention-UNet added with an attention mechanism is used to segment a single modality CT image.

[0082] Existing method five: a UNet network is used to segment a pixel-level fusion PET-CT image.

[0083] Existing method six: a single modality PET is used as supervision information to segment a CT image.

[0084] (3) Simulation results

[0085] In the simulation experiment, the Dice indexes of the above six methods and the application are respectively given under a private PET-CT data set, and the greater the index is, the higher the image category accuracy is, and the indexes are specifically shown in Table 1.

[0086] Table 1: indexes of the haptic generated image under the PET-CT data set

[0087]

[0088] In combination Figure 5The observation of the generation result and the evaluation result of table 1 shows that the segmentation image generated by the method of the application has the highest accuracy. This shows that the model of the application is superior to other models, and for the same generation target, the model of the application has the highest accuracy in semantics, and also verifies the importance of the cross-modal matching network in the model.

[0089] In summary, the medical image segmentation method involving the cross-modal attention mechanism disclosed in the application is mainly used for the segmentation of PET-CT medical images, and a cross-modal interaction network is constructed by combining the attention mechanism, so as to solve the problem of low segmentation accuracy caused by the limitation of the traditional generation model using only a single mode, and Dice commonly used as a performance evaluation index. The application not only considers the characteristics of the tumor on the sample single mode, but also considers the complementarity of the characteristics between different modes, and provides a guarantee for the accuracy of the corresponding segmentation image generated by the multi-modal PET-CT. The implementation steps are as follows: (1) collect the cross-modal PET-CT data set (2) set the model (3) train the model, including feature extraction on the PET image and the CT image sample; extract the attention map based on the PET image; construct the attention interaction network for the PET attention map and the CT image feature, map the attention feature of the PET to the cross-modal public subspace as the supervision information and the feature information of the CT, and strongly match the public representation of each mode according to the category and distribution; utilize the loss function of the similarity between the modes and the intra-modal to perform cross-distribution alignment while maintaining the category consistency of each mode; train the network by using the alternating iteration method; (4) segmentation image generation. The application utilizes the attention mechanism network, realizes the strong matching and semantic fusion of the PET-CT modal data according to the semantic correlation between different modal data, and thus the quality of the corresponding segmentation image generated is better and the accuracy is higher, and the application can be used for multi-modal services such as medical image segmentation, and improves the efficiency of image segmentation.

[0090] The above only describes the preferred embodiments of the application, and it should be noted that for ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A medical image tumor segmentation method involving a cross-modal attention mechanism, characterized in that: The method comprises the following steps: Obtaining positron emission computed tomography (PET) images and computed tomography (CT) images; Scaling the PET images and the CT images into image data pairs with the same resolution; Inputting the image data pairs into a trained cross-modal image segmentation model to generate a target segmentation image; The cross-modal image segmentation model comprises an image feature extraction module, an image feature fusion module and a cross-modal semantic enhancement module, and the cross-modal semantic enhancement module comprises an extended encoder and an aggregated attention unit-based semantic fusion network; The training method of the cross-modal image segmentation model comprises: Obtaining historical positron emission computed tomography (PET) images and computed tomography (CT) images, and scaling the PET images and the CT images into image data pairs with the same resolution, wherein the image data in each image data pair is labeled with tumor segmentation information, and the image data pairs are divided into a training set, a validation set and a test set; Extracting features of the single-channel image data pairs in the training set by the image feature extraction module to obtain PET image features and CT image features in a single modality; Extracting features of the PET images and the CT images in the training set by the extended encoder respectively, and fusing the PET-CT multi-modal fusion feature blocks obtained by the image feature fusion module to output an attention feature map in the PET modality; Inputting the attention feature map in the PET modality into the semantic fusion network together with the CT image features for first semantic fusion to obtain an attention feature map in a fusion modality; Inputting the attention feature map in the fusion modality and the extracted PET-CT multi-modal fusion feature blocks into the semantic fusion network for second semantic fusion by upsampling to restore the feature structure and obtain a final attention fusion feature map; Calculating the similarity between the final attention fusion feature map and the attention feature map in the PET modality, and performing a sigmoid function operation to obtain a weight vector of each segmentation feature corresponding to the current image feature; Judging each segmentation feature based on the weight vector to obtain a target segmentation image most similar to the current image feature; Calculating a joint loss of the image features by the image feature extraction module, wherein the joint loss comprises a cross-entropy loss obtained by comparing the target segmentation image with the tumor segmentation label information, and a dice value loss, and the calculated joint loss is used to update the parameters of the cross-modal image segmentation model, and when the parameters of the model converge, the optimal cross-modal image segmentation model and the parameters thereof are saved.

2. The method of claim 1, wherein the method is a medical image tumor segmentation method involving cross-modal attention mechanism. The image feature extraction module comprises a convolutional neural network (CNN), the convolutional neural network (CNN) comprises a plurality of convolutional layers, and a pooling layer is connected behind each two convolutional layers; historical posit emission computed tomography (PET) images and computed tomography (CT) images are input into each layer of the convolutional neural network (CNN), and PET image features and CT image features are respectively obtained, wherein i represents a layer number, i = 1, 2, 3, 4.

3. The method of claim 2, wherein the method further comprises: determining a cross-modality attention mechanism based on the cross-modality attention mechanism. The extended encoder includes a convolutional neural network (CNN), which comprises multiple convolutional layers, with a pooling layer following every two convolutional layers. The image feature fusion module includes a feature fusion structure f. p The convolutional neural network (CNN), consisting of two convolutional layers connected to a single pooling layer, extracts PET modality information from PET and CT images, respectively. and CT modal information And input into the feature fusion structure f p (.) Perform feature splicing and fusion to determine the fused feature layer. The output is the PET-CT multimodal fusion feature block.

4. The method of claim 3, wherein the method further comprises: The attention aggregation unit includes a skip connector f s (.) and a back propagator, a PET-CT multimodal fusion feature block Input to the skip connector f s (.) and the CT modality information are connected, multi-layer feature re-extraction and aggregation are performed, and finally a sigmoid layer is passed through to determine the output attention feature map. The back propagator uses the attention feature structure to back propagate to the feature extraction stage, performs feature resampling, optimizes the features using the attention feature map, and determines the aggregated attention map wherein, is the image feature of the image data of the i-th layer down-sampling stage, and N is the total number of layers.

5. The method of claim 1, wherein: The joint loss is calculated by a cross-entropy loss function, and the joint loss updates the parameters of the cross-modal image segmentation model by using a gradient descent algorithm. The cross-entropy loss is represented as: Where y represents the label of tumor segmentation label information, the positive class is 1, and the negative class is 0, And represents the probability that the output image is predicted as a positive class.

6. The method of claim 1, wherein: The determining whether the cross-modality image segmentation model is trained includes determining a set similarity measure function dice coefficient, and if the dice coefficient is within a dice coefficient threshold range, it is determined that the cross-modality image segmentation model is trained, wherein the expression of the dice coefficient is represented as: Dice (A, B) = 2 * (A ∩ B) / (A ∪ B), wherein A is the number of tumor pixels of a network output image, B is the number of tumor pixels of tumor segmentation label information, A ∩ B is the number of pixels of which the tumor segmentation label information label is a positive class and the network output image also predicts a positive class, and A ∪ B is the total number of pixels of the tumor regions of the tumor segmentation label information and the network output image.

7. An apparatus for medical image tumor segmentation involving cross-modal attention mechanism, characterized in that, The method comprises the steps of: An image acquisition module is configured to acquire a positron emission computed tomography (PET) image and a computed tomography (CT) image; An image scaling module is configured to scale the PET image and the CT image laterally into a same-resolution image data pair; A cross-modality image segmentation model comprises an image feature extraction module, an image feature fusion module, and a cross-modality semantic enhancement module, wherein the cross-modality semantic enhancement module comprises an extended encoder and an aggregated attention unit-based semantic fusion network; The training method of the cross-modality image segmentation model comprises: Historical positron emission computed tomography (PET) images and computed tomography (CT) images are acquired, and the PET images and the CT images are scaled laterally into a same-resolution image data pair, wherein each image data pair in each group of image data pairs is provided with tumor segmentation label information, and the image data pairs are divided into a training set, a validation set, and a test set; The image feature extraction module is used to extract features from single-channel image data pairs in the training set to obtain PET image features and CT image features in a single modality; The extended encoder is used to extract features from the PET image and the CT image in the training set in a hierarchical manner, and the image feature fusion module is used to fuse the PET-CT multi-modal fusion feature blocks to output an attention feature map in a PET modality; The attention feature map in the PET modality is input into the semantic fusion network together with the CT image features for first semantic fusion to obtain an attention feature map in a fusion modality; The attention feature map in the fusion modality and the extracted PET-CT multi-modal fusion feature blocks are input into the semantic fusion network for up-sampling and second semantic fusion to restore the feature structure and obtain a final attention fusion feature map; The final attention fusion feature map and the attention feature map in the PET modality are subjected to similarity calculation and sigmoid function operation to obtain a weight vector of each segmentation feature corresponding to the current image feature; Based on the weight vector, each segmentation feature is determined to obtain a target segmentation image most similar to the current image feature. The joint loss of the image features is calculated by the image feature extraction module, the joint loss includes a cross-entropy loss of comparing the target segmentation image with the tumor segmentation label information, and a dice value loss, and the calculated joint loss is used to update the parameters of the cross-modal image segmentation model, and when the parameters of the model converge, the optimal cross-modal image segmentation model and its parameters at this time are saved. 8.A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the medical image tumor segmentation method involving the cross-modal attention mechanism according to any one of claims 1-6.

Images