Nasopharyngeal carcinoma target region segmentation method based on multi-scale mixed sampling and feature fusion
By constructing a HSMA DAC UNet network model with multi-scale mixed sampling and feature fusion, the problems of insufficient automatic outline and insufficient morphological perception ability in nasopharyngeal carcinoma target segmentation are solved, and more accurate target segmentation and customization of assisted radiotherapy plans are achieved.
Patent Information
- Application Number
- CN202411008988.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The existing nasopharyngeal carcinoma target segmentation methods have problems such as insufficient automatic outline and poor customization of auxiliary radiotherapy plans. There are few applications of feature fusion and self-attention mechanisms, resulting in insufficient target boundary division and insufficient morphological perception ability.
Using a method based on multi-scale hybrid sampling and feature fusion, the Swin-UNet backbone network model is constructed, and the HSMA module and DAC module are introduced. Through parallel hybrid sampling, multi-scale spatial channel attention and dense fusion section cross-sectional self-attention mechanism, it enhances the Feature extraction and fusion capabilities form the HSMA DAC UNet network model.
It improves the accuracy and morphological perception of target segmentation of nasopharyngeal carcinoma, enhances the detailed perception of the edge area of the target area, overcomes the shortcomings of a single sampling mechanism, and achieves a stronger segmentation effect.
Smart Images

Figure CN118918119A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion. Background Art
[0002] Nasopharyngeal carcinoma is a malignant tumor that occurs in the nasopharynx and is a type of head and neck tumor. The treatment methods for nasopharyngeal carcinoma mainly include a combination of surgery, radiotherapy, and chemotherapy. Radiotherapy is currently the most commonly used and most effective method for treating nasopharyngeal carcinoma, but the size, location, and shape of nasopharyngeal carcinoma tumors vary among patients at different stages of treatment. Therefore, achieving accurate automatic delineation of nasopharyngeal carcinoma targets can effectively help doctors formulate radiotherapy plans, assist in surgery, and evaluate treatment effects.
[0003] Traditional NPC target segmentation is completely manually performed by doctors, which has many shortcomings, such as excessive consumption of doctors' energy, subjective differences and human errors. In the semi-automatic segmentation stage, researchers are committed to designing various image processing algorithms. Common semi-automatic segmentation methods include threshold-based, region-based, edge-based and atlas-based. Although these methods have improved efficiency to a certain extent, they are still limited by the manual selection of image features and the inadaptability to variation. Traditional machine learning methods are also widely used, but they still face problems such as dependence on feature selection and sensitivity to parameter adjustment.
[0004] In recent years, automatic semantic segmentation methods based on deep learning have gradually become the focus of NPC target segmentation research. However, in general, research in the field of NPC target segmentation is still not sufficient. There are problems such as inaccurate target boundary demarcation, weak perception of target location, morphology, and scale diversity, and difficulty in identifying small foreground targets. In terms of specific model implementation, encoding / decoding often adopts a single method, and the application research of attention mechanism is relatively scarce, especially the research on segmentation using Transformer or other self-attention is rare, and the research on feature fusion is relatively small. This shows that there is still a lot of room for exploration and potential to be tapped in the task of NPC target segmentation.
[0005] Therefore, the traditional NPC target segmentation method has many limitations and fails to automatically delineate NPC and assist in the customization of radiotherapy plans. Summary of the invention
[0006] The present invention provides a nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion, so as to solve the defect that the nasopharyngeal carcinoma target area segmentation method in the prior art cannot automatically outline nasopharyngeal carcinoma and assist in the customization of radiotherapy plan.
[0007] In a first aspect, the present invention provides a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion, comprising:
[0008] Collecting nasopharyngeal carcinoma three-dimensional target area image samples, and preprocessing the nasopharyngeal carcinoma three-dimensional target area image samples to obtain preprocessed image samples;
[0009] Build the Swin-UNet backbone network model;
[0010] Constructing a HSMA module, and adding the HSMA module to the Swin-UNet backbone network model to obtain a HSMAUNet network model;
[0011] Construct DAC and embed DAC into the HSMA UNet network model to obtain the HSMA DAC UNet segmentation model;
[0012] Using the preprocessed image samples to train the HSMADAC UNet segmentation model to obtain a nasopharyngeal carcinoma target area segmentation model;
[0013] The nasopharyngeal carcinoma three-dimensional target area image to be processed is input into the nasopharyngeal carcinoma target area segmentation model, and the nasopharyngeal carcinoma target area segmentation result image is output.
[0014] According to a nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, the nasopharyngeal carcinoma three-dimensional target area image sample is preprocessed to obtain a preprocessed image sample, including:
[0015] The DICOM format data and annotations of the nasopharyngeal carcinoma three-dimensional target image sample are converted into NIFTI format, the nasopharyngeal carcinoma three-dimensional target image sample is divided into CT image data DCM_images and annotation data DCM_labels and stored in memory, each segment of DCM_label data is read, and the contour reference metadata DCM_images[luid] is found according to the internal luid, and the contour coordinates (x, y, z) are converted into voxel image coordinates through the coordinates and voxel information of the metadata
[0016] Align the voxel information of the nasopharyngeal carcinoma three-dimensional target area image sample to make the intervals equally spaced in each direction, use a threshold operation to remove the boundary part, separate the body and the machine tool part through an erosion operation, select the body part through a closed expansion operation and a connected domain algorithm, fill the internal holes of the body part to obtain a body mask part, stack each layer of the body mask part to generate a three-dimensional body mask, extract the body part image data through the three-dimensional body mask, perform padding amplification on the body part image data, obtain a nasopharyngeal carcinoma three-dimensional image of a standard size and without interference data, normalize the ray intensity value of the nasopharyngeal carcinoma three-dimensional image, enhance the image contrast, and obtain three-dimensional image data for training;
[0017] Read the upper and lower boundaries A(x min ,y min ,z min ) and B(x max ,y max ,z max ), performing padding expansion or supplementing black edges according to the two boundaries, and adjusting the three-dimensional image data to a preset image size;
[0018] The resized three-dimensional image data is randomly cropped, rotated, and flipped, and Gaussian noise and Gamma correction are introduced. Label Smoothing is used to convert the category label into a probability distribution to obtain the preprocessed image sample.
[0019] According to a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, the contour coordinates (x, y, z) are converted into voxel image coordinates through the coordinates and voxel information of the metadata include:
[0020]
[0021] Among them, origin is the coordinate of the spatial origin, spacing is the voxel spacing, x and y are plane coordinates, and z is the axial slice number.
[0022] According to a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, a Swin-UNet backbone network model is constructed, comprising:
[0023] Determine a Swin-UNet base network comprising 5 layers, wherein the Swin-UNet base network sequentially comprises input processing, downsampling, upsampling, output processing, and skip connections;
[0024] The Swin-UNet basic network includes two-channel outputs, the first channel outputs background pixel results, and the second channel outputs target area pixel results. The final segmentation result is determined according to the output probability.
[0025] According to a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, a HSMA module is constructed, and the HSMA module is added to the Swin-UNet backbone network model to obtain a HSMA UNet network model, including:
[0026] Construct a parallel hybrid sampling module, use Swin Transformer and convolution operation parallel sampling to form encoding / decoding, and concatenate and reduce the sampling results of the Swin Transformer and the convolution operation parallel sampling in the channel dimension to form a hybrid feature;
[0027] Construct a multi-scale spatial channel attention module, use convolution kernels of size 1, 3, and 5 to extract features of different scales, form mixed features, and then perform compression excitation in the spatial or channel dimension, redistribute the weights of features of different scales, and perform mixed output;
[0028] Construct the HSMA encoding / decoding module. The input features pass through two parallel branches. The first branch uses several SwinTransformers for sampling and is connected to a channel attention calculation with three scales of 1, 3, and 5. The second branch uses convolution operations, normalization, activation functions, and Dropout operations for sampling and is connected to a channel attention calculation with three scales of 1, 3, and 5. The sampling results of the two branches are concatenated in the channel dimension and connected to a 1×1×1 convolution to halve the dimension to form the final output.
[0029] Build the HSMAUNet network, add the HSMA module to the Swin-UNet base network, replace the Swin-UNet network, and make the model structure become: Input image The number of channels is 1; the first layer of patch partitioning, the image is converted into D / 3×W / 3×H / 3 patches of size 3×3×3 at intervals of 3; after the second layer of linear embedding and HSMA sampling, the feature map The number of channels is 96; after the third layer of patch merging and HSMA sampling, the feature map The number of channels is 192; the fourth layer feature map The number of channels is 384; the fifth layer feature map The number of channels is 768; a patch expansion is performed to change the number of channels to 384, the feature map The upsampling part uses the convolution method after skip connection and follows the dimension change strategy corresponding to downsampling. A 1×1×1 convolution is performed on the top feature map, and the number of output channels is 2. Segmentation result diagram.
[0030] According to a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, the formula corresponding to the parallel mixed sampling module is:
[0031] F out =C1(Concatenate(S s (F in ),S c (F in )))
[0032] in, is the input and output feature map, double-letter represents a real number set, C1 is a 1×1×1 convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. c For convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used, and Concatenate is a connection operation;
[0033] The formula corresponding to the multi-scale spatial channel attention module is:
[0034] F mix =C c (Add(C1(F in ),C3(F in ),C5(F in ))
[0035] W s =SSK(F mix )
[0036] W c =CSK(F mix )
[0037] F out =W s F mix +W c F mix
[0038] Among them, F in 、F out , Input, output, mixed feature map, double letter represents the set of real numbers, Cc is a c×c×c convolution operation, and SSK is a compression excitation in the spatial dimension, which will form a spatial weight graph W s , CSK is the compression excitation of the channel dimension, which will form 3 channel weight graphs W c , Add is a bitwise addition operation;
[0039] The formula corresponding to the HSMA encoding / decoding module is:
[0040] F smix =Add(C1(S s (F in )),C3(S s (F in )),C5(S s (F in ))
[0041] F s =CSK(F smix )F smix
[0042] F cmix =Add(C1(S c (F in )),C3(S c (F in )),C5(S c (F in ))
[0043] F c =SSK(F cmix )F cmix +CSK(F cmix )F cmix
[0044] F out =C1(Concatenate(F s ,F c ))
[0045] Among them, F in 、F out 、F smix , are input, output, and mixed feature maps, F s and Double letters represents a set of real numbers, representing the sampling results of different branches, C c is a c×c×c convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. cFor convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used. SSK is the compressed excitation of the spatial dimension, CSK is the compressed excitation of the channel dimension, Add is the bitwise addition operation, and Concatenate is the connection operation.
[0046] According to a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, DAC is constructed, and DAC is embedded into the HSMA UNet network model to obtain the HSMA DAC UNet segmentation model, including:
[0047] The features are densely connected at the same level and fused across different levels to achieve full and dense fusion of features. The fused features are split into sections, and the cross-self-attention in any two dimensions is calculated to promote joint feature calculation, suppress irrelevant noise expression, and use serial connection to reduce information isolation caused by splitting. The overall attention calculation process of DAC is as follows:
[0048] (t1,t2,t3)=fuse(f1,f2,f3,f4)
[0049] o i =acsa(t i )i=1,2,3
[0050] Among them, the input f1,f2,f3,f4 represents the four-layer output of downsampling, fuse represents the dense fusion module, o i represents the output of the DAC module, acsa represents the cross-section self-attention module, and a total of 3 o i , participate in the resolution restoration of upsampling;
[0051] DAC is embedded into the skip connection module of the HSMA UNet network model to construct a structure that focuses on feature expression and extraction, as well as feature joint calculation.
[0052] According to a nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention, the HSMADAC UNet segmentation model is trained using the preprocessed image samples to obtain a nasopharyngeal carcinoma target area segmentation model, including:
[0053] Adopt Adam optimization method, using the joint loss function composed of Dice loss function and cross entropy loss function;
[0054] Based on the preprocessed image samples, the HSMADAC UNet segmentation model is trained to output the nasopharyngeal carcinoma target area segmentation model.
[0055] In a second aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion as described in any one of the above.
[0056] In a third aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion as described in any one of the above.
[0057] The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention proposes a HSMA encoding and decoding module based on parallel mixed sampling and multi-scale spatial channel attention. The module uses SwinTransformer and convolution dual-branch parallel sampling, and then mixes the features of the two, which can overcome the shortcomings of a single sampling mechanism and effectively combine the ability of the two to process global / local information. Subsequently, multi-scale spatial channel attention is introduced, and different sizes of receptive fields are used for feature extraction and fusion, so that the model can adapt to targets of different shapes, scales, and positions, and then compress and excite the spatial and channel dimensions respectively to enhance the perception extraction ability of key features; a feature fusion module DAC based on dense fusion and cross-level self-attention is also proposed. The DAC module densely connects and cross-level fuses the output features of the downsampled samples, effectively realizes the reuse of the front layer features and the fusion of features at different levels, and forms multi-scale mixed features. Then the mixed features are split along three dimensions and divided into three serial cross-level self-attention calculations of the cross-level sections, which effectively reduces the spatiotemporal complexity and reduces the information isolation caused by the splitting. Self-attention can suppress irrelevant noise, explore the intrinsic connections of feature maps, and enhance the perception of details in the edge areas of the target area. The HSMA module and DAC module are loaded on the Swin-UNet backbone network to construct HSMA DACUNet, which has achieved a stronger segmentation effect in the nasopharyngeal carcinoma target area segmentation task compared with other mainstream segmentation networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0059] Figure 1 It is a flow chart of a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by the present invention;
[0060] Figure 2 It is the overall framework diagram of the model provided by the present invention;
[0061] Figure 3 It is a structural diagram of the HSMA module provided by the present invention;
[0062] Figure 4 is a structural diagram of a DAC module provided by the present invention;
[0063] Figure 5 This is a visualization result diagram of segmenting the nasopharyngeal carcinoma target area provided by the present invention;
[0064] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] In view of the limitations of the prior art, the present invention proposes a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion. The multi-scale mixed sampling adopted focuses on the representation and extraction of features, uses SwinTransformer and convolution dual branches for parallel sampling, and then mixes the features of the two, which can overcome the shortcomings of a single sampling mechanism and effectively combine the ability of the two to process global / local information. Subsequently, multi-scale spatial channel attention is introduced, and different sizes of receptive fields are used for feature extraction and fusion, so that the model can adapt to targets of different shapes, scales, and positions, and then compress and excite the spatial and channel dimensions respectively to enhance the perception and extraction ability of key features. Feature fusion focuses on the joint calculation of features, densely connects and cross-level fuses the output features of the downsampled samples, effectively realizes the reuse of the front-layer features and the fusion of features at different levels, and forms multi-scale mixed features. Then the mixed features are split along three dimensions and divided into three serial cross-section self-attention calculations, which effectively reduces the spatiotemporal complexity and reduces the information isolation caused by the splitting. Self-attention can suppress irrelevant noise, tap the intrinsic connection of feature maps, and enhance the perception of details in the edge area of the target area.
[0067] Figure 1 is a flow chart of a nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion provided by an embodiment of the present invention, such as Figure 1 As shown, including:
[0068] Step 100: collecting nasopharyngeal carcinoma three-dimensional target area image samples, and preprocessing the nasopharyngeal carcinoma three-dimensional target area image samples to obtain preprocessed image samples;
[0069] Step 200: Build a Swin-UNet backbone network model;
[0070] Step 300: constructing a HSMA module based on parallel hybrid sampling and multi-scale spatial channel attention, and adding the HSMA module to the Swin-UNet backbone network model to obtain a HSMA UNet network model;
[0071] Step 400: construct a dense fusion and cross-section self-attention module DAC, embed DAC into the HSMAUNet network model, and obtain the HSMADAC UNet segmentation model;
[0072] Step 500: using the preprocessed image samples to train the HSMADAC UNet segmentation model to obtain a nasopharyngeal carcinoma target segmentation model;
[0073] Step 600: input the nasopharyngeal carcinoma three-dimensional target area image to be processed into the nasopharyngeal carcinoma target area segmentation model, and output the nasopharyngeal carcinoma target area segmentation result image.
[0074] Specifically, Figure 2 As shown, step 100 is data preprocessing. Data preprocessing is mainly divided into four steps: format conversion, image alignment, cropping, normalization, image size fixation, and data enhancement. The purpose is to convert the original medical data into a format more suitable for deep learning model training, remove the interference of irrelevant noise, enhance image quality, and improve the training quality and speed of the model.
[0075] Format conversion is used to convert DICOM format data and annotations into NIFTI format data. NIFTI format data has faster reading efficiency during training. The conversion steps are as follows:
[0076] The first step is to filter the file names, classify the user data into CT image data DCM_images and label data DCM_lables, and then read them into the memory for processing;
[0077] The second step is to first read DCM_label[sequence] to obtain the contour information, and then use the contour index luid to find the corresponding metadata in DCM_images[luid];
[0078] The third step is to perform contour coordinate transformation to convert physical coordinates (x, y, z) into voxel coordinates. The coordinate transformation formula is as follows:
[0079]
[0080] Among them, origin is the coordinate of the spatial origin, spacing is the voxel spacing, x and y are plane coordinates, and z is the axial slice number.
[0081] The fourth step is to convert DCM_images into NIFTI files NII_image and save NII_image and NII_label in the naming format of patient_image.nii.gz and patient_label.nii.gz.
[0082] The image alignment, cropping, and normalization process mainly includes three parts: data cropping and regularization, data irrelevant feature cleaning, and data normalization. The resolution of each layer of CT image of the original data is 512×512, and the voxel distance is about 1mm×1mm. The processing process includes the following steps:
[0083] In the first step, the patient's image data is read and the voxel spacing is obtained, which is assumed to be (p1, p2, p3). The patient's voxel information is then scaled and aligned according to (p1, p2, p3).
[0084] In the second step, the threshold operation is used to remove the boundary part, and the body and machine tool parts are separated by the corrosion operation. Then, the closed expansion operation and the connected domain algorithm are used to select the body part and fill its internal holes to generate a body mask.
[0085] In the third step, each layer of body mask is stacked to form a three-dimensional body mask, and the image data of the body part is extracted using the mask. After padding expansion, a three-dimensional image of NPC with a standard size and no interference is obtained.
[0086] The fourth step is to normalize the ray intensity values of the image to enhance the contrast of the image and obtain preprocessed three-dimensional image data that can be used for training.
[0087] The purpose of fixing the image size is to make the image adapt to the scale of the network, especially the Swin Transformer, which has strict requirements on the image size. This approach can fix the image to a size such as 120×120×120.
[0088] The purpose of data augmentation is to control the size of the input training images and enrich the input data of the model to avoid overfitting of the model. The operations used include random cropping, rotation, and flipping of images.
[0089] Step 200 is to construct the Swin-UNet backbone network model. The UNet network has three layers, four layers, five layers and other different downsampling times, and different usage scenarios are suitable for different model structures. Too few downsampling times will make it difficult for the model to extract deep features, and too high a number of layers will increase the parameter amount of the model on the one hand, and also cause the smaller target area information to be diffused, making it difficult to obtain the best effect. The Swin-UNet backbone network constructed by the present invention selects the basic 5-layer UNet network as the basic network, and the number of image channels input by the model is 1. First, it will undergo Pacth partitioning and linear embedding, and the number of channels becomes 96, and the size is reduced by three times. Then each time it undergoes downsampling, the number of channels doubles, the size is halved, and the upsampling is similar, the number of channels is halved, and the size is doubled. The upsampling and downsampling use the method of taking elements by interval and then linear transformation. The last linear projection outputs 2-channel results of the original image size, one outputs background pixel results, and one outputs target area pixel results, and the final segmentation result is determined based on its output probability.
[0090] Step 300 is to construct an encoding / decoding module HSMA based on parallel hybrid sampling and multi-scale spatial channel attention, and embed the module into Swin-UNet to construct HSMA UNet, such as Figure 3 shown.
[0091] The first step is to build a parallel mixed sampling module. The encoding / decoding uses Swin Transformer and convolution operations for parallel sampling respectively, and then the sampling results of the two are spliced and reduced in channel dimension to form mixed features. The formula for parallel mixed sampling is defined as follows:
[0092] F out =C1(Concatenate(S s (F in ),S c (F in )))
[0093] where F in , is the input and output feature map, double-letter represents a real number set, C1 is a 1×1×1 convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. c For convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used, and Concatenate is a connection operation.
[0094] The second step is to build a multi-scale spatial channel attention module, using convolution kernels of 1, 3, and 5 to extract features of different scales, form mixed features, and then perform compression excitation in the spatial or channel dimension, redistribute feature weights of different scales, and finally mix the output. The formula for multi-scale spatial channel attention is defined as follows:
[0095] F mix =C C (Add(C1(F in ),C3(F in ),C5(F in ))
[0096] W s =SSK(F mix )
[0097] W c =CSK(F mix )
[0098] F out =W s F mix +W c F mix
[0099] where F in , F out , Input, output, mixed feature map, double letter represents the set of real numbers, C c is a c×c×c convolution operation, and SSK is a compression excitation in the spatial dimension, which will form a spatial weight graph W s , CSK is the compression excitation of the channel dimension, which will form 3 channel weight graphs W c , Add is a bitwise addition operation.
[0100] The third step is to build the HSMA encoding / decoding module. The input features pass through two parallel branches. Branch 1 uses several Swin Transformers for sampling, followed by a channel attention calculation of 1, 3, and 5 scales. Branch 2 uses convolution operations, normalization, activation functions, and Dropout operations for sampling, followed by a spatial channel attention calculation of 1, 3, and 5 scales. The sampling results of the two branches are concatenated in the channel dimension, followed by a 1×1×1 convolution to halve the dimension to form the final output. The formula for constructing the HSMA encoding / decoding is expressed as follows:
[0101] F smix =Add(C1(S s (F in )),C3(S s (Fin )),C5(S s (F in ))
[0102] F s =CSK(F smix )F smix
[0103] F cmix =Add(C1(S c (F in )),C3(S c (F in )),C5(S c (F in ))
[0104] F c =SSK(F cmix )F cmix +CSK(F cmix )F cmix
[0105] F out =C1(Concatenate(F s ,F c ))
[0106] where F in , F out , F smix , are input, output, and mixed feature maps, F s and Double letters represents a set of real numbers, representing the sampling results of different branches, C c is a c×c×c convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. c For convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used. SSK is the compressed excitation of the spatial dimension, CSK is the compressed excitation of the channel dimension, Add is the bitwise addition operation, and Concatenate is the connection operation.
[0107] The fourth step is to build the HSMA UNet network and add the HSMA module to the Swin-UNet basic network. After replacing the Swin-UNet network, the model structure becomes: Input image The number of channels is 1; the first layer of patch partitioning, the image is converted into D / 3×W / 3×H / 3 patches of size 3×3×3 at intervals of 3; after the second layer of linear embedding and HSMA sampling, the feature map The number of channels is 96; after the third layer of patch merging and HSMA sampling, the feature map The number of channels is 192; the fourth layer feature map The number of channels is 384; the fifth layer feature map The number of channels is 768; then a patch expansion is performed to change the number of channels to 384, the feature map The upsampling part uses the convolution method after skip connection and follows the dimension change strategy corresponding to downsampling. A 1×1×1 convolution is performed on the top feature map, and the number of output channels is 2. Segmentation result diagram.
[0108] Step 400 is to construct a feature fusion module DAC based on dense fusion and cross-section self-attention, and embed the DAC module on the basis of HSMAUNet to construct HSMA DAC UNet, such as Figure 4 shown.
[0109] The first step is to build a feature fusion module DAC based on dense fusion and cross-section self-attention. First, the features are densely connected at the same level and cross-level fusion at different levels to achieve full dense fusion of features, then the sections are split, and the cross-self-attention in any two dimensions is calculated to promote the joint calculation of features, suppress the expression of irrelevant noise, and use series connection to reduce information isolation caused by splitting. The overall attention calculation process of the DAC module can be expressed as the following formula:
[0110] (t1,t2,t3)=fuse(f1,f2,f3,f4)
[0111] o i =acsa(t i )i=1,2,3
[0112] Among them, the input f1,f2,f3,f4 represents the four-layer output of downsampling, fuse represents the dense fusion module, o i represents the output of the DAC module, acsa represents the cross-section self-attention module, and a total of 3 o i , participate in the resolution restoration of upsampling.
[0113] The second step is to build HSMA DAC UNet and embed the DAC module into the skip connection module of HSMA UNet to build a structure that focuses on both feature expression and extraction and feature joint calculation.
[0114] Step 500 is to perform training using nasopharyngeal carcinoma three-dimensional images according to the constructed model.
[0115] The model uses the Adam optimization method in training, where the batch size is set to 1 and the training is done for 20,000 epochs; the initial learning rate is set to 0.0001, and each adjustment is 0.9 times, and the learning rate regulator is MultiStepScheduler.
[0116] The loss function used in model training is a joint loss function of the Dice loss function and the cross entropy loss function. The Dice loss function is a very commonly used loss function, and the calculation formula is as follows:
[0117]
[0118] L1=1-S
[0119] Among them, X, Y represent the true image and the predicted image respectively, TP is the correctly predicted foreground pixel, FP is the background pixel predicted as the foreground, TN is the correctly predicted background pixel, and FN is the foreground pixel predicted as the background. Simply put, the Dice coefficient is the ratio of the correctly predicted pixels to the total pixels. The closer its value is to 1, the more accurate the prediction result is, and the closer the Dice Loss is to 0.
[0120] The core idea of the cross entropy loss function is to measure the difference between the probability distribution of the model output and the actual label. When the true category probability of the actual label tends to 1, the value of the loss function tends to 0; and when the true category probability of the actual label tends to 0, the value of the loss function tends to negative infinity, expressing the uncertainty of the model. The calculation formula of the cross entropy loss function is as follows:
[0121]
[0122] Assume that the model has C categories, where t i is the i-th element of the actual label, y i is the i-th element of the probability distribution output by the model.
[0123] The joint loss function uses the Dice loss function and the cross entropy loss function, each accounting for 50%. The calculation formula is as follows:
[0124] L=L1+L2
[0125] Finally, after preprocessing, the NPC 3D image is input into the HSMA DAC UNet network to obtain the segmented result image. The red foreground is the segmented NPC target area, and the black part is the background. The segmentation visualization result is shown as follows: Figure 5 As shown, the visualization results are presented from three directions: axial, sagittal, and coronal.
[0126] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion, the method comprising: collecting nasopharyngeal carcinoma three-dimensional target area image samples, preprocessing the nasopharyngeal carcinoma three-dimensional target area image samples to obtain preprocessed image samples; constructing a Swin-UNet backbone network model; constructing an HSMA module, adding the HSMA module to the Swin-UNet backbone network model to obtain an HSMA UNet network model; constructing a DAC, embedding the DAC into the HSMA UNet network model to obtain an HSMA DAC UNet segmentation model; using the preprocessed image samples to train the HSMA DAC UNet segmentation model to obtain a nasopharyngeal carcinoma target area segmentation model; inputting the nasopharyngeal carcinoma three-dimensional target area image to be processed into the nasopharyngeal carcinoma target area segmentation model, and outputting a nasopharyngeal carcinoma target area segmentation result image.
[0127] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0128] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the nasopharyngeal carcinoma target area segmentation method based on multi-scale mixed sampling and feature fusion provided by the above-mentioned methods, the method comprising: collecting nasopharyngeal carcinoma three-dimensional target area image samples, preprocessing the nasopharyngeal carcinoma three-dimensional target area image samples to obtain preprocessed image samples; constructing a Swin-UNet backbone network model; constructing an HSMA module, adding the HSMA module to the Swin-UNet backbone network model to obtain an HSMA UNet network model; constructing DAC, embedding DAC into the HSMA UNet network model to obtain an HSMA DAC UNet segmentation model; using the preprocessed image samples to train the HSMA DAC UNet segmentation model to obtain a nasopharyngeal carcinoma target area segmentation model; inputting the nasopharyngeal carcinoma three-dimensional target area image to be processed into the nasopharyngeal carcinoma target area segmentation model, and outputting a nasopharyngeal carcinoma target area segmentation result image.
[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion, characterized in that: include: Collecting nasopharyngeal carcinoma three-dimensional target area image samples, and preprocessing the nasopharyngeal carcinoma three-dimensional target area image samples to obtain preprocessed image samples; Build the Swin-UNet backbone network model; Constructing a HSMA module based on parallel hybrid sampling and multi-scale spatial channel attention, and adding the HSMA module to the Swin-UNet backbone network model to obtain a HSMA UNet network model; Construct a DAC module based on dense fusion and cross-section self-attention, embed DAC into the HSMA UNet network model, and obtain the HSMA DAC UNet segmentation model; Using the preprocessed image samples to train the HSMADAC UNet segmentation model to obtain a nasopharyngeal carcinoma target area segmentation model; The nasopharyngeal carcinoma three-dimensional target area image to be processed is input into the nasopharyngeal carcinoma target area segmentation model, and the nasopharyngeal carcinoma target area segmentation result image is output.
2. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: Preprocessing the nasopharyngeal carcinoma three-dimensional target area image sample to obtain a preprocessed image sample includes: The DICOM format data and annotations of the nasopharyngeal carcinoma three-dimensional target image sample are converted into NIFTI format, the nasopharyngeal carcinoma three-dimensional target image sample is divided into CT image data DCM_images and annotation data DCM_labels and stored in memory, each segment of DCM_label data is read, and the contour reference metadata DCM_images[luid] is found according to the internal luid, and the contour coordinates (x, y, z) are converted into voxel image coordinates through the coordinates and voxel information of the metadata Align the voxel information of the nasopharyngeal carcinoma three-dimensional target area image sample to make the intervals equally spaced in each direction, use a threshold operation to remove the boundary part, separate the body and the machine tool part through an erosion operation, select the body part through a closed expansion operation and a connected domain algorithm, fill the internal holes of the body part to obtain a body mask part, stack each layer of the body mask part to generate a three-dimensional body mask, extract the body part image data through the three-dimensional body mask, perform padding amplification on the body part image data, obtain a nasopharyngeal carcinoma three-dimensional image of a standard size and without interference data, normalize the ray intensity value of the nasopharyngeal carcinoma three-dimensional image, enhance the image contrast, and obtain three-dimensional image data for training; Read the upper and lower boundaries A(x min ,y min ,z min ) and B(x max ,y max ,z max ), performing padding expansion or supplementing black edges according to the two boundaries, and adjusting the three-dimensional image data to a preset image size; The resized three-dimensional image data is randomly cropped, rotated, and flipped, and Gaussian noise and Gamma correction are introduced. Label Smoothing is used to convert the category label into a probability distribution to obtain the preprocessed image sample.
3. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: Convert contour coordinates (x, y, z) to voxel image coordinates using metadata coordinates and voxel information include: Among them, origin is the coordinate of the spatial origin, spacing is the voxel spacing, x and y are plane coordinates, and z is the axial slice number.
4. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: Build the Swin-UNet backbone network model, including: Determine a Swin-UNet base network comprising 5 layers, wherein the Swin-UNet base network sequentially comprises input processing, downsampling, upsampling, output processing, and skip connections; The Swin-UNet basic network includes two-channel outputs, the first channel outputs background pixel results, and the second channel outputs target area pixel results. The final segmentation result is determined according to the output probability.
5. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: Construct the HSMA module, add the HSMA module to the Swin-UNet backbone network model, and obtain the HSMAUNet network model, including: Construct a parallel hybrid sampling module, use Swin Transformer and convolution operation parallel sampling to form encoding / decoding, and concatenate and reduce the sampling results of the Swin Transformer and the convolution operation parallel sampling in the channel dimension to form a hybrid feature; Construct a multi-scale spatial channel attention module, use convolution kernels of size 1, 3, and 5 to extract features of different scales, form mixed features, and then perform compression excitation in the spatial or channel dimension, redistribute the weights of features of different scales, and perform mixed output; Construct the HSMA encoding / decoding module. The input features pass through two parallel branches. The first branch uses several SwinTransformers for sampling and is connected to a channel attention calculation with three scales of 1, 3, and 5. The second branch uses convolution operations, normalization, activation functions, and Dropout operations for sampling and is connected to a channel attention calculation with three scales of 1, 3, and 5. The sampling results of the two branches are concatenated in the channel dimension and connected to a 1×1×1 convolution to halve the dimension to form the final output. Build the HSMA UNet network, add the HSMA module to the Swin-UNet base network, replace the Swin-UNet network, and make the model structure become: Input image The number of channels is 1; the first layer of patch partitioning, the image is converted into D / 3×W / 3×H / 3 patches of size 3×3×3 at intervals of 3; after the second layer of linear embedding and HSMA sampling, the feature map The number of channels is 96; after the third layer of patch merging and HSMA sampling, the feature map The number of channels is 192; the fourth layer feature map The number of channels is 384; the fifth layer feature map The number of channels is 768; a patch expansion is performed to change the number of channels to 384, the feature map The upsampling part uses the convolution method after skip connection and follows the dimension change strategy corresponding to downsampling. A 1×1×1 convolution is performed on the top feature map, and the number of output channels is 2. Segmentation result diagram.
6. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 5, characterized in that: The formula corresponding to the parallel mixed sampling module is: F out =C1(Concatenate(S s (F in ),S c (F in ))) Among them, F in , is the input and output feature map, double-letter represents a real number set, C1 is a 1×1×1 convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. c For convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used, and Concatenate is a connection operation; The formula corresponding to the multi-scale spatial channel attention module is: F mix =C c (Add(C1(F in ),C3(F in ),C5(F in )) W s =SSK(F mix ) W c =CSK(F mix ) F out =W s F mix +W c F mix Among them, F in 、F out , Input, output, mixed feature map, double letter represents the set of real numbers, C c is a c×c×c convolution operation, and SSK is a compression excitation in the spatial dimension, which will form a spatial weight graph W a , CSK is the compression excitation of the channel dimension, which will form 3 channel weight graphs W c , Add is a bitwise addition operation; The formula corresponding to the HSMA encoding / decoding module is: F amix =Add(C1(S s (F in )),C3(S s (F in )),C5(S s (F in )) F s =CSK(F smix )F smix F cmix =Add(C1(S c (F in )),C3(S c (F in )),C5(S c (F in )) F c =SSK(F cmix )F cmix +CSK(F cmix )F cmix F out =C1(Concatenate(F s ,F c )) Among them, F in 、F out 、F smix , are input, output, and mixed feature maps, F s and Double letters represents a set of real numbers, representing the sampling results of different branches, C c is a c×c×c convolution operation, S s Swin Transformer sampling, which consists of N Swin Transformer Blocks in series. c For convolution sampling, 3×3×3 convolution, LayerNorm, PReLU, and Dropout operations are used. SSK is the compressed excitation of the spatial dimension, CSK is the compressed excitation of the channel dimension, Add is the bitwise addition operation, and Concatenate is the connection operation.
7. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: Construct DAC and embed DAC into the HSMA UNet network model to obtain the HSMA DAC UNet segmentation model, including: The features are densely connected at the same level and fused across different levels to achieve full and dense fusion of features. The fused features are split into sections, and the cross-self-attention in any two dimensions is calculated to promote joint feature calculation, suppress irrelevant noise expression, and use serial connection to reduce information isolation caused by splitting. The overall attention calculation process of DAC is as follows: (t1,t2,t3)=fuse(f1,f2,f3,f4) or i =acsa(t i )i=1,2,3 Among them, the input f1,f2,f3,f4 represents the four-layer output of downsampling, fuse represents the dense fusion module, o i represents the output of the DAC module, acsa represents the cross-section self-attention module, and a total of 3 o i , participate in the resolution restoration of upsampling; DAC is embedded into the skip connection module of the HSMA UNet network model to construct a structure that focuses on feature expression and extraction, as well as feature joint calculation.
8. The nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion according to claim 1, characterized in that: The HSMA DAC UNet segmentation model is trained using the preprocessed image samples to obtain a nasopharyngeal carcinoma target segmentation model, including: Adopt Adam optimization method, using the joint loss function composed of Dice loss function and cross entropy loss function; Based on the preprocessed image samples, the HSMA DAC UNet segmentation model is trained to output the nasopharyngeal carcinoma target area segmentation model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the nasopharyngeal carcinoma target segmentation method based on multi-scale mixed sampling and feature fusion as described in any one of claims 1 to 8 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for nasopharyngeal carcinoma target segmentation based on multi-scale mixed sampling and feature fusion as described in any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Preoperative intestinal ganglion cell recognition method based on CE-Net algorithm
CN113989264A
CBCT jaw region segmentation method and system based on context fusion mechanism
CN115578406A
Medical image small target segmentation method based on double-branch feature fusion attention
CN116681679A
Lung CT image segmentation method based on mixed Swin Transform U-Net
CN117274147A
Medical image segmentation method based on Swin Transform and CNN parallel network
CN117351030A
Cited By
Automatic identification method of orbital nose skull base anatomical structure, electronic equipment and computer readable storage medium
CN121280872A
