Mandibular neural tube CBCT panoramic image segmentation network training method based on Mmba architecture

Abstract

The invention discloses a mandibular neural tube CBCT panoramic image segmentation network training method based on a Mama architecture, and relates to an image processing technology in the field of computer vision. According to the segmentation network training method, an encoder and decoder architecture is adopted, tubular structure features are extracted by enlarging a receptive field and learning deformation based on feature extraction operation of a tubular convolution kernel, and then a three-vision feature fusion strategy is carried out to further guide a model to supplement attention to key features from different angles; the encoder part models global and multi-scale features through a Mama architecture, and keeps efficient in training and reasoning processes, and finally enables the network to have a more complete segmentation effect on a CBCT image of a tubular structure.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision image processing technology, and in particular, to a mandibular nerve canal CBCT panoramic image segmentation network training method based on Mamba architecture. Background Art

[0002] The research on medical image segmentation of tubular structures has always been an important research topic. Medical images of tubular structures have the characteristics of slender and tortuous structures, and many key research parts of human organs are tubular structures, such as cardiovascular vessels, arterial vessels, mandibular nerve tubes, etc. However, most medical image segmentation networks do not conduct special research on medical images of this type of structure, which makes it difficult to segment and process such medical images. Summary of the invention

[0003] In order to solve the problem of segmentation processing of tubular structure medical images in the prior art, the present invention proposes a mandibular nerve canal CBCT panoramic image segmentation network training method based on Mamba architecture, comprising the following steps: S1. Construct a training data set: collect CBCT images of the mandibular nerve canal, and then use manual annotation to process the collected CBCT images to generate panoramic CBCT images. Annotate the parts that need to be segmented in the panoramic CBCT images and convert them into label maps. The pixel points representing the parts that need to be segmented are assigned a value of 1, and the pixel points representing the background are assigned a value of 0. The label map and the corresponding CBCT images constitute a set of training samples and are sent to the network for training; S2. Constructing the training framework: The training framework consists of three parts: stem layer, encoder part, decoder part, and jump connection part. The encoder part consists of four encoders, each of which includes Mamba module, GSC module and TUC module. The decoder part includes convolution layer, upsampling layer, segmentation head, jump connection and residual connection layer. The process of one iteration is as follows: S2-1. Panoramic CBCT images to be trained First send it to the stem layer, the stem layer uses deep convolution, the image Extract the first feature scale through the stem layer ; then Feed into the Mamba module of the encoder part and the corresponding lower adoption layer; S2-2. In the first Mamba module, the image spatial features from the stem layer are first extracted through the gated spatial convolution module GSC. In each subsequent Mamba module, the image spatial features from the previous Mamba module are first extracted through the gated spatial convolution module GSC. The mth spatial feature of the input First, it is sent to two TUC modules to obtain and ; Then, the two features are multiplied pixel by pixel to control information transmission in a gating mechanism; Finally, another TUC module is used to further fuse the two features into , and the features are transformed through the residual connection layer of the decoder Send to decoder; S2-3. Features obtained through GSC and three TUC modules After that, the ToM module models features from multiple directions. Dependencies between features Expand into three sequences along three different directions, then perform corresponding feature interaction operations, and finally obtain the integrated feature map ; S2-4. Repeat steps S2-1, S2-2, and S2-3 to obtain the feature maps output by the four encoders , , , And input to the jump connection layer to get the result of each jump connection layer , , , ; S2-5. The feature map obtained by the last encoder Input to the decoder for upsampling to obtain the feature map , the decoder part receives , and the feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;right After another convolution upsampling, a feature map with the same resolution as the input feature map is obtained. ; S2-6. The obtained feature map The output is sent through a segmentation head, and the loss function is calculated with the output result and the label map. Back propagation is performed based on the calculated result, and a round of iteration is completed.

[0004] Optionally, in step S2-1, the panoramic CBCT image data is uniformly expressed as: Where C represents the number of input channels, , H, W represent the length, width, and height of the panoramic CBCT image respectively; for the stem layer, the kernel size is , fill size , step length , the first scale feature of the panoramic CBCT image extracted by stem is expressed as: .

[0005] Optionally, in step S2-2, the GSC calculation method is expressed by the following formula: , in, represents the mth spatial feature of the input, and TUC represents the tubular convolution block.

[0006] Optionally, in step S2-2, in TUC, a standard 3D convolution coordinate is first given, and the center coordinate is ; A 3×3×3 convolution kernel K is expressed as: , The change in the x-axis direction is expressed as: The change in the y-axis direction is: The change in the z-axis direction is: Among them, i, j, k represent the changes in the x, y, and z axis directions respectively, and ∆x, ∆y, ∆z represent the offsets in the corresponding directions respectively. Since the offsets are decimals, but the coordinates are in integer form, bilinear interpolation is used, which is expressed as: in, Representing equations ,equation and equation The decimal position of , K' lists all integer spatial positions, B is the bilinear interpolation kernel, which is decomposed into three one-dimensional kernels, namely: , Where b represents the bilinear interpolation kernel in each direction, , , Represents the convolution kernels along three directions respectively; for each K, three feature maps from the lth layer are extracted from the x-axis, y-axis, and z-axis , , , expressed as: , in, , , Represents the weight at the corresponding position, and the features extracted by the convolution kernel K in the lth layer are calculated using the accumulation method.

[0007] Optionally, in step S2-3, the ToM module calculation method can be expressed by the following formula: , Among them, Mamba is the Mamba layer used to model global information in the sequence, represents the feature in the forward direction, Features that indicate the reverse direction, Features representing the orientation between slices; For the mth Mamba layer, the calculation process can be defined as: , , , Among them, GSC and ToM represent the gated spatial convolution module and the three-way Mamba module in the network respectively. , LN represents layer normalization, MLP represents multi-layer perceptron layer, which is used to enrich feature representation, Indicates that from Characteristics of the layer.

[0008] Optionally, in step S2-4, As input, then pass Convolution, residual connection, and finally Convolution performs feature fusion and size adjustment, expressed as: , in represents the final output result of the ith Mamba module jump connection, Indicates a convolution operation.

[0009] Optionally, in step S2-5, the feature map obtained after each upsampling in the decoder is compared with the feature map output by the encoder at the same layer. The concatenation is performed in the channel dimension to obtain the concatenated feature map, which is then passed through the convolution layer and finally output as a feature map of the same size as the original image.

[0010] Optionally, in step S2-6, the final output result of the network is obtained through the segmentation head, and the output result and the label map are used to calculate the cross entropy loss. The calculation formula is as follows: , Where N is the number of samples, C is the number of categories, is the indicator function that pixel i belongs to category c in the true label, is the probability that the model predicts that pixel i belongs to category c.

[0011] The present invention also proposes a mandibular nerve canal CBCT panoramic image processing method, which uses a segmentation network obtained by the aforementioned mandibular nerve canal CBCT panoramic image segmentation network training method based on the Mamba architecture.

[0012] The beneficial effects of the present invention are as follows: through the mandibular nerve canal CBCT panoramic image segmentation network training method based on the Mamba architecture of the present invention, the segmentation capability of the mandibular canal and the mandibular nerve canal is improved when the data source in the field of medical image segmentation is limited, and an effective tubular structure data segmentation solution is provided for intelligent medical image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 It is a network framework diagram of the present invention; Figure 2 It is a schematic diagram of the calculation process of the downsampling layer of the Mamba module of the present invention; Figure 3 is a schematic diagram of the calculation process of the GSC module including TUC of the present invention; Figure 4 It is a schematic diagram of the fusion of three visual features in the present invention; Figure 5 It is a comparison diagram of some output results of the present invention and the existing method. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0016] like Figure 1-Figure 4 As shown, the mandibular nerve canal CBCT panoramic image segmentation network training method based on Mamba architecture disclosed in the present invention includes the following steps: S1. Construct a training data set: collect CBCT images of the mandibular nerve canal, and then use manual annotation to process the collected CBCT images to generate panoramic CBCT images. Annotate the parts that need to be segmented in the panoramic CBCT images and convert them into label maps. The pixel points representing the parts that need to be segmented are assigned a value of 1, and the pixel points representing the background are assigned a value of 0. The label map and the corresponding CBCT images constitute a set of training samples and are sent to the network for training; S2. Constructing the training framework: The training framework consists of three parts: stem layer, encoder part, decoder part, and jump connection part. The encoder part consists of four encoders, each of which includes Mamba module, GSC module and TUC module. The decoder part includes convolution layer, upsampling layer, segmentation head, jump connection and residual connection layer. The process of one iteration is as follows: S2-1. Panoramic CBCT images to be trained First send it to the stem layer, the stem layer uses deep convolution, the image Extract the first feature scale through the stem layer ; then Feed into the Mamba module of the encoder part and the corresponding lower adoption layer; S2-2. In the first Mamba module, the image spatial features from the stem layer are first extracted through the gated spatial convolution module GSC. In each subsequent Mamba module, the image spatial features from the previous Mamba module are first extracted through the gated spatial convolution module GSC. The mth spatial feature of the input First, it is sent to two TUC modules to obtain and ; Then, the two features are multiplied pixel by pixel to control information transmission in a gating mechanism; Finally, another TUC module is used to further fuse the two features into , and the features are transformed through the residual connection layer of the decoder Send to decoder; S2-3. Features obtained through GSC and three TUC modules After that, the ToM module models features from multiple directions. Dependencies between features Expand into three sequences along three different directions, then perform corresponding feature interaction operations, and finally obtain the integrated feature map ; S2-4. Repeat steps S2-1, S2-2, and S2-3 to obtain the feature maps output by the four encoders , , , And input to the jump connection layer to get the result of each jump connection layer , , , ; S2-5. The feature map obtained by the last encoder Input to the decoder for upsampling to obtain the feature map , the decoder part receives , and the feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;right After another convolution upsampling, a feature map with the same resolution as the input feature map is obtained. ; S2-6. The obtained feature map The output is sent through a segmentation head, and the loss function is calculated with the output result and the label map. Back propagation is performed based on the calculated result, and a round of iteration is completed.

[0017] Optionally, in step S2-1, the panoramic CBCT image data is uniformly expressed as: Where C represents the number of input channels, , H, W represent the length, width, and height of the panoramic CBCT image respectively; for the stem layer, the kernel size is , fill size , step length , the first scale feature of the panoramic CBCT image extracted by stem is expressed as: .

[0018] Optionally, in step S2-2, the GSC calculation method is expressed by the following formula: , in, represents the mth spatial feature of the input, and TUC represents the tubular convolution block.

[0019] Optionally, in step S2-2, in TUC, a standard 3D convolution coordinate is first given, and the center coordinate is ; A 3×3×3 convolution kernel K is expressed as: , The change in the x-axis direction is expressed as: The change in the y-axis direction is: The change in the z-axis direction is: Among them, i, j, k represent the changes in the x, y, and z axis directions respectively, and ∆x, ∆y, ∆z represent the offsets in the corresponding directions respectively. Since the offsets are decimals, but the coordinates are in integer form, bilinear interpolation is used, which is expressed as: in, Representing equations ,equation and equation The decimal position of , K' lists all integer spatial positions, B is the bilinear interpolation kernel, which is decomposed into three one-dimensional kernels, namely: , Where b represents the bilinear interpolation kernel in each direction, , , Represents the convolution kernels along three directions respectively; for each K, three feature maps from the lth layer are extracted from the x-axis, y-axis, and z-axis , , , expressed as: , in, , , Represents the weight at the corresponding position, and the features extracted by the convolution kernel K in the lth layer are calculated using the accumulation method.

[0020] Optionally, in step S2-3, the ToM module calculation method can be expressed by the following formula: , Among them, Mamba is the Mamba layer used to model global information in the sequence, represents the feature in the forward direction, Features that indicate the reverse direction, Features representing the orientation between slices; For the mth Mamba layer, the calculation process can be defined as: , , , Among them, GSC and ToM represent the gated spatial convolution module and the three-way Mamba module in the network respectively. , LN represents layer normalization, MLP represents multi-layer perceptron layer, which is used to enrich feature representation, Indicates that from Characteristics of the layer.

[0021] Optionally, in step S2-4, As input, then pass Convolution, residual connection, and finally Convolution performs feature fusion and size adjustment, expressed as: , in represents the final output result of the ith Mamba module jump connection, Indicates a convolution operation.

[0022] Optionally, in step S2-5, the feature map obtained after each upsampling in the decoder is compared with the feature map output by the encoder at the same layer. The concatenation is performed in the channel dimension to obtain the concatenated feature map, which is then passed through the convolution layer and finally output as a feature map of the same size as the original image.

[0023] Optionally, in step S2-6, the final output result of the network is obtained through the segmentation head, and the output result and the label map are used to calculate the cross entropy loss. The calculation formula is as follows: , Where N is the number of samples, C is the number of categories, is the indicator function that pixel i belongs to category c in the true label, is the probability that the model predicts that pixel i belongs to category c.

[0024] The present invention also proposes a mandibular nerve canal CBCT panoramic image processing method, which uses a segmentation network obtained by the aforementioned mandibular nerve canal CBCT panoramic image segmentation network training method based on the Mamba architecture.

[0025] Figure 5This is a comparison diagram of the segmentation effect of the method of the present invention and other four efficient segmentation networks on the mandibular canal and mandibular nerve canal CBCT at three different visual angles. As shown in the figure, compared with other segmentation networks, the method proposed in the present application can achieve more complete segmentation of the mandibular canal and mandibular nerve canal and better processing of subtle crack details than other segmentation methods. At the same time, it has better processing for the overall structure of the panoramic surface and the integrity of the edge area.

[0026] Example

[0027] According to step S1, a training data set is constructed. The data includes a scanned mandibular CBCT image containing a region of interest and a manually annotated label map containing the region of interest. Each CBCT image forms a pair of samples with the corresponding label map.

[0028] According to step S2-1, the CBCT image to be trained is first sent to the stem layer for convolution operation to extract the first scale feature, and then further extracted features through four encoders and down-sampled five times. At the same time, the feature map obtained by downsampling each encoder is used as the input of the skip connection layer.

[0029] According to step S2-2 and step S2-3, the first scale feature extracted from the stem layer is sent to the Mamba module for feature extraction. It first passes through the GSC module, performs convolution and three-visual feature fusion in TUC, and then normalizes the layer. After that, it passes through the ToM module to model the dependency relationship between features, extract features, and finally performs downsampling operation and sends it to the next Mamba module.

[0030] According to step S2-4 and step S2-5, the feature map obtained by skip connection in each layer is concatenated with the feature map obtained by upsampling, and then an upsampling operation is performed. After five upsampling operations, a feature map with the same size as the input feature map is obtained.

[0031] According to step S2-6, the feature map obtained by the last upsampling is output through a segmentation head, the output result of the network is compared with the label map, the loss function is calculated, and back propagation is performed according to the loss function. At this point, a round of training is completed, and the above process is repeated to save the final network model parameters.

[0032] Test: Select sample pairs that have not participated in training for testing, compare the network output results with the labels, and calculate the Dice evaluation index to judge the segmentation effect of the model.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A network training method for mandibular nerve canal CBCT panoramic image segmentation based on Mamba architecture, characterized in that: The following steps are involved: S1. Construct a training data set: collect CBCT images of the mandibular nerve canal, and then use manual annotation to process the collected CBCT images to generate panoramic CBCT images. Annotate the parts that need to be segmented in the panoramic CBCT images and convert them into label maps. The pixel points representing the parts that need to be segmented are assigned a value of 1, and the pixel points representing the background are assigned a value of 0. The label map and the corresponding CBCT images constitute a set of training samples and are sent to the network for training; S2. Constructing the training framework: The training framework consists of three parts: stem layer, encoder part, decoder part, and jump connection part. The encoder part consists of four encoders, each of which includes Mamba module, GSC module and TUC module. The decoder part includes convolution layer, upsampling layer, segmentation head, jump connection and residual connection layer. The process of one iteration is as follows: S2-1. Panoramic CBCT images to be trained First send it to the stem layer, the stem layer uses deep convolution, the image Extract the first feature scale through the stem layer ; then Feed into the Mamba module of the encoder part and the corresponding lower adoption layer; S2-2. In the first Mamba module, the image spatial features from the stem layer are first extracted through the gated spatial convolution module GSC. In each subsequent Mamba module, the image spatial features from the previous Mamba module are first extracted through the gated spatial convolution module GSC. The mth spatial feature of the input First, it is sent to two TUC modules to obtain and ; Then, the two features are multiplied pixel by pixel to control information transmission in a gating mechanism; Finally, another TUC module is used to further fuse the two features into , and the features are transformed through the residual connection layer of the decoder Send to decoder; S2-3. Features obtained through GSC and three TUC modules After that, the ToM module models features from multiple directions. Dependencies between; Features Expand into three sequences along three different directions, then perform corresponding feature interaction operations, and finally obtain the integrated feature map ; S2-4. Repeat steps S2-1, S2-2, and S2-3 to obtain the feature maps output by the four encoders , , , And input to the jump connection layer to get the result of each jump connection layer , , , ; S2-5. The feature map obtained by the last encoder Input to the decoder for convolution upsampling operation to obtain the feature map , the decoder part receives , and the feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;Will With feature map Concatenate in the channel dimension to get ; Then perform convolution upsampling to obtain ;right After another convolution upsampling, a feature map with the same resolution as the input feature map is obtained. ; S2-6. The obtained feature map The output is sent through a segmentation head, and the loss function is calculated with the output result and the label map. Back propagation is performed based on the calculated result, and a round of iteration is completed.

2. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-1, the panoramic CBCT image data is uniformly expressed as: Where C represents the number of input channels, , H, W represent the length, width, and height of the panoramic CBCT image respectively; for the stem layer, the kernel size is , fill size , step length , the first scale feature of the panoramic CBCT image extracted by stem is expressed as: 。 3. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-2, the GSC calculation method is expressed by the following formula: , in, represents the mth spatial feature of the input, and TUC represents the tubular convolution block.

4. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-2, in TUC, a standard 3D convolution coordinate is first given, and the center coordinate is ; A 3×3×3 convolution kernel K is expressed as: , The change in the x-axis direction is expressed as: The change in the y-axis direction is: The change in the z-axis direction is: Among them, i, j, k represent the changes in the x, y, and z axis directions respectively, and ∆x, ∆y, ∆z represent the offsets in the corresponding directions respectively. Since the offsets are decimals, but the coordinates are in integer form, bilinear interpolation is used, which is expressed as: in, Representing equations ,equation and equation The decimal position of , K' lists all integer spatial positions, B is the bilinear interpolation kernel, which is decomposed into three one-dimensional kernels, namely: ， Where b represents the bilinear interpolation kernel in each direction, , , Represents the convolution kernels along three directions respectively; for each K, three feature maps from the lth layer are extracted from the x-axis, y-axis, and z-axis , , , expressed as: , in, , , Represents the weight at the corresponding position, and the features extracted by the convolution kernel K of the lth layer are calculated using the accumulation method.

5. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-3, the ToM module calculation method is expressed by the following formula: , Among them, Mamba is the Mamba layer used to model global information in the sequence, represents the features in the forward direction, Features that indicate the reverse direction, Features representing the orientation between slices; For the mth Mamba layer, the calculation process is defined as: , , , Among them, GSC and ToM represent the gated spatial convolution module and the three-way Mamba module in the network respectively. , LN represents layer normalization, MLP represents multi-layer perceptron layer, which is used to enrich feature representation, Indicates that from Characteristics of the layer.

6. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-4, As input, then pass Convolution, residual connection, and finally Convolution performs feature fusion and size adjustment, expressed as: ， in represents the final output result of the ith Mamba module jump connection, Indicates a convolution operation.

7. The method for training a mandibular canal CBCT panoramic image segmentation network based on the Mamba architecture according to claim 1, characterized in that: In step S2-5, the feature map obtained after each upsampling in the decoder is compared with the feature map output by the encoder at the same layer. The concatenation is performed in the channel dimension to obtain the concatenated feature map, which is then passed through the convolution layer and finally output as a feature map of the same size as the original image.

8. The mandibular canal CBCT panoramic image segmentation network training method based on Mamba architecture according to claim 1, characterized in that: In step S2-6, the final output of the network is obtained through the segmentation head, and the cross entropy loss is calculated between the output result and the label map. The calculation formula is as follows: ， Where N is the number of samples, C is the number of categories, is the indicator function that pixel i belongs to category c in the true label, is the probability that the model predicts that pixel i belongs to category c.

9. A method for processing CBCT panoramic images of mandibular canal, characterized in that: A segmentation network is obtained by the mandibular nerve canal CBCT panoramic image segmentation network training method based on the Mamba architecture as described in any one of claims 1 to 8.

Images