A spatial positioning method of lumbar CT images based on artificial neural network

Abstract

The present invention discloses a method for spatial positioning of lumbar CT images based on artificial neural networks, including: spatial positioning of the entire lumbar region in the original CT image to obtain three-dimensional spatial coordinates representing the entire lumbar region; using the three-dimensional spatial coordinates representing the entire lumbar region to crop the original CT image to obtain a local CT image of the entire lumbar region; positioning the spatial region of each lumbar vertebra in the local CT image of the entire lumbar region to obtain three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra; cropping the local CT image of the entire lumbar region using the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra to obtain a local CT image of the spatial region of each lumbar vertebra. The method can realize fast, accurate, and highly robust positioning of the spatial region of each lumbar vertebra in the CT image, and provide a good foundation for the subsequent processing of the lumbar region in the CT image.

Description

Technical Field

[0001] The invention relates to a medical image recognition method, and in particular to a lumbar CT image spatial positioning method based on an artificial neural network. Background Art

[0002] Lumbar medical CT images include not only the image information of the lumbar spine, but also other spinal regions and soft tissue areas. These useless background information will have an adverse effect on the lumbar CT image processing. Therefore, precise positioning of the lumbar spatial area becomes an important preprocessing process for lumbar CT image processing.

[0003] In order to obtain better results in lumbar vertebrae image processing, it is necessary to eliminate the adverse effects of the similarity of adjacent lumbar vertebrae on image processing, such as image segmentation or image registration of each lumbar vertebra, so it is necessary to accurately locate the spatial area of ​​each lumbar vertebra.

[0004] At present, the position and size information of the entire lumbar spine area is often directly detected and output through an artificial neural network algorithm. However, this method has low positioning accuracy for the entire spatial area of ​​the lumbar spine and low robustness.

[0005] There are usually two ways to identify each lumbar vertebra in the entire lumbar spine area. The first is to locate the spatial key points of each lumbar vertebra, and then set a fixed-size rectangular spatial area with this key point as the center as the spatial area of ​​the lumbar vertebra. This solution can obtain the position information of the lumbar vertebra more accurately, but it cannot accurately obtain the size information of the lumbar vertebra. The second is to use a sufficiently large fixed-size sliding window to slide in the image to determine whether there is a complete lumbar vertebra in the sliding window and whether the lumbar vertebra has appeared. This solution also cannot accurately obtain the size information of the lumbar vertebra, and requires a long computing time and a large amount of computing.

[0006] In summary, it is urgent to develop a spatial positioning method with high accuracy and high robustness for the overall regional positioning of the lumbar spine in CT images. In addition, on the basis of achieving the overall regional positioning of the lumbar spine, it is urgent to develop a method to directly achieve the precise positioning of the spatial region of each lumbar vertebra, without relying on setting a sufficiently large fixed-size spatial region as the spatial region of the lumbar vertebra after achieving the precise positioning of the spatial position of the lumbar spine. Summary of the invention

[0007] In view of the above defects or deficiencies in the prior art, the present invention provides a lumbar CT image spatial positioning method based on an artificial neural network. The method can realize fast, accurate and highly robust positioning of each lumbar vertebra space region in a CT image, providing a good foundation for subsequent processing of the lumbar region in CT images.

[0008] One aspect of the present invention provides a lumbar CT image spatial positioning method based on an artificial neural network, comprising:

[0009] Perform spatial positioning of the entire lumbar spine region in the original CT image to obtain three-dimensional spatial coordinates representing the entire lumbar spine region;

[0010] Using the three-dimensional spatial coordinates representing the entire lumbar region to crop the original CT image, to obtain a local CT image of the entire lumbar region;

[0011] Positioning the spatial region of each lumbar vertebra in the local CT image of the entire lumbar vertebra region to obtain three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra;

[0012] The local CT image of the entire lumbar vertebrae region is cropped using the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebrae to obtain a local CT image of the spatial region of each lumbar vertebrae.

[0013] Furthermore, the step of spatially locating the entire lumbar region in the original CT image to obtain three-dimensional spatial coordinates representing the entire lumbar region includes:

[0014] Inputting the original CT image to be identified into a lumbar vertebra coarse positioning neural network, wherein the lumbar vertebra coarse positioning neural network includes a backbone network and a three-dimensional spatial region regression network;

[0015] The backbone network includes an encoder network and a decoder network connected in sequence, and the backbone network is used to extract features from the original CT image and output a feature image with the same size as the original CT image;

[0016] The three-dimensional spatial region regression network is used to perform nonlinear regression on the characteristic image to obtain three-dimensional spatial coordinates representing the entire lumbar spine region.

[0017] Furthermore, the encoder network includes four network layers connected in sequence, wherein:

[0018] A first encoder network layer, comprising a first convolution processing module and a second convolution processing module connected in sequence;

[0019] A second encoder network layer includes a first convolutional pooling processing module, a third convolutional processing module, and a fourth convolutional processing module connected in sequence, wherein the output of the first convolutional processing module serves as the input of the first convolutional pooling processing module;

[0020] A third encoder network layer, comprising a second convolutional pooling processing module, a fifth convolutional processing module and a sixth convolutional processing module connected in sequence, wherein the output of the third convolutional processing module serves as the input of the second convolutional pooling processing module;

[0021] A fourth encoder network layer, comprising a third convolution and pooling processing module, a seventh convolution processing module, an eighth convolution processing module, a ninth convolution processing module and a first continuous upsampling module connected in sequence, wherein the output of the fifth convolution processing module is used as the input of the third convolution and pooling processing module;

[0022] The first to ninth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0023] The first convolution pooling processing module to the third convolution pooling processing module each include a three-dimensional convolution module, a BatchNorma operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence;

[0024] The first continuous upsampling module includes a plurality of continuous upsampling processing units, a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence.

[0025] Furthermore, the decoder network includes three network layers connected in sequence, wherein:

[0026] A first decoder network layer, comprising a first convolution upsampling module, a tenth convolution processing module, and a second continuous upsampling module connected in sequence, wherein the second convolution processing module and the first convolution upsampling module perform tensor splicing;

[0027] A second decoder network layer, comprising a second convolution upsampling module, an eleventh convolution processing module and a third continuous upsampling module connected in sequence, wherein the fourth convolution processing module and the second convolution upsampling module perform tensor splicing, and the output of the eleventh convolution processing module is used as the input of the first convolution upsampling module;

[0028] A third decoder network layer, comprising a third convolution upsampling module, a twelfth convolution processing module and a fourth continuous upsampling module connected in sequence, wherein the sixth convolution processing module and the third convolution upsampling module are tensor spliced, and the output of the twelfth convolution processing module is used as the input of the second convolution upsampling module;

[0029] The tenth convolution processing module, the eleventh convolution processing module and the twelfth convolution processing module each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0030] The first convolution upsampling module, the second convolution upsampling module and the third convolution upsampling module each include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence;

[0031] The second to fourth continuous upsampling modules include a plurality of continuous upsampling processing units, a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence.

[0032] Further, the three-dimensional spatial region regression network includes a thirteenth convolution processing module, a fourteenth convolution processing module, a fourth convolution pooling processing module, a fifteenth convolution processing module, a fifth convolution pooling processing module, a sixteenth convolution processing module, a Reshape tensor operation module and a fully connected layer connected in sequence;

[0033] Stacking the outputs of the first to fourth continuous upsampling modules, and using the stacked outputs as inputs of the thirteenth convolution processing module;

[0034] The thirteenth to sixteenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0035] The fourth convolutional pooling processing module and the fifth convolutional pooling processing module both include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module that are connected in sequence.

[0036] Further, the step of locating the spatial region of each lumbar vertebra in the local CT image of the entire lumbar vertebra region to obtain the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra specifically includes:

[0037] Inputting the local CT image of the whole lumbar region into each lumbar vertebra positioning network, wherein each lumbar vertebra positioning network includes a backbone network, a feature extraction network and a multi-three-dimensional spatial region regression network;

[0038] The backbone network includes an encoder network and a decoder network connected in sequence, and the backbone network is used to extract deep features from a local CT image of the entire lumbar spine area to locate the approximate spatial positions of the five vertebrae of the lumbar spine;

[0039] The feature extraction network is used to extract shallow features from the local CT image of the entire lumbar spine area;

[0040] The multi-three-dimensional spatial region regression network is used to perform nonlinear regression processing on the fused feature map obtained by adding the feature maps output by the backbone network and the feature extraction network pixel by pixel, so as to obtain the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra.

[0041] Furthermore, the encoder network includes four network layers connected in sequence, wherein:

[0042] A first encoder network layer, comprising a first convolution processing module and a second convolution processing module connected in sequence;

[0043] A second encoder network layer includes a first convolutional pooling processing module, a third convolutional processing module, and a fourth convolutional processing module connected in sequence, wherein the output of the second convolutional processing module serves as the input of the first convolutional pooling processing module;

[0044] A third encoder network layer, comprising a second convolutional pooling processing module, a fifth convolutional processing module and a sixth convolutional processing module connected in sequence, wherein the output of the third convolutional processing module serves as the input of the second convolutional pooling processing module;

[0045] A fourth encoder network layer, comprising a third convolutional pooling processing module, a seventh convolutional processing module, an eighth convolutional processing module and a ninth convolutional processing module connected in sequence, wherein the output of the fifth convolutional processing module serves as the input of the third convolutional pooling processing module;

[0046] The first to ninth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0047] The first convolution pooling processing module to the third convolution pooling processing module all include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence.

[0048] Furthermore, the decoder network includes three network layers connected in sequence,

[0049] A first decoder network layer, comprising a first convolution upsampling module, a tenth convolution processing module and an eleventh convolution processing module connected in sequence, wherein the second convolution processing module and the first convolution upsampling module perform tensor splicing;

[0050] A second decoder network layer, comprising a second convolution upsampling module and a twelfth convolution processing module connected in sequence, wherein the fourth convolution processing module and the second convolution upsampling module perform tensor splicing, and the output of the twelfth convolution processing module serves as the input of the first convolution upsampling module;

[0051] A third decoder network layer, comprising a third convolution upsampling module and a thirteenth convolution processing module connected in sequence, wherein the sixth convolution processing module and the third convolution upsampling module perform tensor splicing, and the output of the thirteenth convolution processing module serves as the input of the second convolution upsampling module;

[0052] The tenth to thirteenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0053] The first convolution upsampling module, the second convolution upsampling module and the third convolution upsampling module each include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence.

[0054] Further, the feature extraction network includes a fourteenth convolution processing module, a fourth convolution pooling processing module, a fifteenth convolution processing module, a sixteenth convolution processing module, a fourth convolution upsampling module and a seventeenth convolution processing module connected in sequence;

[0055] The fourteenth to seventeenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence;

[0056] The fourth convolution pooling processing module includes a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence;

[0057] The fourth convolution upsampling module includes a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence.

[0058] Furthermore, the multi-three-dimensional spatial region regression network includes a fifth convolution pooling processing module, a sixth convolution pooling processing module, a seventh convolution pooling processing module, an eighth convolution pooling processing module, a ninth convolution pooling processing module and a Reshape tensor operation module connected in sequence; the output end of the Reshape tensor operation module is respectively connected to the input end of 5 fully connected layers;

[0059] The outputs of the eleventh convolution processing module and the seventeenth convolution processing module are added pixel by pixel and used as the input of the fifth convolution pooling processing module;

[0060] The fifth convolutional pooling processing module to the ninth convolutional pooling processing module all include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence.

[0061] The present invention provides a lumbar CT image spatial positioning method based on an artificial neural network, which has the following beneficial effects:

[0062] (1) The lumbar vertebrae rough positioning neural network can realize the fusion of multi-scale feature information, which greatly improves the algorithm's spatial regional positioning accuracy of the lumbar vertebrae;

[0063] (2) The backbone network in each lumbar vertebra positioning neural network is responsible for locating the spatial position of each lumbar vertebra, and the feature extraction network is responsible for extracting the size information of each lumbar vertebra, thereby cooperating with each other to achieve accurate positioning of the spatial area of ​​each lumbar vertebra;

[0064] (3) A new loss function is proposed, which combines the advantages of MSELoss and 3D_CIOULoss and can quickly and effectively supervise the training of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0066] Figure 1 It is an overall schematic diagram of a lumbar CT image spatial positioning method provided by an embodiment of the present application;

[0067] Figure 2 is a flow chart of a lumbar CT image spatial positioning method provided by an embodiment of the present application;

[0068] Figure 3 This is a network structure diagram of a lumbar vertebrae rough positioning neural network provided by an embodiment of the present application;

[0069] Figure 4 is a network structure diagram of a continuous upsampling module provided by an embodiment of the present application;

[0070] Figure 5 This is a network structure diagram of a lumbar vertebra positioning neural network provided by an embodiment of the present application;

[0071] Figure 6 This is a processing effect diagram of the backbone network of the lumbar vertebra positioning neural network provided by an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0074] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0075] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0076] It should be noted that the directional words such as "upper", "lower", "left", and "right" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is formed "on" or "under" another element, it can not only be formed directly "on" or "under" another element, but also be formed "on" or "under" another element indirectly through an intermediate element.

[0077] In order to solve the problems existing in the background technology, the present invention uses LRP-Net (lumbar coarse positioning network) to spatially locate the entire lumbar area in CT images, LMD-Net (lumbar positioning network) to locate the spatial area of ​​each segment of the lumbar vertebrae, CUS_Block (continuous upsampling network module) to achieve multi-scale feature information fusion, and 3D-MseCIOULoss loss function for supervising network training, to achieve the spatial region of interest in CT images (defined by z min ,z max ,y min ,y max ,xmin ,x max To indicate the precise positioning of

[0078] The specific solutions are as follows:

[0079] See also Figure 1 An embodiment of the present invention provides a lumbar CT image spatial positioning method based on an artificial neural network, which is divided into two stages. First, the first stage realizes the positioning of the entire lumbar spatial area in the original CT image. The second stage, based on the first stage, realizes the precise positioning of each lumbar spatial area.

[0080] See also Figure 2 , the specific steps are as follows:

[0081] Step S101, spatially positioning the entire lumbar region in the original CT image to obtain three-dimensional spatial coordinates representing the entire lumbar region.

[0082] Specifically, the original CT image to be identified is input into the lumbar vertebra coarse positioning neural network to obtain the three-dimensional spatial coordinates representing the entire lumbar vertebra area.

[0083] See also Figure 3 In the present invention, the lumbar coarse positioning neural network is called LRP-Net network. LRP-Net consists of two parts, namely Backbone (backbone network) and 3D-Roi Regression (three-dimensional spatial region regression) network. The Backbone network is mainly used to extract features from CT images. It is improved on the basis of the U-Net network and mainly includes Encode (encoding) and Decode (decoding) parts. It can output feature images with the same size as the input image. The ROI is obtained by performing nonlinear regression on the feature map through the designed 3D-Roi Regression network. Lumber (Overall lumbar spine space area).

[0084] The Encode part of the Backbone network consists of 4 layers, each of which includes two convolutions and one MaxPooling. In order to extract features more effectively, a convolution operation is performed before MaxPooling. Since CT images are three-dimensional, the convolution and pooling operations in the network are all three-dimensional operations, which can more effectively utilize the three-dimensional spatial feature information in the CT image. LumberFor positioning, only features such as shape and HU value need to be extracted, and CT images are three-dimensional, so considering various factors, the number of feature channels in each layer of the Backbone network is the same. At the same time, a large number of experiments have proved that increasing the number of feature channels layer by layer does not significantly improve the positioning accuracy of this task. Also through experiments, it is found that when the number of feature channels is 32, it can have a good effect. Further increasing the number of feature channels does not significantly improve the effect, and it will greatly increase the amount of calculation, which will greatly increase the requirements for computer hardware.

[0085] The Decode part of the Backbone network performs one convolution operation and one upsampling operation per layer. Also, in order to extract features more effectively, a three-dimensional convolution operation is performed before each upsampling operation. Through the continuous convolution of the Encode and Decode parts of the Backbone, the feature information of the CT image can be effectively extracted. The Encode part extracts relatively shallow features, which contain sufficient spatial position information, while the Decode part extracts deeper semantic information. By stacking feature channels, the deep feature information can be fused with the shallow feature information to obtain a fused feature map with rich spatial position information and semantic information. In order to prevent the channel dimension from increasing continuously, a three-dimensional convolution operation is performed after splicing to restore the number of feature channels and achieve feature fusion.

[0086] Since the feature maps in each layer of the Decode part are feature maps of different depths, the feature information they possess is still different. However, due to the different sizes of the feature maps of each layer of the Decode, it is not possible to directly stack the feature channels for feature fusion operations. In order to achieve the stacking of feature channels of the feature maps of each layer, it is necessary to unify the size, but the pooling operation will lose information, so the size of the feature maps with smaller sizes needs to be enlarged. This embodiment adds a very simple structure, Continuous up-sampling block (CUS_Block), to the Backbone network. This structure quickly achieves the size expansion of the feature maps of each layer of the Decode through continuous upsampling, and unifies the feature maps of each layer to the same size as the input image. The structure of CUS_Block is not fixed, but is dynamically adjusted according to the scale transformation that needs to be achieved. Taking the feature image scale transformation from (32×16×16×16) to (4×112×112×112) as an example, the corresponding CUS_Block related structure can be found in Figure 3 .

[0087] For further information, see Figure 4, the structure of CUS_Block is not fixed. The number of 3D upsampling modules is determined by the size of the input feature image and the size of the output feature image. Assuming there are 3 upsampling modules, the first two of them enlarge the image size by twice, and the last one is for size fine-tuning, so as to achieve multiple continuous upsampling to enlarge the original feature image to the target size;

[0088] For further information, see Figure 4 ,The CUS_Block module includes N continuous upsampling and a CBR module. Continuous upsampling is used to scale the ,image, and the CBR module is used for feature integration. ,The input feature image is first upsampled to the target size and then ,is integrated through the CBR module.

[0089] Through many experiments, it is found that the size expansion achieved by multiple consecutive upsampling with a smaller scale and the size expansion achieved by a single upsampling with a larger scale have better effects in this task. In order to prevent the loss of feature information due to multiple consecutive upsampling, it is necessary to perform a three-dimensional convolution operation on the feature map obtained by multiple consecutive upsampling, and set the number of its output feature channels to CUS_Count. After unifying the size of the feature maps of each layer in the Decode part, the feature channel dimension of the four feature maps with exactly the same size is stacked to obtain a feature fusion feature map with a channel number of 4×CUS_Count. In the three-dimensional spatial region regression 3D-Roi Regression network, convolution and Max Pooling operations are performed alternately to condense the feature information and shrink the size of the feature image, thereby reducing the amount of calculation for the subsequent nonlinear regression of the fully connected layer. After multiple pooling operations, the size of the feature map is reduced to 4×56×28×28. Finally, nonlinear regression is achieved through three layers of fully connected layers to obtain the ROI representing the overall spatial region of the lumbar spine. Lumber The three-dimensional space coordinates (z min ,z max ,y min ,y max ,x min ,x max ).

[0090] The LRP-Net of this embodiment involves the following network processing modules, and the meanings of each network processing module are as follows:

[0091] 1. CBR module (convolution processing module, a total of 16). The CBR module includes a three-dimensional convolution module, a BatchNormal operation module, and a ReLU activation function connected in sequence. The CBR module affects the number of input and output channels but does not affect the size of the input and output (for example: the input shape is bs×1×224×112×112=>bs×32×224×112×112).

[0092] 2. CBRP module (convolution and pooling processing module, a total of 5). The CBRP module includes a 3D convolution module, a BatchNormal operation module, a ReLU activation function, and a 3D maximum pooling operation module. The CBRP module does not affect the number of input and output channels, but the output size is half of the input size (for example: the input shape is bs×32×224×112×112=>bs×32×112×56×56).

[0093] 3. CBRU module (convolution upsampling module, 3 in total). The CBRU module includes a 3D convolution module, a BatchNormal operation module, a ReLU activation function, and a 3D upsampling operation module. The CBRP module does not affect the number of input and output channels, but the output size is twice the input size (for example: the input shape is bs×32×112×56×56=>bs×32×224×112×112).

[0094] 4. CUS module (continuous upsampling module, 4 in total). The CUS module is the CUS_Block module, which includes multiple continuous upsampling processing units connected in sequence and a CBR module. That is, the CUS module includes multiple continuous upsampling processing units connected in sequence, a three-dimensional convolution module, a BatchNormal operation module, and a ReLU activation function.

[0095] 5. The Roi_Block module is a fully connected layer.

[0096] 6. Reshape is a tensor operation and a specific processing method in the field of neural networks.

[0097] See also Figure 3 ,The specific processing flow of LRP-Net (lumbar vertebrae rough positioning neural network) is introduced in detail below.

[0098] First, the size of the image input to the LRP-Net network of this embodiment is 224×112×112, so the original CT image needs to be resampled to the target size, but the image data needs to be converted into a tensor with a shape of bs×1×224×112×112.

[0099] Then, the resampled CT image is input into the network and processed by a CBR module (convolution first, then the result of the convolution is processed with BatchNormal, and the result of the processing is processed by the ReLU activation function to obtain an output) to obtain the output as the input of the next CBR module and CBRP module (a three-dimensional maximum pooling operation is added on the basis of the CBR module). After two CBR module processings, the shape of the feature image X1 is bs×32×224×112×112, and the shape of X2 obtained after one CBR and one CBRP processing is bs×32×112×56×56. In this way, one layer of operation of the encoder part is completed, including two CBR modules and one CBRP module, and two outputs X1 and X2 are obtained, of which X2 is used as the input of the second layer.

[0100] Furthermore, the operations of the first, second and third layers of the Encode part are exactly the same, and the fourth layer includes three CBR modules and one CUS module.

[0101] Furthermore, the operation of the fourth layer of Encode is: the output of the third layer is used as the input of the fourth layer (shape is bs×32×28×14×14), and after three CBR module operations, the output X3 is obtained, and its shape is bs×32×28×14×14. X3 is used as the input of the CUS module and processed to obtain X4, and the shape of X4 is bs×5×224×112×112.

[0102] Furthermore, the Decode part includes three layers, each layer includes a CBRU module, a CBR module and a CUS module, and the operations are exactly the same. Taking the first layer as an example, the output X3 of the fourth layer of the Encode part is taken as input, and after processing by the CBRU module, X5 is obtained, whose shape is bs×32×56×28×28. X5 is stacked with the output of the third layer of the corresponding Decode part to obtain a tensor with a shape of bs×64×56×28×28 as the input of CBR, and its shape is processed into bs×32×56×28×28 by the CBR module. The output of CBR is used as the input of CUS and the next layer of CBRU. After processing by CUS, X6 is obtained, whose shape is bs×5×224×112×112, and the operations of the subsequent layers are also the same.

[0103] Finally, the outputs of the four CUS (all with shape of bs×5×224×112×112) are stacked to obtain X7, whose shape is bs×20×224×112×112, and it is used as the input of the 3D-Roi Regression (three-dimensional spatial region regression) subnetwork. First, it is processed by two CBR modules and the number of channels is compressed to obtain a feature image with a shape of bs×8×224×112×112, which is used as the subsequent input. The subsequent operations alternately perform CBR and CBRP operations to obtain a feature image with a shape of bs×4×56×28×28, and then perform a reshape operation to transform its shape into a tensor of bs×4×43904. Finally, it is processed by the Roi_Block module (i.e., the fully connected layer) to obtain the overall regional coordinate information of the lumbar spine in the shape of bs×6.

[0104] Step S102, using the three-dimensional spatial coordinates representing the entire lumbar region to crop the original CT image to obtain a local CT image of the entire lumbar region;

[0105] Step S103, positioning the spatial region of each lumbar vertebra in the local CT image of the entire lumbar vertebra region to obtain three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra;

[0106] Specifically, the local CT image of the entire lumbar spine is input into the lumbar localization network LMD-Net of each segment, see Figure 5 The lumbar vertebra positioning network LMD-Net includes a backbone network (Backbone), a feature extraction network (FeatureExtraction) and a multi-three-dimensional spatial region regression network (Multi 3D-Roi Regression).

[0107] The backbone network includes an encoder network and a decoder network connected in sequence, and the backbone network is used to extract deep features from a local CT image of the entire lumbar spine area to locate the approximate spatial positions of the five vertebrae of the lumbar spine;

[0108] The feature extraction network (Feature Extraction) is used to extract shallow features from the local CT image of the entire lumbar spine area;

[0109] The multi 3D-Roi Regression network is used to perform nonlinear regression processing on the fused feature map obtained by adding the feature maps output by the backbone network and the feature extraction network pixel by pixel, so as to obtain the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebra.

[0110] It should be pointed out that the backbone network of LMD-Net is roughly the same as that of LRP-Net, with the following three main differences:

[0111] (1) The number of feature channels is 48. Because LMD-Net needs to realize ROI L1 , ROI L2 …ROI L5 For accurate positioning, it is necessary to extract richer features, so the number of feature channels of LRP-Net needs to be improved to a certain extent.

[0112] (2) When the feature maps corresponding to the Encode and Decode parts are fused, it is achieved by pixel-by-pixel addition, not by channel stacking. Since the LMD-Net structure is more complex than the LRP-Net, and the input image sizes of the two networks are very different, if the feature channel dimension is stacked, the hardware level requirements are extremely high, so it is chosen to perform pixel-by-pixel addition to achieve feature fusion.

[0113] (3) The four layers of feature maps are not resized and stacked for feature fusion. In LMD-Net, feature fusion is achieved by adding the feature maps output by the FeatureExtraction network to the feature maps output by the Backbone network pixel by pixel. Compared with resizing the feature maps of each layer in the Decode part of Backbone and then performing feature fusion, L1 , ROI L2 …ROI L5 Performed better in detection tasks.

[0114] Furthermore, the Backbone in the LMD-Net network is mainly used to roughly locate the spatial position of each lumbar vertebra. The Backbone network is used to perform heat map regression on the lumbar landmark (the center point of the vertebral canal close to the cone), and the MSELoss (mean square error loss function) is used to minimize the difference between the network predicted heat map and the target heat map. Since the Backbone provides a certain "inspiration" for the position of the lumbar spatial area, the Backbone network needs to be pre-trained before the overall LMD-Net training.

[0115] Furthermore, the feature map output by the Backbone network of the LMD-Net network is a feature map with stronger features in the area where the lumbar landmark is located, expressing the precise spatial position information of the lumbar spine, but the ability to express the size information of the lumbar spine is weak. Therefore, it is necessary to extract the size information of the lumbar spine in the local CT image through the Feature Extraction network. By adding the feature maps output by the Backbone network and the Feature Extraction network pixel by pixel, the position feature and the size feature can be fused, thereby obtaining a feature map that can be used for ROI. L1 , ROI L2 …ROI L5 Positioning feature map. The processing effect of the Backbone network of the LMD-Net network can be found in Figure 6 .

[0116] Furthermore, the Multi 3D-Roi Regression network of the LMD-Net network first performs convolution and Max Pooling operations alternately on the fused feature maps processed by the Backbone and Feature Extraction networks to obtain a 6×3×3 feature map with 5 channels. Finally, three fully connected layers are used to perform nonlinear regression on the feature maps of each channel to obtain ROI L1 , ROI L2 …ROI L5 .

[0117] The LMD-Net network can effectively reduce the impact of image deformation on the spatial positioning accuracy of each lumbar vertebrae segment. In addition, the overall size of the lumbar spine of each patient is limited. Therefore, the input of the LMD-Net network is a CT image of size (192×96×96) obtained by resampling the local CT image containing only the overall lumbar spine area.

[0118] The network processing module involved in the LMD-Net network of this embodiment is basically the same as the network processing module of the LRP-Net network, and the structure and meaning of each network processing module can refer to each other.

[0119] The detailed processing flow of the LMD-Net network is as follows:

[0120] First, the size of the image input to the LMD-Net network is 192×96×96, so the original CT image needs to be resampled to the target size, but the image data needs to be converted into a tensor with a shape of bs×1×192×96×96.

[0121] In the backbone network:

[0122] The resampled CT image is input into the network and processed by a CBR module (convolution first, then the result of the convolution is processed by BatchNormal, and the result of the processing is processed by the ReLU activation function to obtain an output) to obtain the output as the input of the next CBR module and CBRP module (a three-dimensional maximum pooling operation is added on the basis of the CBR module). After two CBR module processings, the shape of the feature image X1 is bs×48×192×96×96, and the shape of X2 obtained after one CBR and one CBRP processing is bs×48×96×48×48. In this way, one layer of operation of the Decode part is completed, including two CBR modules and one CBRP module, and two outputs X1 and X2 are obtained, of which X2 is used as the input of the second layer.

[0123] Furthermore, the operations of the first, second, and third layers of the Encode part are exactly the same, and the fourth layer includes three CBR modules;

[0124] Furthermore, the operation of the fourth layer of Encode is: the output of the third layer is used as the input of the fourth layer (shape is bs×48×24×12×12), and after three CBR module operations, the output X3 is obtained, whose shape is bs×48×24×12×12.

[0125] Furthermore, the Decode part includes three layers, each layer includes a CBRU module and a CBR module, and the operations are exactly the same. Taking the first layer as an example, the output X3 of the fourth layer of the Encode part is taken as input, and processed by the CBRU module to obtain X5, whose shape is bs×48×48×24×24. X5 is added pixel by pixel to the output of the third layer of the corresponding Decode part, and the obtained tensor with a shape of bs×48×48×24×24 is used as the input of CBR. The CBR module processes its shape into bs×48×48×24×24, and then the output of CBR is used as the input of the CBRU of the next layer. The operations of the subsequent layers are also the same.

[0126] Finally, the third layer of the Decode part of the Backbone network outputs a feature image with a shape of bs×48×192×96×96, which is input into a CBR module for processing to obtain a feature image X6 with a shape of bs×5×192×96×96, where the number of channels 5 corresponds to 5 lumbar vertebrae, and each channel is responsible for locating one lumbar vertebra.

[0127] In the Feature Extraction network:

[0128] The resampled CT image is passed into Feature Extraction and first processed by the CBR module to obtain a feature image with a shape of bs×5×192×96×96, and then processed by the CBRP module to obtain a feature image with a shape of bs×5×96×48×48, and then processed by the CBR module to obtain a feature image with a shape of bs×5×96×48×48, and then processed by the CBRU module to obtain a feature image with a shape of bs×5×192×96×96, and then processed by the CBR module to obtain a feature image X7 with a shape of bs×5×192×96×96.

[0129] In the Multi 3D-Roi Regression network:

[0130] X6 and X7 are added pixel by pixel and input into Multi 3D-Roi Regression and processed through 5 CBRP modules to reduce the feature image size to bs×5×6×3×3.

[0131] Then, the tensor is reshaped to obtain a feature tensor with a shape of bs×5×54. Each channel of this feature tensor is subjected to nonlinear regression using the Roi_Block module (i.e., fully connected layer) to obtain the three-dimensional coordinates of the spatial regions of the five lumbar vertebrae.

[0132] Step S104, using the three-dimensional spatial coordinates representing the spatial regions of each lumbar vertebra to crop the local CT image of the entire lumbar vertebra area, to obtain a local CT image of the spatial region of each lumbar vertebra.

[0133] The spatial positioning method of the above-mentioned lumbar CT image in this embodiment adopts the lumbar coarse positioning neural network LRP-Net and the lumbar segment positioning neural network LMD-Net, which are different from the prior art. Among them, the lumbar coarse positioning neural network can realize multi-scale feature information fusion, which greatly improves the algorithm's spatial regional positioning accuracy of the lumbar spine; the backbone network in the lumbar segment positioning neural network is responsible for locating the spatial position of each lumbar segment, and the feature extraction network is responsible for extracting the size information of each lumbar segment, so as to cooperate with each other to realize the precise positioning of the spatial area of ​​each lumbar segment.

[0134] Another embodiment of the present invention further proposes a 3D-MseCIOULoss loss function that is different from the prior art and can effectively supervise the training of the LRP-Net network and the LMD-Net network.

[0135] The 3D-MseCIOULoss loss function is calculated as follows:

[0136]

[0137] in:

[0138]

[0139] in:

[0140]

[0141]

[0142] in:

[0143]

[0144] in:

[0145] MSELoss is the mean square error loss function;

[0146] is the weight coefficient of the mean square error function;

[0147] β is the weight coefficient of 3D_CIOULoss function;

[0148] 3D_IOU represents the intersection-over-union ratio between the 3D target region and the 3D region predicted by the network;

[0149] Dis Cen Represents the distance between the coordinates of the center point of the 3D target area and the coordinates of the center point of the 3D area predicted by the network;

[0150] Dis Cor Represents the distance between the vertex coordinates of the 3D target area and the vertex coordinates of the 3D area predicted by the network;

[0151] ZDis Pre The length of the three-dimensional area in the z direction predicted by the network;

[0152] YDis Pre The length in the y direction of the three-dimensional area predicted by the network;

[0153] XDis Pre The length in the x direction of the three-dimensional region predicted by the network;

[0154] ZDis Tar is the length of the three-dimensional target area in the z direction;

[0155] YDis Tar is the length of the three-dimensional target area in the y direction;

[0156] XDis Tar is the length of the three-dimensional target area in the x direction;

[0157] V Inter Spatial region cuboid ROI predicted by the network Pre The cuboid ROI with the actual target space area Tar Intersection ROI Inter The volume of the intersection of

[0158] V Pre ROI Pre Volume; V Tar ROI Tar volume.

[0159] The purpose of the 3D-MseCIOULoss loss function proposed in this embodiment is to calculate the ROI predicted by the network during the network training process. Pre ROI with real space area Tar The difference is obtained by using back propagation to supervise the optimization network and reduce the difference. 3D-CIOULoss is a three-dimensional expansion of CIOULoss. Pre and ROI Tar The feature information such as the intersection-over-union ratio, center distance, and shape difference between them can effectively supervise the network to reduce ROI Pre and ROI Tar The difference between them can improve the positioning accuracy of the network. Since 3D_CIOULoss supervises network learning through geometric features, in the early stage of network training, when the network is not initialized correctly, 3D_CIOULoss cannot effectively supervise network learning. MSELoss can directly supervise and reduce ROI Pre and ROI Tar Therefore, it can effectively supervise network learning in the early stage of network training. However, in the middle and late stages of network training, MSELoss does not consider the spatial relationship between the coordinates, so its supervision effect on the network is not as good as 3D-CIOULoss. Therefore, 3D-MseCIOULoss combines the advantages of MSELoss in the early stage of network training and the advantages of 3D-CIOULoss in the middle and late stages of network training to obtain a loss function that has good effects in all stages of network training. By adjusting the weight coefficients α and β, the influence of MSELoss and 3D_CIOULoss on network training and learning can be adjusted. Since MSELoss solves the problem of 3D_CIOULoss's difficulty in supervising network training in the early stage of network training, it is generally set and The settings are usually very small.

[0160] The 3D-MseCIOULoss loss function proposed in this embodiment can quickly and effectively supervise the training of the neural network.

[0161] The above description is only a preferred embodiment of the present invention. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other to form a technical solution.

Claims

1. A lumbar CT image spatial positioning method based on artificial neural network, characterized in that: include: Inputting the original CT image to be identified into a lumbar vertebra coarse positioning neural network, the lumbar vertebra coarse positioning neural network comprising a first backbone network and a three-dimensional spatial region regression network; wherein the first backbone network comprises an encoder network and a decoder network connected in sequence, the first backbone network is used to extract features from the original CT image and output a feature image of the same size as the original CT image; The three-dimensional spatial region regression network is used to perform nonlinear regression on the characteristic image to obtain three-dimensional spatial coordinates representing the entire lumbar region; Using the three-dimensional spatial coordinates representing the entire lumbar region to crop the original CT image, to obtain a local CT image of the entire lumbar region; Inputting the local CT image of the whole lumbar area into the positioning network of each lumbar vertebra, the positioning network of each lumbar vertebra includes a second backbone network, a feature extraction network and a multi-three-dimensional spatial region regression network; the second backbone network includes an encoder network and a decoder network connected in sequence, the second backbone network is used to perform deep feature extraction on the local CT image of the whole lumbar area to locate the approximate spatial position of the five lumbar spines; the feature extraction network is used to perform shallow feature extraction on the local CT image of the whole lumbar area; the multi-three-dimensional spatial region regression network is used to perform nonlinear regression processing on the fused feature map obtained by adding the feature map output by the second backbone network and the feature extraction network pixel by pixel, so as to obtain the three-dimensional spatial coordinates representing the spatial area of ​​each lumbar vertebra; The local CT image of the entire lumbar vertebrae region is cropped using the three-dimensional spatial coordinates representing the spatial region of each lumbar vertebrae to obtain a local CT image of the spatial region of each lumbar vertebrae.

2. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 1, characterized in that: The encoder network in the first backbone network includes four network layers connected in sequence, wherein: A first encoder network layer, comprising a first convolution processing module and a second convolution processing module connected in sequence; A second encoder network layer includes a first convolutional pooling processing module, a third convolutional processing module, and a fourth convolutional processing module connected in sequence, wherein the output of the first convolutional processing module serves as the input of the first convolutional pooling processing module; A third encoder network layer, comprising a second convolutional pooling processing module, a fifth convolutional processing module and a sixth convolutional processing module connected in sequence, wherein the output of the third convolutional processing module serves as the input of the second convolutional pooling processing module; A fourth encoder network layer, comprising a third convolution and pooling processing module, a seventh convolution processing module, an eighth convolution processing module, a ninth convolution processing module and a first continuous upsampling module connected in sequence, wherein the output of the fifth convolution processing module is used as the input of the third convolution and pooling processing module; The first to ninth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The first convolution pooling processing module to the third convolution pooling processing module each include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence; The first continuous upsampling module includes a plurality of continuous upsampling processing units, a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence.

3. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 2, characterized in that: The decoder network in the first backbone network includes three network layers connected in sequence, wherein: A first decoder network layer, comprising a first convolution upsampling module, a tenth convolution processing module, and a second continuous upsampling module connected in sequence, wherein the second convolution processing module and the first convolution upsampling module perform tensor splicing; A second decoder network layer, comprising a second convolution upsampling module, an eleventh convolution processing module and a third continuous upsampling module connected in sequence, wherein the fourth convolution processing module and the second convolution upsampling module perform tensor splicing, and the output of the eleventh convolution processing module is used as the input of the first convolution upsampling module; A third decoder network layer, comprising a third convolution upsampling module, a twelfth convolution processing module and a fourth continuous upsampling module connected in sequence, wherein the sixth convolution processing module and the third convolution upsampling module are tensor spliced, and the output of the twelfth convolution processing module is used as the input of the second convolution upsampling module; The tenth convolution processing module, the eleventh convolution processing module and the twelfth convolution processing module each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The first convolution upsampling module, the second convolution upsampling module and the third convolution upsampling module each include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence; The second to fourth continuous upsampling modules include a plurality of continuous upsampling processing units, a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence.

4. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 3, characterized in that: The three-dimensional spatial region regression network includes a thirteenth convolution processing module, a fourteenth convolution processing module, a fourth convolution pooling processing module, a fifteenth convolution processing module, a fifth convolution pooling processing module, a sixteenth convolution processing module, a Reshape tensor operation module and a fully connected layer connected in sequence; Stacking the outputs of the first to fourth continuous upsampling modules, and using the stacked outputs as inputs of the thirteenth convolution processing module; The thirteenth to sixteenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The fourth convolutional pooling processing module and the fifth convolutional pooling processing module both include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module that are connected in sequence.

5. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 1, characterized in that: The encoder network in the second backbone network includes four network layers connected in sequence, wherein: A first encoder network layer, comprising a first convolution processing module and a second convolution processing module connected in sequence; A second encoder network layer includes a first convolutional pooling processing module, a third convolutional processing module, and a fourth convolutional processing module connected in sequence, wherein the output of the second convolutional processing module serves as the input of the first convolutional pooling processing module; A third encoder network layer, comprising a second convolutional pooling processing module, a fifth convolutional processing module and a sixth convolutional processing module connected in sequence, wherein the output of the third convolutional processing module serves as the input of the second convolutional pooling processing module; A fourth encoder network layer, comprising a third convolutional pooling processing module, a seventh convolutional processing module, an eighth convolutional processing module and a ninth convolutional processing module connected in sequence, wherein the output of the fifth convolutional processing module serves as the input of the third convolutional pooling processing module; The first to ninth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The first convolution pooling processing module to the third convolution pooling processing module all include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence.

6. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 5, characterized in that: The decoder network in the second backbone network includes three network layers connected in sequence. A first decoder network layer, comprising a first convolution upsampling module, a tenth convolution processing module and an eleventh convolution processing module connected in sequence, wherein the second convolution processing module and the first convolution upsampling module perform tensor splicing; A second decoder network layer, comprising a second convolution upsampling module and a twelfth convolution processing module connected in sequence, wherein the fourth convolution processing module and the second convolution upsampling module perform tensor splicing, and the output of the twelfth convolution processing module serves as the input of the first convolution upsampling module; A third decoder network layer, comprising a third convolution upsampling module and a thirteenth convolution processing module connected in sequence, wherein the sixth convolution processing module and the third convolution upsampling module perform tensor splicing, and the output of the thirteenth convolution processing module serves as the input of the second convolution upsampling module; The tenth to thirteenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The first convolution upsampling module, the second convolution upsampling module and the third convolution upsampling module each include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence.

7. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 6, characterized in that: The feature extraction network includes a fourteenth convolution processing module, a fourth convolution pooling processing module, a fifteenth convolution processing module, a sixteenth convolution processing module, a fourth convolution upsampling module and a seventeenth convolution processing module connected in sequence; The fourteenth to seventeenth convolution processing modules each include a three-dimensional convolution module, a BatchNormal operation module and a ReLU activation function connected in sequence; The fourth convolution pooling processing module includes a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence; The fourth convolution upsampling module includes a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional upsampling operation module connected in sequence.

8. The method for spatial positioning of lumbar CT images based on artificial neural network according to claim 7, characterized in that: The multi-three-dimensional spatial region regression network comprises a fifth convolution pooling processing module, a sixth convolution pooling processing module, a seventh convolution pooling processing module, an eighth convolution pooling processing module, a ninth convolution pooling processing module and a Reshape tensor operation module which are connected in sequence; the output end of the Reshape tensor operation module is respectively connected to the input end of 5 fully connected layers; The outputs of the eleventh convolution processing module and the seventeenth convolution processing module are added pixel by pixel as the input of the fifth convolution pooling processing module; The fifth convolutional pooling processing module to the ninth convolutional pooling processing module all include a three-dimensional convolution module, a BatchNormal operation module, a ReLU activation function and a three-dimensional maximum pooling operation module connected in sequence.

Images