A system and method for lumbar sacral positioning and key point detection based on deep learning

Abstract

The present invention relates to a lumbar sacral positioning and key point detection system and method based on deep learning. A lumbar vertebra detection device is set to achieve preliminary positioning of the lumbar sacral region, and a post-processing device is set to obtain relatively accurate lumbar sacral region information. On this basis, a key point detection device is set to extract vertebral semantic features based on contextual information of the lumbar sacral region displayed in a fusion image, and the vertebral semantic features are decoupled through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebra, thereby capturing the spatial distribution information of the lumbar key points and achieving accurate positioning of the key points.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a lumbar sacral positioning and key point detection system and method based on deep learning. Background Art

[0002] For automatic key point detection of the spine in patients' lumbar X-ray images, deep learning-based methods have shown great potential. This method has various forms and can be mainly divided into two categories: two-stage detection methods and single-stage detection methods.

[0003] In the two-stage detection method, each vertebra is usually detected first, and the detection area is delineated, and then the corner point detection of the vertebra is performed based on the detection area. In the single-stage detection method, it is usually divided into three methods: coordinate regression-based, heat map regression-based, and center point plus offset-based. Whether it is a two-stage detection method or a single-stage detection method, the lumbar spine X-ray image based on it usually needs to show the information of the entire spine, and the vertebral body information in the lumbar spine X-ray image is basically the information required for the full spine sampling task. The number of vertebrae is fixed, and the processing is relatively simpler.

[0004] However, in the detection task of lumbar stability and slippage degree, the lumbar X-ray images usually taken in clinical practice show five lumbar vertebrae, one sacrum and an unfixed number of thoracic vertebrae, among which the area where the five lumbar vertebrae are located (called the lumbar area) and the area where the one sacrum is located (called the sacral area) constitute the lumbar sacral area. In the process of completing the lumbar stability and slippage degree detection task, only the vertebral information of the lumbar sacral area in the lumbar X-ray image needs to be paid attention to. The area where the unfixed number of thoracic vertebrae is located is redundant information. This redundant information will interfere with the key point detection task, and ultimately lead to the defect of overall misalignment of the key point detection, thereby increasing the difficulty of the lumbar stability and lumbar slippage degree detection task. At the same time, the lumbar sacral area only contains local information of the spine, and there is less contextual information available for reference. It is difficult to achieve high-precision key point detection with less information. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to overcome the technical defects of the prior art that the detection accuracy of lumbar stability and lumbar spondylolisthesis is low due to the existence of redundant information. In order to overcome the defects of the above prior art, the present invention provides a lumbar sacral positioning and key point detection system and method based on deep learning, including a lumbar sacral positioning and key point detection system based on deep learning and a lumbar sacral positioning and key point detection method based on deep learning.

[0006] The present invention provides a lumbar sacral positioning and key point detection system based on deep learning, comprising:

[0007] The lumbar vertebra detection device is configured to perform a convolution operation on the lumbar vertebra X-ray image to extract a plurality of feature maps with different hierarchical features, and fuse these feature maps into a plurality of resolution images showing feature information of targets of different scales by combining the convolution operation with feature pyramid pooling, and then locate the lumbar sacral region on each of the resolution images respectively to obtain the corresponding lumbar sacral region images;

[0008] A post-processing device is configured to perform confidence screening and non-maximum suppression processing on the positioning of the lumbar sacral region on all lumbar vertebrae sacral region images obtained by the lumbar vertebrae detection device, and then perform image fusion to obtain a fused image;

[0009] A key point detection device is configured to extract vertebral semantic features based on context information of the lumbar sacral region displayed in the fused image, and decouple the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebral body, and obtain the corner point coordinates between the vertebrae;

[0010] The post-processing device communicates with the lumbar vertebra detection device, and the key point detection device communicates with the post-processing device.

[0011] The deep learning-based lumbar sacral positioning and key point detection system disclosed in the present invention can achieve the following beneficial effects:

[0012] First, by setting up a lumbar vertebra detection device, the preliminary positioning of the lumbar sacral region can be achieved. Since the lumbar vertebra detection device uses a combination of convolution operation and feature pyramid pooling, this method is robust to lumbar X-ray images of uncertain vertebrae, can eliminate the interference of irrelevant redundant information, and can effectively utilize the prior knowledge of the lumbar vertebrae to obtain the features of the lumbar sacral region from the global image, providing accurate lumbar sacral region information for subsequent key point detection tasks.

[0013] Secondly, more accurate lumbar sacral region information can be obtained by setting up a post-processing device. Since the post-processing device performs confidence screening on all the obtained lumbar sacral region images, the lumbar sacral region positioning with a low confidence level can be removed, and the repeated positioning of the lumbar sacral region can be deleted through non-maximum suppression processing, so that the fused image obtained after the image fusion can more accurately display the lumbar sacral region information.

[0014] Finally, in order to achieve high-precision key point detection in the lumbar sacral region (where there is less contextual information available for reference), the present invention further sets up a key point detection device, which effectively improves the ability to extract global contextual information from detailed information in the image. Since the key point detection device extracts the vertebral semantic features based on the contextual information of the lumbar sacral region displayed in the fused image, and decouples the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebra, it can not only simultaneously obtain low-level features rich in details and high-level features rich in semantics, but also capture the spatial distribution information of the lumbar key points, improve the robustness of the system operation, and achieve accurate positioning of the key points.

[0015] In a possible implementation, the lumbar vertebra detection device includes:

[0016] A backbone network module is configured to perform a convolution operation on the lumbar spine X-ray image to extract a plurality of feature maps with different hierarchical features;

[0017] The neck network module is configured to fuse all feature maps obtained by the backbone network module into a plurality of resolution images showing feature information of objects of different scales by combining convolution operation with feature pyramid pooling;

[0018] The detection head module is configured to locate the lumbar vertebrae sacrum region for each of the resolution images respectively, and obtain the corresponding lumbar vertebrae sacrum region images;

[0019] The neck network module communicates with the backbone network module, the detection head module communicates with the neck network module, and the key point detection device communicates with the detection head module;

[0020] This scheme can achieve multi-level and multi-scale feature fusion, and thus more accurately detect and locate the lumbar sacral region. It is robust to spinal X-ray images with an uncertain number of vertebrae and provides accurate lumbar sacral region information for subsequent key point detection tasks.

[0021] In a possible implementation, the key point detection device includes:

[0022] A high-resolution network module is configured to perform feature encoding on the fused image to obtain a feature Figure 1 ,feature Figure 2 ,feature Figure 3 and Features Figure 4 , and the fused image, the feature Figure 1 , the characteristics Figure 2 , the characteristics Figure 3 and the characteristics Figure 4The image sizes are the first five terms in the geometric sequence whose common ratio is less than 1;

[0023] decoder, is set to the feature Figure 1 , the characteristics Figure 2 , the characteristics Figure 3 and the characteristics Figure 4 Performing upsampling and feature fusion to extract the vertebral semantic features;

[0024] A mapping module is configured to decouple the vertebral semantic features through three independent decoupling mappings to extract the center point information, the center point offset information and the corner point offset information of the vertebral body;

[0025] A calculation module, configured to obtain the coordinates of corner points between vertebrae based on the center point information, the center point offset information and the corner point offset information;

[0026] The high-resolution network module communicates with the post-processing device, the decoder communicates with the high-resolution network module, the mapping module communicates with the decoder, and the calculation module communicates with the mapping module;

[0027] This scheme forms an encoder and decoder structure based on a high-resolution network module, which can extract the vertebral semantic information contained in the image. Subsequently, by setting up a mapping module, the vertebral semantic features are decoupled with three independent decoupled mappings, and the vertebral center point information, center point offset information and offset information of the corner points between vertebrae are extracted, thereby achieving the purpose of calculating the corner point coordinates by determining the vertebral center point and corner point offset information, weakening the interference information such as background and noise, and increasing the accuracy and robustness of the key point coordinate estimation.

[0028] In a possible implementation manner, the high-resolution network module is a network structure having four network branches, wherein the first network branch is used to obtain the feature Figure 1 , the second network branch is used to obtain the characteristics Figure 2 , the third network branch is used to obtain the characteristics Figure 3 , the fourth network branch is used to obtain the characteristics Figure 4 , the output end of each network branch communicates with the decoder; thereby achieving frequent information exchange between feature maps of different resolutions and information fusion of each network branch, which can excel in maintaining high-resolution features and multi-scale feature fusion, and can also effectively capture global and local information, reflecting the overall shape of the lumbar sacrum and the local correlation between vertebrae, significantly improving the quality and accuracy of feature extraction.

[0029] In a possible implementation, the decoder is a communication network structure formed by three sub-decoders connected in series, and each of the sub-decoders includes an upsampling module, a multi-scale attention module, and a convolutional rectification module;

[0030] In each of the sub-decoders, the multi-scale attention module communicates with the upsampling module, and the convolutional rectification module communicates with the multi-scale attention module;

[0031] In the first sub-decoder, the upsampling module communicates with the output end of the fourth network branch in the high-resolution network module, the multi-scale attention module also communicates with the output end of the third network branch in the high-resolution network module, and the upsampling module is configured to Figure 4 An upsampling operation is performed to obtain a first upsampled image, and the multi-scale attention module is configured to combine the first upsampled image with the feature in a feature fusion manner combining channel splicing and pooling. Figure 3 Performing feature fusion to obtain a first fused image, the convolution rectification module being configured to sequentially perform convolution processing and linear rectification function mapping processing on the first fused image to obtain a first semantic feature image;

[0032] In the sub-decoder located in the middle, the upsampling module communicates with the convolutional rectifier module in the sub-decoder located in the first position, the multi-scale attention module also communicates with the output end of the second network branch in the high-resolution network module, the upsampling module is configured to perform an upsampling operation on the first semantic feature image to obtain a second up-sampled image, and the multi-scale attention module is configured to combine the second up-sampled image with the feature image in a feature fusion method combining channel splicing and pooling. Figure 2 Performing feature fusion to obtain a second fused image, the convolution rectification module being configured to sequentially perform convolution processing and linear rectification function mapping processing on the second fused image to obtain a second semantic feature image;

[0033] In the sub-decoder located at the end, the upsampling module communicates with the convolutional rectifier module in the sub-decoder located in the middle, the multi-scale attention module also communicates with the output end of the first network branch in the high-resolution network module, the mapping module communicates with the convolutional rectifier module, the upsampling module is configured to perform an upsampling operation on the second semantic feature image to obtain a third upsampled image, and the multi-scale attention module is configured to combine the third upsampled image with the feature image in a feature fusion method combining channel splicing and pooling. Figure 1Performing feature fusion to obtain a third fused image, wherein the convolution rectification module is configured to sequentially perform convolution processing and linear rectification function mapping processing on the third fused image to obtain the vertebral semantic features;

[0034] The multi-scale attention module is introduced into each layer of the decoder formed by this scheme, which can realize feature fusion and feature selection of deep and shallow features in each layer, make full use of the context information of the lumbar sacral area to extract relevant features such as the vertebral center area and lumbar corner points, further improve the efficiency of obtaining the spatial distribution information of the lumbar spine and its corner points, and at the same time weaken the interference information such as background and noise, further increase the accuracy and robustness of key point coordinate estimation. More importantly, this scheme can also quickly realize the feature Figure 1 ,feature Figure 2 ,feature Figure 3 and Features Figure 4 Orderly upsampling and feature fusion are performed to ensure accurate extraction of vertebral semantic features.

[0035] In a possible implementation, each of the multi-scale attention modules includes a first channel splicing unit, a maximum pooling unit, an average pooling unit, a first fully connected layer unit, a second fully connected layer unit, a pixel summation unit, a first dot product unit, a second dot product unit, a first spatial attention enhancement unit, a second spatial attention enhancement unit, and a second channel splicing unit;

[0036] The first channel splicing unit communicates with the upsampling module and the output end of a network branch in the high-resolution network module respectively, the maximum pooling unit communicates with the first channel splicing unit, and the first fully connected layer unit communicates with the maximum pooling unit;

[0037] The average pooling unit communicates with the first channel concatenation unit, and the second fully connected layer unit communicates with the average pooling unit;

[0038] The pixel summing unit communicates with the first fully connected layer unit and the second fully connected layer unit respectively, the first dot product unit communicates with the pixel summing unit and the output end of a network branch in the high-resolution network module respectively, and the second dot product unit communicates with the pixel summing unit and the upsampling module respectively;

[0039] The first spatial attention enhancement unit communicates with the first dot product unit, the second spatial attention enhancement unit communicates with the second dot product unit, and the second channel splicing unit communicates with the first spatial attention enhancement unit and the second spatial attention enhancement unit respectively;

[0040] The convolution rectifier module communicates with the second channel splicing unit;

[0041] This solution can ensure that the feature fusion method combining channel splicing and pooling is completed in the operation of the decoder to fuse the features of each upsampled image with each feature map in order, and complete the feature fusion and feature selection of deep features and shallow features.

[0042] In one possible implementation, each of the multi-scale attention modules is configured to operate as follows:

[0043] The first channel splicing unit combines the first upsampled image with the feature Figure 3 Or the second upsampled image and the feature Figure 2 Or the third upsampled image and the feature Figure 1 Performing a channel stitching operation to obtain a first stitched image, and transmitting the first stitched image to the maximum pooling unit and the average pooling unit respectively;

[0044] The maximum pooling unit performs a maximum pooling operation on the first spliced ​​image to obtain a first pooled image, and transmits the first pooled image to the first fully connected layer unit;

[0045] The first fully connected layer unit performs a fully connected layer mapping process on the first pooled image to obtain a first connected image, and transmits the first connected image to the pixel summation unit;

[0046] The average pooling unit performs an average pooling operation on the first spliced ​​image to obtain a second pooled image, and transmits the second pooled image to the second fully connected layer unit;

[0047] The second fully connected layer unit performs a fully connected layer mapping process on the second pooled image to obtain a second connected image, and transmits the second connected image to the pixel summation unit;

[0048] The pixel summing unit performs a pixel summing operation on the first connected image and the second connected image, and then performs activation processing on the summing operation result by using an S-type activation function to obtain a first sum image, and transmits the first sum image to the first dot product unit and the second dot product unit respectively;

[0049] The first dot product unit combines the first sum image with the feature Figure 3 or with the characteristics Figure 2 or with the characteristics Figure 1 Performing a dot multiplication operation to obtain a first product image, and transmitting the first product image to the first spatial attention enhancement unit;

[0050] The first spatial attention enhancement unit performs feature enhancement processing on the first product image by combining channel average pooling and maximum pooling to obtain a first enhanced image, and transmits the first enhanced image to the second channel stitching unit;

[0051] The second dot multiplication unit performs a dot multiplication operation on the first sum image and the first up-sampled image or the second up-sampled image or the third up-sampled image to obtain a second product image, and transmits the second product image to the second spatial attention enhancement unit;

[0052] The second spatial attention enhancement unit performs feature enhancement processing on the second product image by combining channel average pooling and maximum pooling to obtain a second enhanced image, and transmits the second enhanced image to the second channel stitching unit;

[0053] The second channel stitching unit performs a channel stitching operation on the first enhanced image and the second enhanced image to obtain the first fused image or the second fused image or the third fused image, and transmits the image to the convolution rectification module;

[0054] This scheme uses the average pooling method to retain the global information of each feature map, ensuring the extraction of the overall lumbar sacral regional structure; at the same time, the maximum pooling method can obtain the important information of each feature map separately, which helps to focus on key details by selecting the pixels with the strongest response in the feature map; and the results of average pooling and maximum pooling are summed through the corresponding fully connected layer mapping, and the summation result can be obtained through the S-type activation function to obtain the first or second sum image showing the channel attention weight distribution; finally, it is combined into a multiplication operation, and the result is sent to the spatial attention enhancement unit to finally obtain the third fused image, thereby ensuring that the obtained image can accurately reflect the overall shape of the lumbar sacrum and the local correlation between vertebrae, significantly improving the quality and accuracy of feature extraction.

[0055] In a possible implementation, the first spatial attention enhancement unit is configured to perform the following steps:

[0056] A1: performing a channel average pooling operation on the first product image to obtain a first image, and performing a channel maximum pooling operation on the first product image to obtain a second image;

[0057] A2: performing a channel stitching operation on the first image and the second image to obtain a third image;

[0058] A3: performing a number of convolution operations on the third image to obtain a fourth image;

[0059] A4: performing a dot multiplication operation on the fourth image and the first product image to obtain the first enhanced image;

[0060] This scheme can capture the local details and global context information of the lumbar sacral region in the first product image by performing channel average pooling and channel maximum pooling operations on the first product image, significantly improving the performance of key point detection in the lumbar sacral region. At the same time, the subsequent channel splicing and convolution operations can highlight more relevant features such as the vertebral center area and the lumbar corner area, which helps to extract the spatial distribution information of the lumbar vertebrae and their corners, while weakening useless features such as background and noise, enhancing the fusion between deep features and front-layer features, and improving the robustness of the decoder and even the entire system.

[0061] In a possible implementation manner, the three independent decoupling mappings used by the mapping module are a first decoupling mapping, a second decoupling mapping, and a third decoupling mapping;

[0062] The mathematical formula of the first decoupling mapping is as follows:

[0063] ,

[0064] In the formula,

[0065] Represents the semantic features of the vertebral body;

[0066] represents the first intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 3×3 on the pyramidal semantic feature;

[0067] represents the second intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 1×1 on the first intermediate result;

[0068] represents the center point information, which is the result obtained after mapping the second intermediate result through the S-type activation function;

[0069] The mathematical formula of the second decoupling mapping is as follows:

[0070] ,

[0071] In the formula,

[0072] Represents the center point offset information;

[0073] The mathematical formula of the third decoupling mapping is as follows:

[0074] ,

[0075] In the formula,

[0076] represents the third intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 7×7 on the pyramidal semantic feature;

[0077] represents the corner point offset information, which is a result obtained by performing a convolution operation with a convolution kernel size of 7×7 on the third intermediate result;

[0078] This scheme independently obtains the center point information, center point offset information and corner point offset information through three decoupled mappings, thereby jointly determining the coordinates of the key points, ensuring that the obtained information contains the semantic information of the position and shape of each vertebra, which helps to determine the position of the key points more accurately.

[0079] Another technical solution of the present invention is to provide a lumbar sacral positioning and key point detection method based on deep learning, comprising the following steps:

[0080] S1: Perform convolution operation on the lumbar spine X-ray image through the lumbar spine detection device to extract multiple feature maps with different hierarchical features;

[0081] S2: fusing all feature maps obtained in step S1 into a plurality of resolution images showing feature information of objects of different scales by combining convolution operation with feature pyramid pooling through the lumbar vertebra detection device;

[0082] S3: positioning the lumbar sacral region of each of the resolution images by using the lumbar vertebra detection device to obtain the corresponding lumbar sacral region images;

[0083] S4: performing confidence screening and non-maximum suppression processing on all the lumbar sacral region images obtained in step S3 by a post-processing device, and then performing image fusion to obtain a fused image;

[0084] S5: extracting vertebral semantic features from the fused image through a key point detection device, and decoupling the vertebral semantic features through three independent decoupling mappings to extract vertebral center point information, center point offset information, and corner point offset information;

[0085] S6: Obtaining the coordinates of the corner points between the vertebrae according to the center point information, center point offset information and corner point offset information of the vertebrae through the key point detection device.

[0086] The method disclosed in the present application, through the preliminary positioning of the lumbar sacral region performed in steps S1, S2 and S3, can effectively utilize the prior knowledge of the lumbar spine, fully extract the features of the lumbar sacral region from the global image, eliminate the interference of irrelevant redundant information, and thus have robustness for spinal X-ray images of uncertain vertebrae. Combined with the process of step S4, it can provide accurate lumbar sacral region information for subsequent key point detection tasks. At the same time, considering the fact that high-precision key point detection in the lumbar sacral region must fully utilize the contextual information contained in the lumbar sacral region, the present invention extracts multi-level features of the lumbar X-ray image by executing step S5. These features are distributed throughout the feature content from low-level detail features to high-level semantic features, effectively promoting the ability to extract from detail information to global contextual information in image processing, and finally extracting the spatial distribution information of the lumbar key points (center point information, center point offset information and corner point offset information) to achieve accurate positioning of the key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a schematic diagram of the structure of a lumbar sacral positioning and key point detection system based on deep learning disclosed in an embodiment of the present application;

[0088] Figure 2 It is a schematic diagram of the operation flow of the lumbar vertebra detection device combined with the post-processing device disclosed in the embodiment of the present application;

[0089] Figure 3 A schematic diagram of the decoder structure disclosed in the embodiments of the present application;

[0090] Figure 4 This is a schematic diagram of the operation flow of the key point detection device disclosed in the embodiments of the present application;

[0091] Figure 5 Schematic diagram of the structure of the multi-scale attention module disclosed in the embodiments of the present application;

[0092] Figure 6 This is a schematic diagram of the operation flow of the multi-scale attention module disclosed in the embodiments of the present application;

[0093] Figure 7 It is a schematic diagram of the operation flow of the first spatial attention enhancement unit or the second spatial attention enhancement unit disclosed in the embodiments of the present application;

[0094] Figure 8 This is a flow chart of the method disclosed in the embodiments of this application.

[0095] Description of reference numerals:

[0096] 1. Lumbar vertebra detection device, 11. Backbone network module, 12. Neck network module, 13. Detection head module, 2. Post-processing device, 3. Key point detection device, 31. High-resolution network module, 32. Decoder, 321. Upsampling module, 322. Multi-scale attention module, 322-1. First channel splicing unit, 322-2. Maximum pooling unit, 322-3. Average pooling unit, 322-4. First fully connected layer unit, 322-5. Second fully connected layer unit, 322-6. Pixel summation unit, 322-7. First dot product unit, 322-8. Second dot product unit, 322-9. First spatial attention enhancement unit, 322-10. Second spatial attention enhancement unit, 322-11. Second channel splicing unit, 323. Convolutional rectifier module, 33. Mapping module, 34. Computation module. DETAILED DESCRIPTION

[0097] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0098] In the embodiments of the present application, unless otherwise clearly specified and limited, the communication or communication connection between the first feature and the second feature refers to the transmission of information between the first feature and the second feature. This information transmission can be either unidirectional or bidirectional, and the communication connection can be achieved by electrical connection of wires, radio connection, electrical connection of electromagnetic media (such as semiconductors), communication achieved by channels, etc.

[0099] In the embodiments of the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0100] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0101] See also Figure 1 to Figure 7 As shown, the embodiment of the present application discloses a lumbar sacral positioning and key point detection system based on deep learning. Figure 1It is a structural diagram of the system, which includes a lumbar vertebra detection device 1, a post-processing device 2 and a key point detection device 3, wherein the post-processing device 2 communicates with the lumbar vertebra detection device 1, and the key point detection device 3 communicates with the post-processing device 2.

[0102] In the system, the lumbar vertebra detection device 1 is configured to perform a convolution operation on the lumbar vertebra X-ray image to extract multiple feature maps with different levels of features, and fuse these feature maps into several resolution images showing the feature information of targets of different scales by combining the convolution operation with feature pyramid pooling, and then locate the lumbar sacrum region of each resolution image respectively to obtain the corresponding lumbar sacrum region image.

[0103] For details, see Figure 1 and Figure 2 As shown, the lumbar vertebra detection device 1 in this embodiment is the eighth version of the target detection network (YOLOv8), Figure 2 The schematic diagram of the operation flow of the lumbar vertebra detection device 1 combined with the post-processing device 2 is shown in FIG. The lumbar vertebra detection device 1 includes a backbone network module 11, a neck network module 12 and a detection head module 13, wherein the neck network module 12 communicates with the backbone network module 11, the detection head module 13 communicates with the neck network module 12, and the key point detection device 3 communicates with the detection head module 13. The backbone network module 11 is configured to perform a convolution operation on the lumbar vertebra X-ray image to extract a plurality of feature maps with different hierarchical features; the neck network module 12 is configured to fuse all feature maps obtained by the backbone network module 11 into three resolution images showing feature information of targets of different scales by combining a convolution operation with feature pyramid pooling (FPN). The detection head module 13 is configured to locate the lumbar sacral region of each resolution image respectively to obtain the corresponding lumbar sacral region images. Since the number of resolution images output by the neck network module 12 is three, the detection head module 13 needs to include three detection heads to correspond to the three resolution images. Finally, each of the three detection heads will output an image of the lumbar sacral region, and each image of the lumbar sacral region is marked with the positioning information of the lumbar sacral region.

[0104] It should be noted that the output end of the backbone network module 11 corresponds one to one with the input end of the neck network module 12, and the structures of the two can be determined according to the predetermined number of output ports of the backbone network module 11. Generally speaking, the number of output ports of the backbone network module 11 should not be less than the number of output ports of the neck network module 12, and the number of output ports of the neck network module 12 can be set according to specific needs, generally not less than three.

[0105] The lumbar vertebra detection device 1 with such a structure can detect and locate the lumbar sacral region more accurately, is robust to spinal X-ray images of vertebrae with an uncertain number of segments, and provides accurate lumbar sacral region information for subsequent key point detection tasks.

[0106] Please continue to see Figure 1 and Figure 2 As shown, in the system, the post-processing device 2 is configured to perform confidence screening and non-maximum suppression processing on the positioning of the lumbar sacral region on all the lumbar vertebrae sacral region images obtained by the lumbar vertebrae detection device 1, and then perform image fusion to obtain a fused image. Specifically, the post-processing device 2 is configured to operate according to the following steps: first, the confidence screening of the positioning of the lumbar sacral region is performed on all the lumbar vertebrae sacral region images obtained by the lumbar vertebrae detection device 1 to obtain images of the positioning of the lumbar sacral region with a confidence level not lower than a preset confidence level (a confidence level of 95% is used in this embodiment); then, these images are subjected to non-maximum suppression processing to delete repeated positioning; finally, the above processing results are subjected to image fusion operation to obtain a fused image.

[0107] See also Figure 1 and Figure 3 As shown, in the system, the key point detection device 3 is configured to extract vertebral semantic features based on the contextual information of the lumbar sacral region displayed in the fused image, and decouple the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebra, and obtain the corner point coordinates between the vertebrae.

[0108] Specifically, see Figure 1 , Figure 3 and Figure 4 As shown, the key point detection device 3 includes a high-resolution network module 31, a decoder 32, a mapping module 33 and a calculation module 34, wherein the high-resolution network module 31 communicates with the post-processing device 2, the decoder 32 communicates with the high-resolution network module 31, the mapping module 33 communicates with the decoder 32, and the calculation module 34 communicates with the mapping module 33.

[0109] See also Figure 1 and Figure 3 As shown, in the key point detection device 3, the high-resolution network module 31 is configured to perform feature encoding on the fused image to obtain the feature Figure 1 ,feature Figure 2 ,feature Figure 3 and Features Figure 4 , and, fuse images, features Figure 1 ,feature Figure 2 ,feature Figure 3 and Features Figure 4The image sizes are the first five items in the geometric progression whose common ratio is less than 1. The high-resolution network module 31 is a network structure with four network branches, which propagate data in four different directions and establish communication relationships with each other. The first network branch is used to obtain the feature Figure 1 , the second network branch is used to obtain the feature Figure 2 , the third network branch is used to obtain the feature Figure 3 , the fourth network branch is used to obtain the characteristics Figure 4 , the output end of each network branch communicates with the decoder 32. As an implementation method, the high-resolution network module 31 used in this embodiment is the eighteenth version of the high-resolution network (HRNet-18). The frequent information exchange between feature maps of different resolutions and the information fusion of each branch performed by the network can excel in maintaining high-resolution features and multi-scale feature fusion, can effectively capture global and local information, reflect the overall shape of the lumbar sacrum and the local association between vertebrae, and significantly improve the quality and accuracy of feature extraction.

[0110] See also Figure 3 to Figure 7 As shown, in the key point detection device 3, the decoder 32 is configured to detect the feature Figure 1 ,feature Figure 2 ,feature Figure 3 and Features Figure 4 Upsampling and feature fusion are performed to extract the semantic features of the vertebral body. Specifically, see Figure 3 As shown, in this embodiment, the decoder 32 is a communication network structure formed by three sub-decoders connected in series, and each sub-decoder includes an upsampling module 321, a multi-scale attention module 322 and a convolutional rectification module 323. In each sub-decoder, the multi-scale attention module 322 communicates with the upsampling module 321, and the convolutional rectification module 323 communicates with the multi-scale attention module 322.

[0111] See also Figure 3 and Figure 4 As shown, in the first sub-decoder, the upsampling module 321 communicates with the output end of the fourth network branch in the high-resolution network module 31, and the multi-scale attention module 322 also communicates with the output end of the third network branch in the high-resolution network module 31. The upsampling module 321 is set to Figure 4 The upsampling operation is performed to obtain a first upsampled image, and the multi-scale attention module 322 is configured to combine the first upsampled image with the feature by a feature fusion method combining channel splicing and pooling. Figure 3 Feature fusion is performed to obtain a first fused image, and the convolution rectification module 323 is configured to sequentially perform convolution processing and linear rectification function mapping processing on the first fused image to obtain a first semantic feature image.

[0112] See also Figure 3 and Figure 4 As shown, in the sub-decoder located in the middle, the upsampling module 321 communicates with the convolutional rectifier module 323 in the sub-decoder located in the first position, and the multi-scale attention module 322 also communicates with the output end of the second network branch in the high-resolution network module 31. The upsampling module 321 is configured to perform an upsampling operation on the first semantic feature image to obtain a second upsampled image, and the multi-scale attention module 322 is configured to combine the second upsampled image with the feature image in a feature fusion method combining channel splicing and pooling. Figure 2 Feature fusion is performed to obtain a second fused image, and the convolution rectification module 323 is configured to sequentially perform convolution processing and linear rectification function mapping processing on the second fused image to obtain a second semantic feature image.

[0113] See also Figure 3 and Figure 4 As shown, in the sub-decoder located at the end, the upsampling module 321 communicates with the convolution rectifier module 323 in the sub-decoder located in the middle, the multi-scale attention module 322 also communicates with the output end of the first network branch in the high-resolution network module 31, the mapping module 33 communicates with the convolution rectifier module 323, the upsampling module 321 is configured to upsample the second semantic feature image to obtain a third upsampled image, and the multi-scale attention module 322 is configured to combine the third upsampled image with the feature image in a feature fusion method combining channel splicing and pooling. Figure 1 Feature fusion is performed to obtain a third fused image, and the convolution rectification module 323 is configured to sequentially perform convolution processing and linear rectification function mapping processing on the third fused image to obtain vertebral semantic features.

[0114] See also Figure 5 and Figure 6As shown, in this embodiment, each multi-scale attention module 322 includes a first channel splicing unit 322-1, a maximum pooling unit 322-2, an average pooling unit 322-3, a first fully connected layer unit 322-4, a second fully connected layer unit 322-5, a pixel summation unit 322-6, a first dot product unit 322-7, a second dot product unit 322-8, a first spatial attention enhancement unit 322-9, a second spatial attention enhancement unit 322-10 and a second channel splicing unit 322-11. The first channel splicing unit 322-1 communicates with the output of a network branch in the upsampling module 321 and the high-resolution network module 31, respectively. Specifically, for the multi-scale attention module 322 in the first sub-decoder, the first channel splicing unit 322-1 communicates with the output of the third network branch in the high-resolution network module 31; for the multi-scale attention module 322 in the middle sub-decoder, the first channel splicing unit 322-1 communicates with the output of the second network branch in the high-resolution network module 31; for the multi-scale attention module 322 in the last sub-decoder, the first channel splicing unit 322-1 communicates with the output of the first network branch in the high-resolution network module 31. The maximum pooling unit 322-2 communicates with the first channel splicing unit 322-1, and the first fully connected layer unit 322-4 communicates with the maximum pooling unit 322-2.

[0115] See also Figure 5 and Figure 6 As shown, in each multi-scale attention module 322, the average pooling unit 322-3 communicates with the first channel splicing unit 322-1, and the second fully connected layer unit 322-5 communicates with the average pooling unit 322-3; the pixel summing unit 322-6 communicates with the first fully connected layer unit 322-4 and the second fully connected layer unit 322-5 respectively, and the first dot product unit 322-7 communicates with the pixel summing unit 322-6 and the output end of a network branch in the high-resolution network module 31 respectively. Specifically, for the first pixel For the multi-scale attention module 322 in the sub-decoder, the first dot product unit 322-7 communicates with the output end of the third network branch in the high-resolution network module 31; for the multi-scale attention module 322 in the middle sub-decoder, the first dot product unit 322-7 communicates with the output end of the second network branch in the high-resolution network module 31; for the multi-scale attention module 322 in the sub-decoder at the end, the first dot product unit 322-7 communicates with the output end of the first network branch in the high-resolution network module 31. The second dot product unit 322-8 communicates with the pixel summing unit 322-6 and the upsampling module 321 respectively.

[0116] See also Figure 5 and Figure 6As shown, in each multi-scale attention module 322: the first spatial attention enhancement unit 322-9 communicates with the first dot product unit 322-7, the second spatial attention enhancement unit 322-10 communicates with the second dot product unit 322-8, and the second channel splicing unit 322-11 communicates with the first spatial attention enhancement unit 322-9 and the second spatial attention enhancement unit 322-10 respectively; the convolution rectifier module 323 communicates with the second channel splicing unit 322-11.

[0117] See also Figure 6 As shown, each multi-scale attention module 322 is configured to operate as follows:

[0118] First, the first channel concatenation unit 322-1 combines the first upsampled image with the feature Figure 3 Perform channel concatenation to obtain the first concatenated image, or concatenate the second upsampled image with the feature Figure 2 Perform channel concatenation to obtain the first concatenated image, or concatenate the third upsampled image with the feature Figure 1 A channel splicing operation is performed to obtain a first spliced ​​image, and the first spliced ​​image is transmitted to the maximum pooling unit 322-2 and the average pooling unit 322-3 respectively; specifically, for the multi-scale attention module 322 in the first sub-decoder, the first channel splicing unit 322-1 combines the first up-sampled image with the feature Figure 3 A channel splicing operation is performed to obtain a first spliced ​​image; for the multi-scale attention module 322 in the middle sub-decoder, the first channel splicing unit 322-1 combines the second up-sampled image with the feature Figure 2 A channel splicing operation is performed to obtain a first spliced ​​image; for the multi-scale attention module 322 in the sub-decoder at the end, the first channel splicing unit 322-1 combines the third up-sampled image with the feature Figure 1 A channel stitching operation is performed to obtain a first stitched image.

[0119] Subsequently, the maximum pooling unit 322-2 performs a maximum pooling operation on the first stitched image to obtain a first pooled image, and transmits the first pooled image to the first fully connected layer unit 322-4; the first fully connected layer unit 322-4 performs a mapping process of the fully connected layer on the first pooled image to obtain a first connected image, and transmits the first connected image to the pixel summing unit 322-6. At the same time, the average pooling unit 322-3 performs an average pooling operation on the first stitched image to obtain a second pooled image, and transmits the second pooled image to the second fully connected layer unit 322-5; the second fully connected layer unit 322-5 performs a mapping process of the fully connected layer on the second pooled image to obtain a second connected image, and transmits the second connected image to the pixel summing unit 322-6.

[0120] Afterwards, the pixel summing unit 322-6 performs a pixel summing operation on the first connected image and the second connected image, and then activates the summing operation result with an S-type activation function to obtain a first sum image, and transmits the first sum image to the first dot multiplication unit 322-7 and the second dot multiplication unit 322-8 respectively.

[0121] Next, the first dot multiplication unit 322-7 combines the first sum image with the feature Figure 1 or Features Figure 2 or Features Figure 3 A dot multiplication operation is performed to obtain a first product image, and the first product image is transmitted to the first spatial attention enhancement unit 322-9; specifically, for the multi-scale attention module 322 in the first sub-decoder, the first dot multiplication unit 322-7 combines the first sum image with the feature Figure 3 A dot multiplication operation is performed to obtain a first product image; for the multi-scale attention module 322 in the middle sub-decoder, the first dot multiplication unit 322-7 combines the first sum image with the feature Figure 2 A dot multiplication operation is performed to obtain a first product image; for the multi-scale attention module 322 in the sub-decoder at the end, the first dot multiplication unit 322-7 combines the first sum image with the feature Figure 1 A dot multiplication operation is performed to obtain a first product image; the first spatial attention enhancement unit 322-9 performs feature enhancement processing on the first product image by combining channel average pooling and maximum pooling to obtain a first enhanced image, and transmits the first enhanced image to the second channel stitching unit 322-11. At the same time, the second dot multiplication unit 322-8 performs a dot multiplication operation on the first sum image and the first up-sampled image or the second up-sampled image or the third up-sampled image to obtain a second product image, and transmits the second product image to the second spatial attention enhancement unit 322-10; specifically, for the multi-scale attention module 322 in the first sub-decoder, the second dot multiplication unit 322-8 performs a dot multiplication operation on the first sum image and the first up-sampled image or the second up-sampled image or the third up-sampled image to obtain a second product image, and transmits the second product image to the second spatial attention enhancement unit 322-10. A dot multiplication operation is performed on the first upsampled image to obtain a second product image; for the multi-scale attention module 322 in the middle sub-decoder, the second dot multiplication unit 322-8 performs a dot multiplication operation on the first sum image and the second upsampled image to obtain a second product image; for the multi-scale attention module 322 in the sub-decoder at the end, the second dot multiplication unit 322-8 performs a dot multiplication operation on the first sum image and the third upsampled image to obtain a second product image; the second spatial attention enhancement unit 322-10 performs feature enhancement processing on the second product image by combining channel average pooling and maximum pooling to obtain a second enhanced image, and transmits the second enhanced image to the second channel stitching unit 322-11.

[0122] Finally, the second channel stitching unit 322-11 performs a channel stitching operation on the first enhanced image and the second enhanced image to obtain a first fused image or a second fused image or a third fused image. Specifically, for the multi-scale attention module 322 in the sub-decoder located at the first position, the second channel stitching unit 322-11 obtains the first fused image; for the multi-scale attention module 322 in the sub-decoder located in the middle, the second channel stitching unit 322-11 obtains the second fused image; for the multi-scale attention module 322 in the sub-decoder located at the end, the second channel stitching unit 322-11 obtains the third fused image, and then transmits the first fused image, the second fused image or the third fused image to the convolution rectification module 323.

[0123] Figure 7 The first spatial attention enhancement unit 322-9 or the second spatial attention enhancement unit 322-10 is a schematic diagram of the operation flow. Specifically, the first spatial attention enhancement unit 322-9 is configured to perform the following steps;

[0124] A1: performing a channel average pooling operation on the first product image to obtain a first image, and performing a channel maximum pooling operation on the first product image to obtain a second image;

[0125] A2: performing channel concatenation operation on the first image and the second image to obtain a third image;

[0126] A3: performing several convolution operations on the third image to obtain a fourth image;

[0127] A4: Perform a dot multiplication operation on the fourth image and the first product image to obtain a first enhanced image.

[0128] The second spatial attention enhancement unit 322-10 is configured to perform the following steps;

[0129] B1: performing a channel average pooling operation on the second product image to obtain the first image, and performing a channel maximum pooling operation on the first product image to obtain the second image;

[0130] B2: performing channel concatenation operation on the first image and the second image to obtain a third image;

[0131] B3: performing several convolution operations on the third image to obtain a fourth image;

[0132] B4: Perform a dot multiplication operation on the fourth image and the second product image to obtain a second enhanced image.

[0133] See also Figure 1As shown, in the key point detection device 3, the mapping module 33 is configured to decouple the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebral body. Specifically, the three independent decoupling mappings used by the mapping module 33 are the first decoupling mapping, the second decoupling mapping and the third decoupling mapping, wherein the mathematical formula of the first decoupling mapping is as follows:

[0134] ,

[0135] In the formula,

[0136] Represents the semantic features of the vertebral body;

[0137] represents the first intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 3×3 on the pyramidal semantic features;

[0138] represents the second intermediate result, which is the result obtained by performing a convolution operation with a convolution kernel size of 1×1 on the first intermediate result;

[0139] Represents the center point information, which is the result obtained by mapping the second intermediate result through the S-type activation function;

[0140] The mathematical formula of the second decoupling mapping is as follows:

[0141] ,

[0142] In the formula,

[0143] Represents the center point offset information;

[0144] The mathematical formula of the third decoupling mapping is as follows:

[0145] ,

[0146] In the formula,

[0147] represents the third intermediate result, which is the result obtained after performing a convolution operation on the pyramidal semantic features with a convolution kernel size of 7×7;

[0148] Represents the corner point offset information, which is the result obtained after performing a convolution operation with a convolution kernel size of 7×7 on the third intermediate result.

[0149] See also Figure 1As shown, in the key point detection device 3, the calculation module 34 is configured to obtain the corner point coordinates between vertebrae based on the center point information, the center point offset information and the corner point offset information. Specifically, the calculation module 34 is configured to perform the following steps: first, each vertebra is positioned by the center point information of the vertebrae, and the quantization error of each vertebrae center point is eliminated by the center point offset information to obtain the center point of each vertebrae. Then, the corner point coordinates between vertebrae are calculated by the corner point offset information and the center point obtained in the above step.

[0150] See also Figure 8 As shown, an application method corresponding to the deep learning-based lumbar sacral positioning and key point detection system in this embodiment will be further disclosed below. The method includes the following steps:

[0151] S1: performing a convolution operation on the lumbar spine X-ray image by the lumbar spine detection device 1 to extract a plurality of feature maps with different hierarchical features;

[0152] S2: All feature maps obtained in step S1 are fused into several resolution images showing feature information of objects of different scales by the lumbar vertebra detection device 1 in a manner combining convolution operation with feature pyramid pooling;

[0153] S3: positioning the lumbar sacral region of each resolution image by the lumbar vertebra detection device 1 to obtain the corresponding lumbar sacral region image;

[0154] S4: performing confidence screening and non-maximum suppression processing on all the lumbar sacral region images obtained in step S3 by the post-processing device 2, and then performing image fusion to obtain a fused image;

[0155] S5: extracting vertebral semantic features from the fused image through the key point detection device 3, and decoupling the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebral body;

[0156] S6: Obtain the corner point coordinates between vertebrae according to the center point information, center point offset information and corner point offset information of the vertebral body through the key point detection device 3.

[0157] In order to verify the superiority of the deep learning-based lumbar sacral positioning and key point detection system in this embodiment, this embodiment is compared with the classic key point detection system and other advanced vertebral key point detection systems on public data sets and private data sets, and the comparative experimental results are shown in Tables 1 and 2. Among them, the bold content represents the optimal result, and the underlined content represents the suboptimal result.

[0158] In the table, hourglass network and high-resolution network are classic networks in the field of key point detection. Hourglass-2 represents a system consisting of two stacked hourglass modules, and hourglass-8 represents a system consisting of eight stacked hourglass modules. Yi system and Ao system are systems built based on cutting-edge methods in the field of spine key point detection. The input of all comparative experiments is the lumbar sacral region image after being processed by the first stage target detection network (corresponding to the processing of lumbar detection device 1).

[0159] Table 1 Comparative experimental results on public datasets

[0160]

[0161] Table 2 Comparative experimental results on private datasets

[0162]

[0163] The results show that the deep learning-based lumbar sacral positioning and key point detection system in this embodiment achieves the minimum value in both the mean absolute error and the mean radial error on both the public and private datasets. For the successful detection rate, on the private dataset, the deep learning-based lumbar sacral positioning and key point detection system in this embodiment achieves the highest value in all error ranges, especially when the error range is 4mm, the successful detection rate has reached 95%. On the public dataset, the deep learning-based lumbar sacral positioning and key point detection system in this embodiment achieves the highest value in most error ranges, although it is slightly lower than the Ao system when the error range is 6 pixels (px), but it significantly surpasses other systems in other error ranges. In particular, when the error range is 9 pixels and 10 pixels, only the deep learning-based lumbar sacral positioning and key point detection system in this embodiment achieves a successful detection rate of more than 90%. These experimental results show that the deep learning-based lumbar sacral positioning and key point detection system in this embodiment is highly robust and more comprehensive, and can achieve high key point detection accuracy on different datasets.

[0164] In summary, the deep learning-based lumbar sacral positioning and key point detection system disclosed in this embodiment obtains prediction results by setting a lumbar detection device 1 to adopt a convolution operation combined with feature pyramid pooling. This method is robust for lumbar X-ray images of uncertain vertebrae, can eliminate the interference of irrelevant redundant information, and can effectively utilize the prior knowledge of the lumbar spine to obtain the characteristics of the lumbar sacral region from the global image, providing accurate lumbar sacral region information for subsequent key point detection tasks. By setting a post-processing device 2 to perform confidence screening on all the obtained lumbar sacral region images, the lumbar sacral region positioning with a low confidence level can be removed, and by setting the non-maximum suppression processing of the post-processing device 2, the repeated positioning of the lumbar sacral region can be deleted, so that the fused image obtained after the image fusion can more accurately display the lumbar sacral region information; finally, by further setting a key point detection device 3, the ability to extract from detail information to global context information in the image is effectively improved. Since the key point detection device 3 extracts the vertebral semantic features based on the contextual information of the lumbar sacral area displayed in the fused image, and decouples the vertebral semantic features through three independent decoupling mappings to extract the center point information, center point offset information and corner point offset information of the vertebra, it can not only simultaneously obtain low-level features with rich details and high-level features with rich semantics, but also capture the spatial distribution information of the lumbar key points, thereby improving the robustness of the system operation and achieving accurate positioning of the key points.

[0165] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0166] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A lumbar sacral positioning and key point detection system based on deep learning, characterized in that: include: The lumbar vertebra detection device (1) is configured to perform a convolution operation on a lumbar vertebra X-ray image to extract a plurality of feature maps having different hierarchical features, and fuse these feature maps into a plurality of resolution images showing feature information of targets of different scales by combining the convolution operation with feature pyramid pooling, and then locate the lumbar sacral region on each of the resolution images to obtain the corresponding lumbar sacral region images; A post-processing device (2) is configured to perform confidence screening and non-maximum suppression processing on the positioning of the lumbar sacral region on all lumbar sacral region images obtained by the lumbar vertebra detection device (1), and then perform image fusion to obtain a fused image; A key point detection device (3) is configured to extract vertebral semantic features based on context information of the lumbar sacral region displayed in the fused image, and decouple the vertebral semantic features through three independent decoupling mappings to extract vertebral center point information, center point offset information and corner point offset information, and obtain corner point coordinates between vertebrae; The post-processing device (2) communicates with the lumbar vertebra detection device (1), and the key point detection device (3) communicates with the post-processing device (2); The key point detection device (3) comprises: The high-resolution network module (31) is configured to perform feature encoding on the fused image to obtain a feature map 1, a feature map 2, a feature map 3, and a feature map 4, and the image sizes of the fused image, the feature map 1, the feature map 2, the feature map 3, and the feature map 4 are, in order, the first five items in a geometric progression whose common ratio is less than 1; A decoder (32) is configured to perform upsampling and feature fusion on the feature map 1, the feature map 2, the feature map 3 and the feature map 4 to extract the vertebral semantic features; A mapping module (33) is configured to decouple the vertebral semantic features through three independent decoupling mappings to extract the vertebral center point information, center point offset information and corner point offset information; A calculation module (34) configured to obtain the coordinates of corner points between vertebrae based on the center point information, the center point offset information and the corner point offset information; The high-resolution network module (31) communicates with the post-processing device (2), the decoder (32) communicates with the high-resolution network module (31), the mapping module (33) communicates with the decoder (32), and the calculation module (34) communicates with the mapping module (33).

2. The deep learning-based lumbar sacral positioning and key point detection system according to claim 1, characterized in that: The lumbar vertebra detection device (1) comprises: A backbone network module (11) is configured to perform a convolution operation on the lumbar spine X-ray image to extract a plurality of feature maps having different hierarchical features; The neck network module (12) is configured to fuse all feature maps obtained by the backbone network module (11) into a plurality of resolution images showing feature information of objects of different scales by combining convolution operation with feature pyramid pooling; The detection head module (13) is configured to locate the lumbar vertebrae sacrum region of each of the resolution images respectively, and obtain the corresponding lumbar vertebrae sacrum region images; The neck network module (12) communicates with the backbone network module (11), the detection head module (13) communicates with the neck network module (12), and the key point detection device (3) communicates with the detection head module (13).

3. The deep learning-based lumbar sacral positioning and key point detection system according to claim 2, characterized in that: The high-resolution network module (31) is a network structure having four network branches, wherein the first network branch is used to obtain the feature map 1, the second network branch is used to obtain the feature map 2, the third network branch is used to obtain the feature map 3, and the fourth network branch is used to obtain the feature map 4, and the output end of each network branch communicates with the decoder (32).

4. The deep learning-based lumbar sacral positioning and key point detection system according to claim 3, characterized in that: The decoder (32) is a communication network structure formed by three sub-decoders connected in series, each of which includes an upsampling module (321), a multi-scale attention module (322) and a convolutional rectification module (323); In each of the sub-decoders, the multi-scale attention module (322) communicates with the up-sampling module (321), and the convolution rectification module (323) communicates with the multi-scale attention module (322); In the sub-decoder located at the first position, the upsampling module (321) communicates with the output end of the fourth network branch in the high-resolution network module (31), and the multi-scale attention module (322) also communicates with the output end of the third network branch in the high-resolution network module (31), the upsampling module (321) is configured to perform an upsampling operation on the feature map four to obtain a first upsampled image, the multi-scale attention module (322) is configured to perform feature fusion on the first upsampled image and the feature map three in a feature fusion manner combining channel splicing and pooling to obtain a first fused image, and the convolution rectification module (323) is configured to perform convolution processing and linear rectification function mapping processing on the first fused image in sequence to obtain a first semantic feature image; In the sub-decoder located in the middle, the upsampling module (321) communicates with the convolutional rectification module (323) in the sub-decoder located at the first position, and the multi-scale attention module (322) also communicates with the output end of the second network branch in the high-resolution network module (31). The upsampling module (321) is configured to perform an upsampling operation on the first semantic feature image to obtain a second up-sampled image. The multi-scale attention module (322) is configured to perform feature fusion on the second up-sampled image with the feature image 2 in a feature fusion manner combining channel splicing and pooling to obtain a second fused image. The convolutional rectification module (323) is configured to perform convolution processing and linear rectification function mapping processing on the second fused image in sequence to obtain a second semantic feature image. In the sub-decoder located at the end, the upsampling module (321) communicates with the convolutional rectification module (323) in the sub-decoder located in the middle, the multi-scale attention module (322) also communicates with the output end of the first network branch in the high-resolution network module (31), the mapping module (33) communicates with the convolutional rectification module (323), the upsampling module (321) is configured to perform an upsampling operation on the second semantic feature image to obtain a third up-sampled image, the multi-scale attention module (322) is configured to perform feature fusion on the third up-sampled image with the feature map 1 in a feature fusion method combining channel splicing and pooling to obtain a third fused image, and the convolutional rectification module (323) is configured to perform convolution processing and linear rectification function mapping processing on the third fused image in sequence to obtain the pyramidal semantic feature.

5. The deep learning-based lumbar sacral positioning and key point detection system according to claim 4, characterized in that: Each of the multi-scale attention modules (322) includes a first channel splicing unit (322-1), a maximum pooling unit (322-2), an average pooling unit (322-3), a first fully connected layer unit (322-4), a second fully connected layer unit (322-5), a pixel summing unit (322-6), a first dot product unit (322-7), a second dot product unit (322-8), a first spatial attention enhancement unit (322-9), a second spatial attention enhancement unit (322-10) and a second channel splicing unit (322-11); The first channel splicing unit (322-1) communicates with the upsampling module (321) and the output end of a network branch in the high-resolution network module (31), respectively; the maximum pooling unit (322-2) communicates with the first channel splicing unit (322-1); and the first fully connected layer unit (322-4) communicates with the maximum pooling unit (322-2); The average pooling unit (322-3) communicates with the first channel concatenation unit (322-1), and the second fully connected layer unit (322-5) communicates with the average pooling unit (322-3); The pixel summing unit (322-6) communicates with the first fully connected layer unit (322-4) and the second fully connected layer unit (322-5) respectively, the first dot product unit (322-7) communicates with the pixel summing unit (322-6) and an output end of a network branch in the high-resolution network module (31) respectively, and the second dot product unit (322-8) communicates with the pixel summing unit (322-6) and the upsampling module (321) respectively; The first spatial attention enhancement unit (322-9) communicates with the first dot product unit (322-7), the second spatial attention enhancement unit (322-10) communicates with the second dot product unit (322-8), and the second channel splicing unit (322-11) communicates with the first spatial attention enhancement unit (322-9) and the second spatial attention enhancement unit (322-10) respectively; The convolution rectification module (323) communicates with the second channel splicing unit (322-11).

6. The deep learning-based lumbar sacral positioning and key point detection system according to claim 5, characterized in that: Each of the multi-scale attention modules (322) is configured to operate as follows: The first channel stitching unit (322-1) performs a channel stitching operation on the first up-sampled image and the feature map three, or on the second up-sampled image and the feature map two, or on the third up-sampled image and the feature map one to obtain a first stitching image, and transmits the first stitching image to the maximum pooling unit (322-2) and the average pooling unit (322-3) respectively; The maximum pooling unit (322-2) performs a maximum pooling operation on the first spliced ​​image to obtain a first pooled image, and transmits the first pooled image to the first fully connected layer unit (322-4); The first fully connected layer unit (322-4) performs a fully connected layer mapping process on the first pooled image to obtain a first connected image, and transmits the first connected image to the pixel summing unit (322-6); The average pooling unit (322-3) performs an average pooling operation on the first spliced ​​image to obtain a second pooled image, and transmits the second pooled image to the second fully connected layer unit (322-5); The second fully connected layer unit (322-5) performs a fully connected layer mapping process on the second pooled image to obtain a second connected image, and transmits the second connected image to the pixel summing unit (322-6); The pixel summing unit (322-6) performs a pixel summing operation on the first connected image and the second connected image, and then performs activation processing on the summing operation result using an S-type activation function to obtain a first sum image, and transmits the first sum image to the first dot product unit (322-7) and the second dot product unit (322-8) respectively; The first dot multiplication unit (322-7) performs a dot multiplication operation on the first sum image and the feature map 3 or the feature map 2 or the feature map 1 to obtain a first product image, and transmits the first product image to the first spatial attention enhancement unit (322-9); The first spatial attention enhancement unit (322-9) performs feature enhancement processing on the first product image by combining channel average pooling and maximum pooling to obtain a first enhanced image, and transmits the first enhanced image to the second channel stitching unit (322-11); The second dot multiplication unit (322-8) performs a dot multiplication operation on the first sum image and the first up-sampled image or the second up-sampled image or the third up-sampled image to obtain a second product image, and transmits the second product image to the second spatial attention enhancement unit (322-10); The second spatial attention enhancement unit (322-10) performs feature enhancement processing on the second product image by combining channel average pooling and maximum pooling to obtain a second enhanced image, and transmits the second enhanced image to the second channel stitching unit (322-11); The second channel stitching unit (322-11) performs a channel stitching operation on the first enhanced image and the second enhanced image to obtain the first fused image or the second fused image or the third fused image, and transmits the image to the convolution rectification module (323).

7. The deep learning-based lumbar sacral positioning and key point detection system according to claim 6, characterized in that: The first spatial attention enhancement unit (322-9) is configured to perform the following steps; A1: performing a channel average pooling operation on the first product image to obtain a first image, and performing a channel maximum pooling operation on the first product image to obtain a second image; A2: performing a channel stitching operation on the first image and the second image to obtain a third image; A3: performing a number of convolution operations on the third image to obtain a fourth image; A4: Perform a dot multiplication operation on the fourth image and the first product image to obtain the first enhanced image.

8. The deep learning-based lumbar sacral positioning and key point detection system according to any one of claims 2 to 7, characterized in that: The three independent decoupling maps used by the mapping module (33) are a first decoupling map, a second decoupling map and a third decoupling map; The mathematical formula of the first decoupling mapping is as follows: ， In the formula, Represents the semantic features of the vertebral body; represents the first intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 3×3 on the pyramidal semantic feature; represents the second intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 1×1 on the first intermediate result; represents the center point information, which is the result obtained after mapping the second intermediate result through the S-type activation function; The mathematical formula of the second decoupling mapping is as follows: ， In the formula, Represents the center point offset information; The mathematical formula of the third decoupling mapping is as follows: ， In the formula, represents the third intermediate result, which is the result obtained after performing a convolution operation with a convolution kernel size of 7×7 on the pyramidal semantic feature; Represents the corner point offset information, which is the result obtained after performing a convolution operation with a convolution kernel size of 7×7 on the third intermediate result.

9. A method for lumbar sacral positioning and key point detection based on deep learning, characterized in that: The deep learning-based lumbar sacral positioning and key point detection system applicable to any one of claims 1 to 8 comprises the following steps: S1: performing a convolution operation on the lumbar spine X-ray image through the lumbar spine detection device (1) to extract multiple feature maps with different levels of features; S2: using the lumbar vertebra detection device (1) to combine convolution operation with feature pyramid pooling to fuse all feature maps obtained in step S1 into a plurality of resolution images showing feature information of objects of different scales; S3: using the lumbar vertebra detection device (1) to locate the lumbar sacral region of each of the resolution images, respectively, to obtain the corresponding lumbar sacral region images; S4: performing confidence screening and non-maximum suppression processing on all the lumbar sacral region images obtained in step S3 by a post-processing device (2), and then performing image fusion to obtain a fused image; S5: extracting vertebral semantic features from the fused image through a key point detection device (3), and decoupling the vertebral semantic features through three independent decoupling mappings to extract vertebral center point information, center point offset information, and corner point offset information; S6: Obtaining the coordinates of the corner points between the vertebrae according to the center point information, center point offset information and corner point offset information of the vertebrae through the key point detection device (3).

Images