A method for constructing a key point positioning model of a spine and application thereof

Abstract

The application discloses a kind of construction methods and applications of spinal key point positioning model, belong to image processing technical field;The application constructs a kind of model from coarse to fine according to the order from root node to vertebral center point to corner point, the whole model uses the average key point information collected in advance to ensure the topological structure of spinal key point positioning result, while the key point positioning process is guided by point, first align a root node can let the center of whole spine align, so that the approximate center point area of the spinal scoliosis image of various conditions can be aligned first, then further corner positioning is carried out, so as to reduce the positioning error generated by the average center point position deviation from the position area of spinal vertebra in some scoliosis conditions;At the same time, each positioning module includes a positioning unit, and the interpolation of the feature map is differentiable, so that end-to-end training of the whole model becomes possible, and the spinal key points can be accurately positioned.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more particularly relates to a method for constructing a spine key point positioning model and application thereof. BACKGROUND

[0002] Scoliosis is a common spinal disease characterized by lateral curvature of the spine with vertebral rotation. Patients with severe scoliosis have abnormal body appearance, reduced thoracic and abdominal cavity space, and a high risk of heart, lung, and digestive tract compression and dysfunction. Before scoliosis surgery, a reasonable diagnosis and treatment plan is usually formulated based on the curvature of the spine in the X-ray image of the patient's spine. Compared with computed tomography (CT), X-ray is widely used in scoliosis diagnosis due to its economy, speed, and relatively small radiation dose.

[0003] The front and back X-ray images of the human spine contain 12 thoracic vertebrae and 5 lumbar vertebrae. The spine landmarks are usually defined as four corner points of each of the 17 spines, totaling 17x4=68 key points. Automatic positioning of these spine key points is not only of great clinical significance and value for formulating treatment plans for spine correction, selecting surgical methods, and postoperative effect evaluation, but also a prerequisite for downstream applications such as Cobb angle calculation and biomechanical load analysis. However, the superposition of heterogeneous soft tissues caused by X-ray projection makes the image of the spine part blurred and low in contrast, and the uniform appearance of the vertebrae often leads to confusion with similar bones or structures, which brings great challenges to the automatic and accurate positioning of the vertebral label.

[0004] In recent years, with the rise and development of deep learning technology, many research methods have attempted to overcome this challenging task. One of the spine key point positioning models includes a center point positioning network and an angle point positioning network trained in steps. After the coordinates of the center points of each spine in the spine image are positioned by the trained center point positioning network, the spine image is cropped to obtain a local spine sub-image centered on the positioned coordinates of each spine center point, which is then input into the trained angle point positioning network for angle point positioning. Although this model can better maintain the shape constraint of the spine and the position of the key points, it has fewer missed detections and false positives, but its multi-cascade structure brings additional computational overhead, and the non-end-to-end training strategy causes the prediction results of the latter model to be heavily dependent on the accuracy of the former model, which cannot accurately position the spine key points. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a method for constructing a spine key point positioning model and application thereof to solve the technical problem that the prior art cannot accurately position the spine key points.

[0006] To achieve the above object, in a first aspect, the application provides a method for constructing a spine key point positioning model, comprising:

[0007] constructing the spine key point positioning model, and inputting a pre-acquired training set of spine images into the spine key point positioning model for training;

[0008] The spine key point positioning model comprises a global feature extraction module, a root node positioning module , a center point positioning module and a corner point positioning module ; 、 、 Each of the modules includes m 1、 m 2、 m 3 positioning units; m 1、 m 2、 m 3 is greater than or equal to 1; the root node is the center point of any vertebral body in the spine; ; H is the total number of vertebrae in the spine;

[0009] The global feature extraction module is used to extract the features of the input spine image , to obtain a global feature map;

[0010] The input of the first positioning unit in is the global feature map and an initial root node coordinate set ; is the average value of the center point coordinates of the vertebral body in each spine image in the pre-acquired spine image set; ; is the root node coordinate positioning result of ;

[0011] The input of the first positioning unit in is the global feature map and an initial center point coordinate set ; h The coordinate in h is the initial coordinate of the center point of the vertebral body in ; ; is the center point coordinate positioning result of the h vertebral body in ; For each spinal image in the pre-acquired spinal image set, the first... h Average coordinates of the center point of the vertebral body; ;

[0012] The input to the first localization unit is the global feature map and the initial set of corner coordinates. ; The first in k The coordinates are The Middle k Initial coordinates of the corner points ; For the first k The vertebral body number where each corner point is located; For each spinal image in the pre-acquired spinal image set, the first... k The average coordinates of the corner points; Output ; for The Middle k The coordinates of the corner points; ; K This represents the total number of angle points in the spine;

[0013] for , or When the number of its internal positioning units is multiple, its internal first... i The input coordinate set of the positioning unit is the innermost unit. i -1 coordinate set output by the positioning unit; i ≥2;

[0014] The localization unit includes an offset learning network; the localization unit interpolates the input global feature map to obtain a corresponding continuous feature map; for each coordinate point in the input coordinate set, it interpolates the continuous feature map with... The size ratio is determined by obtaining a local window corresponding to the coordinate point in the continuous feature map, calculating the sum of the average values ​​of all pixels within the local window as the local feature value corresponding to the coordinate point; the local feature values ​​corresponding to each coordinate point are input into the offset learning network to obtain the coordinate offset of each coordinate point in the input coordinate set; each coordinate point in the input coordinate set is summed with its corresponding coordinate offset to obtain the updated coordinate set.

[0015] More preferably, for each coordinate point in the input coordinate set Based on this continuous feature map and The size ratio, obtained from the coordinates of the continuous feature map. The corresponding coordinates of the top-left and bottom-right corners of the local window are respectively and ; wherein, ; ; ; ; and are preset width and height, respectively; is a scaling factor, specifically a scaling ratio of the continuous feature map relative to .

[0016] Further preferably, the global feature extraction module is configured to extract features of different scales from the input spine image , to obtain M global feature maps with scales sequentially increasing from small to large; M ; wherein, the global feature map at the M-th scale is taken as the input of the M-th positioning unit, m . m .

[0017] Further preferably, the global feature extraction module comprises a cascaded encoder and decoder;

[0018] The encoder comprises M cascaded encoding units; the decoder comprises M cascaded decoding units; the input of the first decoding unit is the output of the last encoding unit; the input of the M-th decoding unit is the result of fusion of the output of the M-1-th decoding unit and the output of the M+1-th encoding unit; M M r r r ;

[0019] The encoder is configured to encode the input spine image step by step to obtain M encoding feature maps with scales sequentially decreasing from large to small;

[0020] The decoder is configured to decode the encoding feature maps output by the encoder step by step to obtain M decoding feature maps with scales sequentially increasing from small to large;

[0021] The encoding feature map output by the last encoding unit and the decoding feature map output by the M-th decoding unit constitute a set of M global feature maps with scales sequentially increasing from small to large. Further preferably, the above training of the spine key point positioning model by inputting the pre-collected spine image training set comprises:

[0022] Each spine image in the spine image training set is input into the spine key point positioning model, and the root node coordinate positioning result of each spine image is minimized by the spine key point positioning model,

[0023] ​​​​​​​​​​ the coordinate positioning result of each corner point in the middle of each vertebral body, the difference loss between the coordinate positioning result of each corner point in the middle and the corresponding real coordinate, and training the spine key point positioning model.

[0024] In a second aspect, the present application provides a spine key point positioning method, comprising: inputting a to-be-positioned spine image into a spine key point positioning model to obtain the coordinates of each corner point of the spine in the to-be-positioned spine image.

[0025] The spine key point positioning model is constructed by using the construction method of the spine key point positioning model provided in the first aspect of the present application.

[0026] In a third aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method provided in the first aspect or the second aspect of the present application.

[0027] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect or the second aspect of the present application.

[0028] In a fifth aspect, the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, executes the method provided in the first aspect or the second aspect of the present application.

[0029] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0030] 1. The present application provides a construction method of a spine key point positioning model, which constructs a model for key point positioning from coarse to fine in the order of root node, vertebral body center point and corner point. The entire model uses the average key point information collected in advance to ensure the topological structure of the spine key point positioning result, and the key point positioning process uses points as guides, aligns the root node (an arbitrary center point) first, which can align the center of the entire spine, so that the average center point position can be near the range of the spine, thereby reducing the positioning error caused by the deviation of the average center point position from the position range of the vertebrae in some scoliosis cases. At the same time, each positioning module is implemented by a positioning unit, and the differentiability of the interpolation of the feature map by the positioning unit makes end-to-end training of the entire model possible, avoiding the problem that the prediction result of the later stage in a multi-stage model depends heavily on the accuracy of the former stage, and enabling accurate positioning of the spine key points.

[0031] 2、Further, the method for constructing the spine key point positioning model provided by the application extracts different scale features of the input spine image by the global feature extraction module in the constructed spine key point positioning model, obtains global feature maps with scales sequentially increasing from small to large, and ensures that the global feature map at the first scale is input to the first positioning unit, M m m The application uses the abstract global feature map with a smaller scale as the input of the root node positioning module and uses the fine global feature map with a larger scale as the input of the corner point positioning module, realizes the key point detection process from coarse to fine, highly couples the abstract to specific global multi-scale features in the semantic and fine degree, efficiently utilizes the features of different scales, highly couples the abstract to specific global multi-scale features in the semantic and fine degree from coarse to fine, realizes more accurate local feature extraction, and further improves the positioning accuracy.

[0032] 3、Further, the method for constructing the spine key point positioning model provided by the application inputs each spine image in the spine image training set into the spine key point positioning model, trains the spine key point positioning model by minimizing the difference loss of the root node coordinate positioning result, the center point coordinate positioning result of each vertebra in the middle, and the coordinate positioning result of each corner point, and the corresponding real coordinate, and optimizes the accuracy of the whole stage spine key point prediction in an end-to-end manner through the comprehensive positioning loss function of the root node, the vertebra center point and the corner point, and further avoids the problem that the prediction result of the later stage in the multi-cascade model seriously depends on the accuracy of the former stage model. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The positioning process schematic diagram of the spine key point positioning model provided by the embodiment of the application is shown.

[0034] Figure 2 The spine initial model schematic diagram provided by the embodiment of the application is shown.

[0035] Figure 3 The structure schematic diagram of the spine key point positioning model provided by the embodiment of the application is shown.

[0036] Figure 4 The point feature guided local feature extraction process schematic diagram provided by the embodiment of the application is shown.

[0037] Figure 5 ​​​​​​A comparison chart of the change trend of the detection rate with the increase of the error radius for the spine key point positioning method based on the spine key point positioning model provided by the application and other existing spine key point positioning methods;

[0038] Figure 6 A comparison chart of the positioning visualization results on the public data set for the spine key point positioning method based on the spine key point positioning model provided by the application and other existing spine key point positioning methods. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0040] In order to achieve the above-mentioned purpose, in a first aspect, the present application provides a construction method of a spine key point positioning model, comprising:

[0041] The spine key point positioning model is built, and the pre-acquired spine image training set is input into the spine key point positioning model for training.

[0042] The spine key point positioning model comprises:

[0043] a global feature extraction module for extracting the features of the input spine image to obtain a global feature map;

[0044] a root node positioning module for taking the root node coordinates in the spine initial model as the root node initial coordinates of , and obtaining the root node local features of from the global feature map, and then updating the root node coordinates of to obtain the root node coordinate positioning result of ;

[0045] a center point positioning module for obtaining the coordinate offset of each vertebral center point relative to the root node from the spine initial model, and respectively adding the root node coordinate positioning result of , and taking the obtained result as the initial coordinates of each vertebral center point in , and obtaining the local features of each vertebral center point in from the global feature map, and then updating the coordinates of each vertebral center point in to obtain the coordinates of each vertebral center point in ​a coordinate positioning result of each center point of the vertebrae in the spine in the coordinate system of the global feature map;

[0046] corner point positioning module , for obtaining coordinate offsets of each corner point of the spine relative to the center point of the vertebra where the corner point is located in the initial model of the spine, and respectively adding the coordinate offsets to the coordinate positioning results of the corresponding center points of the vertebrae in the coordinate system of the global feature map in the coordinate system of the global feature map, and taking the obtained results as initial coordinates of each corner point in the coordinate system of the global feature map;

[0047] wherein, the root node is a center point of any th vertebra of the spine; the root node coordinate in the initial model of the spine is an average value of the center point coordinates of the th vertebra in each spine image in the pre-acquired spine image set; the coordinate offset of the center point of the th vertebra of the spine relative to the root node in the initial model of the spine is h ; the coordinate offset of the th corner point of the spine relative to the center point of the k th vertebra where the corner point is located in the initial model of the spine is ; is an average value of the coordinates of the th corner point in each spine image in the pre-acquired spine image set; k , H is a total number of vertebrae in the spine; , K is a total number of corner points in the spine.

[0048] Specifically, the root node positioning module , the center point positioning module , and the corner point positioning module each include m 1, m 2, m 3 positioning units; m 1, m 2, m 3 are greater than or equal to 1;

[0049] The input of the first positioning unit in the root node positioning module is the global feature map and the initial root node coordinate set ; the output of the root node positioning module ; is ​​​​​​the root node coordinate positioning result of the first positioning unit in the root node positioning module

[0050] the center point positioning module The input of the first positioning unit in the center point positioning module is the global feature map and the initial center point coordinate set The first coordinate in the center point positioning module is the initial coordinate of the center point of the first lumbar vertebra in the first lumbar vertebra positioning module h The output of the center point positioning module h The output of the center point positioning module The output of the center point positioning module The output of the center point positioning module h The output of the center point positioning module h The output of the center point positioning module

[0051] The first positioning unit in the corner point positioning module The input of the first positioning unit in the corner point positioning module is the global feature map and the initial corner point coordinate set The first coordinate in the corner point positioning module is the initial coordinate of the first corner point in the first corner point positioning module k The output of the corner point positioning module k The output of the corner point positioning module The output of the corner point positioning module The output of the corner point positioning module k The output of the corner point positioning module

[0052] For the root node positioning module , the center point positioning module or the corner point positioning module , when the number of positioning units in it is multiple, the input coordinate set of the first positioning unit in it is the coordinate set output by the first-1 positioning unit in it i i ≥2 i

[0053] The positioning unit comprises an offset learning network; the positioning unit is used for interpolating the input global feature map to obtain a corresponding continuous feature map; for each coordinate point in the input coordinate set, the continuous feature map and the coordinate point are used as input of the offset learning network to obtain an offset value; and the offset value is added to the coordinate point to obtain a new coordinate point ​​​​​​​​​​​​​​​The size ratio of the coordinate point, a local window corresponding to the coordinate point is obtained in a continuous feature map, the sum average value of each pixel in the local window is calculated as the local feature value corresponding to the coordinate point, and the local feature values corresponding to each coordinate point are collectively input into the offset learning network to obtain the coordinate offset of each coordinate point in the input coordinate set.

[0054] The spine key point positioning model constructed by the application includes a root node positioning module, a center point positioning module and a corner point positioning module; the application uses the pre-counted key point coordinate average value information of a large number of spine images as prior information, firstly uses the initial root node coordinate in the prior information as the starting point through the root node positioning module, updates the root node coordinate based on the local feature of the root node, and positions the root node in the input spine image; then, the center point positioning module is used to route the obtained root node coordinate positioning result to each vertebral center point of the input spine image based on the coordinate offset information of the vertebral center point relative to the root node in the prior information, and update the coordinates of each vertebral center point based on the local feature of each vertebral center point, so as to position each vertebral center point in the input spine image; finally, the corner point positioning module is used to route the obtained coordinate positioning result of each vertebral center point to each corner point of the input spine image based on the coordinate offset of the corner point relative to the vertebral center point where the corner point is located in the prior information, and update the coordinates of each corner point based on the local feature of each corner point, so as to position each corner point in the input spine image; the application constructs a model for key point positioning from coarse to fine in the order of root node, vertebral center point and corner point, and the entire model uses the average key point information collected in advance to ensure the topological structure of the spine key point positioning result, and the key point positioning process uses a point as a guide, so that the center of the entire spine can be aligned by aligning a center point first, so that the average center point position can be near the range of the spine, thereby enabling the approximate center point region to be aligned first and then the corner point positioning to be further performed on the spine scoliosis image in various situations, and thus the positioning error caused by the displacement of the average center point position from the vertebra position region in some scoliosis situations is reduced; at the same time, considering that the gradient will be broken in the process of extracting a local feature map from a global feature map when the feature map is discrete, end-to-end training cannot be performed, therefore, each positioning module in the application is implemented through a positioning unit, the interpolation of the feature map is made differentiable through the positioning unit, so that end-to-end training of the entire model becomes possible, the problem that the prediction result of the last stage in a multi-stage model is seriously dependent on the accuracy of the previous stage model is avoided, and the spine key points can be accurately positioned.

[0055] It should be noted that the input global feature map can be interpolated by using a bilinear interpolation, a bicubic interpolation, a Lanczos interpolation, a spline interpolation or the like.

[0056] In an optional implementation, for each coordinate point in the input coordinate set , based on the size ratio of the continuous feature map and , the left top corner coordinate and the right bottom corner coordinate of the local window corresponding to the coordinate point in the continuous feature map are respectively and ; wherein ; ; ; ; and are respectively a preset width and a preset height; is a scaling factor, specifically a scaling ratio of the continuous feature map relative to .

[0057] It should be noted that the global feature map can be one scale global feature map, or can include multiple scale global feature maps. In an optional implementation, the global feature extraction module is configured to extract features of different scales of the input spine image , to obtain M global feature maps with scales sequentially increasing from small to large; ; wherein the global feature map at the m th scale is input to the m th positioning unit, .

[0058] It should be noted that the global feature extraction module can use multiple backbone networks such as VGG, ResNet, EfficientNet, etc., and can be combined with a skip connection layer, an attention mechanism, multi-scale convolution, etc. to further improve the performance of the model in extracting features.

[0059] In an optional implementation, the global feature extraction module includes a cascaded encoder and a decoder.

[0060] The encoder includes M cascaded encoding units; the decoder includes M cascaded decoding units; the input of the first decoding unit is the output of the last encoding unit; the input of the r th decoding unit is the result of fusion of the output of the r -1th decoding unit and the output of the r +1th encoding unit; ;

[0061] The encoder is configured to encode the input spine image step by step to obtain encoding feature maps with scales sequentially decreasing from large to small;

[0062] The decoder is used to decode the encoded feature map output by the encoder step by step to obtain decoded feature maps with scales increasing sequentially from small to large.

[0063] The encoded feature map output by the last-level coding unit, and the first The decoded feature maps output by the level decoding unit constitute a set of global feature maps with scales increasing sequentially from small to large.

[0064] In one optional implementation, the pre-acquired spinal image training set is input into the spinal keypoint localization model for training, including:

[0065] Each spine image in the spine image training set The data is input into the spinal key point localization model and minimized. Root node coordinate positioning results The coordinates of the center point of each vertebral body, and The difference loss between the coordinate positioning results of each corner point and the corresponding real coordinates is used to train the spinal key point positioning model.

[0066] It should be noted that the above training method is an optimal one, and it can be minimized. The coordinates of the center point of each vertebral body, and The difference between the coordinate positioning result of each corner point and its corresponding true coordinates, or simply minimizing the loss. The difference between the coordinate localization result of each corner point and its corresponding true coordinate is used to train the spinal key point localization model. No specific limitations are imposed here.

[0067] It should also be noted that the loss function can be L1 loss function, L2 loss function, SmoothL1 Loss, Huber Loss, etc., and there is no limitation here.

[0068] To further illustrate the method for constructing the spinal key point localization model provided by the present invention, a specific embodiment is described in detail below:

[0069] like Figure 1 The diagram shows the localization process of the spinal key point localization model. The entire process includes:

[0070] 1) Establishment of the initial spinal model:

[0071] A set of spinal images is pre-collected, and the coordinate annotation information of key points (vertebral body center point and corner point) is used to establish an initial root node model, an initial center point model, and an initial corner point model, providing prior guidance for subsequent localization algorithms;

[0072] It should be noted that the spinal image set here can be a pre-acquired spinal image training set. In this embodiment, the spinal image training set is directly used as the spinal image set.

[0073] Each training set image contains the coordinates of corner points for 12 thoracic vertebrae and 5 lumbar vertebrae, totaling 68 corner points. These coordinates can be represented as: ,in Indicates the first k The absolute coordinates of each corner point.

[0074] In the middle, each set of four adjacent points represents the upper left, upper right, lower left, and lower right corners of the same vertebral body. Therefore, according to the first... h The coordinates of the center point of a vertebra can be obtained from the four corner points of that vertebra:

[0075]

[0076] in, h =1,…,17 are the indices of the center points. The coordinates of these 17 vertebral center points can be represented as: .

[0077] Based on the positions of the corner points and center points in each spine image in the spine image set, the average corner point coordinates and average center point coordinates of the entire spine image set can be calculated, where the average corner point coordinates are denoted as: , No. k The average coordinates of the corner points are calculated as follows:

[0078]

[0079] in, n Image index for the spine image set. N This represents the total number of spine images in the spine image set. Represents the first image in the spinal image set. n The first image k The absolute coordinates of each corner point. The average center point coordinates are calculated in the same way, denoted as... .

[0080] After obtaining the average corner coordinates and average center point coordinates Then, the initial root node model, initial center point model, and initial corner point model can be established according to the following rules to jointly form the initial spine model.

[0081] Initial root node model This embodiment first roughly aligns the model with the entire spine by aligning a point, called the root node. Technically, any point can be chosen from the average center points as the initial root node model. For ease of understanding, this embodiment defines the 9th average center point as the initial root node model, denoted as . .

[0082] Initial center point model The average center point coordinates are the initial center point model, denoted as . .

[0083] Initial corner model The average corner coordinates are the initial corner model, denoted as: .

[0084] Visualization results of the initial spinal model as follows Figure 2 As shown, since the initial model of the spine is based on an average model obtained from a large number of spine images, it is approximately straight.

[0085] 2) Multi-scale global feature extraction:

[0086] This embodiment utilizes an encoder-decoder network to extract multi-scale global features from a single-channel X-ray spinal image. Figure 3 The architecture of the proposed spinal keypoint localization model (denoted as RouterNet network) is shown in detail. Given a single-channel X-ray spinal image with a size of 704×384, the encoder-decoder network is first used to extract multi-scale global features.

[0087] Encoder: such as Figure 3 As shown in the encoder section, this embodiment modifies EfficientNet-B3 as the backbone network to extract multi-scale encoded features. Specifically, the encoder of this method contains five encoding blocks, each consisting of one or more convolutional layers called "MBConv". These convolutional layers are inverted bottleneck layers that incorporate squeeze-and-excitation optimization, allowing the model to better capture dependencies between channels and improve accuracy. Each encoding unit (i.e., encoding block) outputs an encoded feature map, with the dimension gradually halved. Through the encoder, five encoded features are obtained, which are downsampled by a ratio of 2^35 to the size of the input spine image. 2 4 2 8 2 16 2 and 32 2 Here they are represented as .

[0088] Decoder: such as Figure 3As shown in the decoder part in FIG. 1, the embodiment can obtain decoding feature maps of different sizes, denoted as . Except that and are the same, the remaining decoding feature maps are fused by jump connection of low-scale decoding feature maps and corresponding encoding feature maps, and gradually recover semantic information. For example, first, up-sampling (up-sampling 2 2 times) is performed on , so that the image features after up-sampling are the same in size as the corresponding encoding feature map , and then the corresponding encoding feature map is spliced by channel, and finally the spliced feature map is generated through 1x1 convolution to generate the next decoding feature map . The embodiment adds a group normalization layer (Group Normalization, GN) in each scale of the convolution layer to accelerate the convergence speed of the network.

[0089] 3) Point-guided local feature extraction:

[0090] After obtaining the initial model of the spine and a series of global feature maps of different sizes, a point feature guided local feature extraction module is used to extract a series of local features from the global features in a differentiable operation form for subsequent updating of the point coordinates.

[0091] After obtaining the initial model ( , , ) and a series of global feature maps of different sizes , in order to extract more representative features, the embodiment designs a point feature guided local feature extraction module PLFE to extract local features around each point in a differentiable operation form for subsequent coordinate updating.

[0092] As shown in Figure 4 , given global feature , point set and local window , local feature can be extracted through PLFE, and the operation process can be represented as:

[0093]

[0094] Specifically, the embodiment uses a continuously differentiable PrPool layer to clip the global feature, which guarantees the gradient backpropagation while refining the window center coordinates.

[0095] Let be a given global feature map at a discrete position i, j If the vector at a discrete feature point is considered as a continuous coordinate at x , y , then the discrete feature at any continuous coordinate

[0096]

[0097] where IC is the interpolation coefficient, which can be represented as .

[0098] For the coordinate of the m th point in the point set , the PrPool layer is used to extract local features with a window size of on the global feature map , which can be represented as:

[0099]

[0100] where is the range of the scaled bounding box, , ; and are the scaled window sizes, , , is the scaling factor, which is the downsampling rate of the current feature relative to the input spine image size. In this embodiment, takes the value of 96, takes the value of 64, takes the value of 2 2 , 4 2 , 8 2 , 16 2 , and 32 2 .

[0101] Iterating the above operation for all points in the point set , the local features around the point set can be obtained.

[0102] Iterating the above operation for all points in the point set , the local features around the point set can be obtained.

[0103] 4) Routing keypoint localization:

[0104] ​​​With the extracted series of features, firstly, the initial root node model is updated to the root node to realize the alignment of the whole spine; then, the root node is routed (diffused) to the center point of the vertebra according to the initial center point model and the position is updated to realize the alignment of the center of each vertebra; finally, the center point is routed to the corner point of the vertebra according to the initial corner point model and the position is updated to realize the accurate positioning of the corner point of each vertebra.

[0105] RouterNet uses the extracted global feature map in a proportionally increasing order for the three stages of positioning, firstly, the initial root node model is initialized and updated to the root node to realize the alignment of the whole spine; then, the root node is routed (diffused) to the center point of the vertebra according to the initial center point model and is constantly updated to realize the alignment of the center of each vertebra; finally, the center point is routed to the initial corner point according to the initial corner point model and is constantly updated to realize the alignment of the corner point of each vertebra.

[0106] In the root node positioning stage, firstly, the initial root node model is taken as the initialization starting point of this stage , then on the lowest scale feature map , with as the center, the local features with a window size of are extracted by PLFE, denoted as ; the local features are then updated to a new position by a learnable updater (offset learning network), and the process can be represented as follows, wherein represents the parameters to be optimized in the updater.

[0107]

[0108] In this process, the updater will use a fully connected layer to map the local features to a vector, denoted as , which represents the offset from the input point set to the target point set , and the coordinates of the input point set are then updated as follows:

[0109]

[0110] The goal of the root node positioning stage is the center of the whole spine, i.e. the labeled position of the 9th center point of the image, denoted as . Given this, the embodiment adopts an L1 loss function to calculate the root node positioning loss between the labeled root node and the updated root node , denoted as:

[0111]

[0112] In the center point localization stage, first, a parameter-free router is used, taking the output of the root node localization stage as the origin , according to the initial center point model , to route (diffuse) 1 root node to 17 points to get the starting point of the center point localization stage . For convenience of representation, based on the initial center point model , the offset from the average center point coordinates to the initial root node is obtained, denoted as . The routing process of the center point localization stage can be represented as follows, where represents copying 17 times to make it the same shape as .

[0113]

[0114] Similar to the root node localization stage, the initial center point is gradually updated twice using the two-layer features , and the corresponding updater, and the updated center point coordinates are finally obtained, which is represented as:

[0115] ,

[0116]

[0117] The goal of the center point routing stage is the center point of the 17 vertebrae, that is, the labeled position of the 17 center points . Similar to , the L1 loss function is used in this stage to calculate the center point localization loss between the labeled position of the 17 vertebrae center and the updated vertebrae center point , which can be represented as:

[0118]

[0119] In the corner point localization stage, first, a router is also used, taking the output of the center point localization stage as the origin, according to the initial corner point model , to route (diffuse) 17 center points to 68 points to get the starting point of the corner point localization stage . For convenience of representation, based on the initial corner point model , the offset from the average corner point coordinates of each vertebra to the average center point of the vertebra is obtained, denoted as . Therefore, the routing process of the corner point localization stage can be represented as:

[0120]

[0121] Similar to the previous stage, the initial corner points are gradually updated twice using the two-layer feature and the corresponding updater, and the updated corner point coordinates are finally obtained. The goal of the corner positioning stage is to locate 68 vertebral corner points. Similar to the previous stage, the L1 loss function is used to calculate the corner positioning loss between the labeled corner points and the updated vertebral corner point coordinates

[0122] , which can be expressed as:

[0123]

[0124] In this embodiment, the overall loss function of the spinal key point positioning model can be expressed as follows, where are the hyperparameters of each loss term, which are empirically set to 1, 5, and 40, respectively.

[0125]

[0126] The spinal key point positioning model is trained by minimizing .

[0127] To further demonstrate the performance of the spinal key point positioning model provided by the present application, the present application tests on the disclosed anteroposterior spinal X-ray image dataset AASCE and compares the proposed spinal key point positioning model RouterNet with eight advanced methods.

[0128] Table 1 shows the comparison results of the present application method and other advanced algorithms on the dataset. Compared with the heat map regression-based method, the point coordinate regression-based method achieves better performance in NMSE and OR indicators, and reduces about an order of magnitude in the NMSE indicator for measuring spinal key point positioning, mainly because the regression method can easily preserve the topological structure between the labeled points.

[0129] Table 1

[0130]

[0131] ​​​In these point coordinate regression-based methods, compared with the current most advanced algorithm Cascaded CNNs, the method of the application reduces the NMSE by 73.8% again, and almost no missed detection and false positive prediction. Such excellent performance not only benefits from the advantages of point coordinate regression-based methods, but also benefits from the routing divide-and-conquer positioning strategy proposed by the application, that is, the complex task is divided into three sub-tasks for processing. For CNN, this way of decomposition can greatly simplify the positioning difficulty of each layer network, so as to obtain more accurate key point positioning results.

[0132] The application further analyzes the trend of the detection rate (1-outlier ratio) with the increase of the error radius, and the results are shown in Figure 5 As the error radius increases, more and more points are detected. Due to the existence of inherent shape constraints, three coordinate regression-based methods can successfully detect 80% of the key points at a small error radius of 0.02 (normalized error radius), while some heat map regression-based methods reach this index at a radius of 0.05. In contrast, the method of the application detects almost all key points at an error radius of 0.05. This excellent performance shows that the method proposed by the application can well preserve the shape of the spine marker points, and almost no missed detection and false positive prediction occurs.

[0133] Figure 6 The prediction results of various methods on the dataset are shown. According to the figure, it can be observed that in the area where the features are obvious, almost all methods can accurately regress the position of the corner point; but in the area where there is tissue occlusion and imaging blur in the image, the prediction result of the heat map regression-based method is poor, and there are many missed detections and false positive predictions (such as the arrow part in the figure), and the contrast method based on coordinate regression has less missed detection in the prediction result due to the existence of inherent shape constraints, but the positioning accuracy is poor. Compared with other contrast algorithms, the application has better processing results in areas where image features are not obvious, and the overall prediction result can not only maintain a good topological structure, but also has high positioning accuracy, which is closer to the position of the manually annotated key points.

[0134] In summary, the application provides a routing spine key point positioning model based on point feature guidance, which decouples the complex key point positioning task to the multi-scale features of the convolutional neural network through the idea of divide and conquer, and realizes the accurate positioning of the spine key points in the spine image without cascading the model. Compared with the performance of other most advanced technologies, the spine key point positioning model provided by the application improves the NMSE index by 73.8% in the spine key point positioning task.

[0135] In a second aspect, the present application provides a method for locating key points of a spine, comprising: inputting a spine image to be located into a key point locating model of a spine to obtain coordinates of each corner point of the spine in the spine image to be located.

[0136] The key point locating model of the spine is constructed by using the method for constructing a key point locating model of a spine provided in the first aspect of the present application.

[0137] The related technical solution is the same as the method for constructing a key point locating model of a spine provided in the first aspect of the present application, and thus is not described herein.

[0138] In a third aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory storing a computer program, and the processor executing the computer program to execute the method provided in the first aspect or the second aspect of the present application.

[0139] The related technical solution is the same as the method for constructing a key point locating model of a spine provided in the first aspect of the present application and the method for locating key points of a spine provided in the second aspect of the present application, and thus is not described herein.

[0140] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect or the second aspect of the present application.

[0141] The related technical solution is the same as the method for constructing a key point locating model of a spine provided in the first aspect of the present application and the method for locating key points of a spine provided in the second aspect of the present application, and thus is not described herein.

[0142] In a fifth aspect, the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, executes the method provided in the first aspect or the second aspect of the present application.

[0143] The related technical solution is the same as the method for constructing a key point locating model of a spine provided in the first aspect of the present application and the method for locating key points of a spine provided in the second aspect of the present application, and thus is not described herein.

[0144] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a key point positioning model of a spine, characterized in that, The method comprises the following steps: a spine key point positioning model is built, and a pre-acquired spine image training set is input into the spine key point positioning model for training; The spine key point positioning model comprises a global feature extraction module, a root node positioning module, a center point positioning module, and an angle point positioning module; the modules respectively comprise 1, 2, and 3 positioning units; 1, 2, and 3 are greater than or equal to 1; the root node is the center point of any first vertebra in the spine; the total number of vertebrae in the spine is n; and the center point of each vertebra is determined by the center point positioning module. m m m m m m H ​​​​​​​​​​​​​​​ The global feature extraction module is used to extract the input spine image. Based on the characteristics, a global feature map is obtained; The input of the first positioning unit in the above formula is the global feature map and the initial root node coordinate set ; ; The output of the first positioning unit in the above formula is the average value of the center point coordinates of the first lumbar vertebra of each spine image in the pre-acquired spine image set ; ; ; The output of the first positioning unit in the above formula is the root node coordinate positioning result of the above formula ; The input of the first positioning unit in the first positioning unit is a global feature map and an initial center point coordinate set ; ; The first h coordinate in the first sectional vertebra center point initial coordinate h ; The output of the first ; ; ; The output of the first positioning unit in the first positioning unit is a global feature map and an initial center point coordinate set ; h ; The output of the first positioning unit in the first positioning unit is a global feature map and an initial center point coordinate set h ; ; The The input to the first localization unit is the global feature map and the initial set of corner coordinates. ; The first in k The coordinates are The Middle k Initial coordinates of the corner points ; For the first k The vertebral body number where each corner point is located; For each spinal image in the pre-acquired spinal image set, the first... k The average coordinates of the corner points; Output ; for The Middle k The coordinates of the corner points; ; K The total number of angle points in the spine; for the aforementioned , or When the number of its internal positioning units is multiple, its internal first... i The input coordinate set of the positioning unit is the innermost unit. i -1 coordinate set output by the positioning unit; i ≥2; The positioning unit comprises an offset learning network; the positioning unit is configured to interpolate the input global feature map to obtain a corresponding continuous feature map; for each coordinate point in the input coordinate set, a local window corresponding to the coordinate point is obtained in the continuous feature map based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map and the local window is calculated based on the size ratio of the continuous feature map 2. The method of claim 1, wherein, For each coordinate point in the input coordinate set , based on the size ratio of the continuous feature map and , the left upper corner coordinate and the right lower corner coordinate of the local window corresponding to the coordinate point in the continuous feature map are respectively and ; wherein ; ; ; ; and are respectively a preset width and a preset height; is a scaling factor, specifically a scaling ratio of the continuous feature map relative to .

3. The method of claim 1, wherein, The global feature extraction module is configured to extract the input spine image Different scales of features are obtained in order of increasing scale from small to large M global feature maps; ; wherein the global feature map at the first scale is input to the first positioning unit, m m .​​ 4. The method of claim 3, wherein, the global feature extraction module comprises a cascaded encoder and decoder; The encoder comprises M M cascaded encoding units; the decoder comprises M M cascaded decoding units; the input of the first decoding unit is the output of the last encoding unit; the input of the r Mth decoding unit is the output of the r M-1th decoding unit and the output of the M+1th encoding unit after fusion; r ;​ The encoder is used for encoding the input spine image in stages to obtain encoded feature maps with scales decreasing in turn from large to small. the decoder is configured to decode the encoded feature map output by the encoder step by step to obtain decoded feature maps with scales sequentially increasing from small to large; The encoded feature map output by the last-level coding unit, and the first The decoded feature maps output by the level decoding unit constitute a set of global feature maps with scales increasing sequentially from small to large.

5. The method of claim 1-4, wherein, the step of inputting the pre-acquired spine image training set into the spine key point positioning model for training comprises the following steps: each of the spinal column images in the training set of spinal column images input into the spinal column key point positioning model by minimizing the root node coordinate positioning result, the center point coordinate positioning result of each vertebra in the spinal column key point positioning model, the difference loss between the coordinate positioning result of each corner point in the spinal column key point positioning model and the corresponding real coordinate.

6. A method of locating key points of a spine, the method comprising: The method comprises the following steps: inputting the spine image to be positioned into the spine key point positioning model to obtain the coordinates of each corner point of the spine in the spine image to be positioned; wherein the spine key point positioning model is built by using the construction method of the spine key point positioning model in any one of claims 1-5.

7. An electronic device, comprising: The method comprises the following steps: a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method in any one of claims 1-6.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-6. The computer program is executed by the processor to implement the steps of the method in any one of claims 1-6.

Images