Feature point analysis method and apparatus in x-ray image

Abstract

A method and apparatus for analyzing feature points in X-ray images are disclosed. The feature point analysis method may include: a step of training a center extraction model such that predicted center points of the lumbar vertebrae and sacral vertebrae included in the X-ray images are extracted in response to X-ray images from which an arbitrary lumbar region has been removed; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted predicted center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model such that predicted feature points of the lumbar vertebrae and sacral vertebrae included in the crop images are detected in response to the crop images being input to the feature detection model.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for analyzing feature points included in an X-ray image using deep learning. Background Technology

[0002] Spinal diseases can occur without specific symptoms, and since it is difficult for patients to fully recover once they have suffered from the condition, a rapid and accurate diagnosis is essential. X-ray, CT, or MRI images may be used to diagnose a patient's spinal disease.

[0003] Among these, CT and MRI images can diagnose spinal diseases more accurately, but they have the disadvantage of consuming a significant amount of time and cost. Therefore, most users diagnose spinal diseases by utilizing X-ray imaging, which is relatively simple and inexpensive.

[0004] However, when using X-ray images to diagnose a patient's spinal disease, there was a problem in that the user could not easily visually identify characteristic points of the spine because shadows of complex three-dimensional structures, such as the thorax, overlapped with the spinal region. Prior Art: Registered Patent Publication No. 10-2357001 (2022.01.25.) The problem to be solved

[0005] The present invention aims to provide a method and apparatus for detecting feature points included in an X-ray region by analyzing the characteristics of an X-ray image and the morphological and positional characteristics of the spine through deep learning.

[0006] In addition, the present invention aims to provide a method and apparatus capable of diagnosing a patient's spinal disease in real time through a feature point analysis network of a relatively small size.

[0007] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist. means of solving the problem

[0008] A feature point analysis method according to one embodiment of the present invention may include: a step of training a center extraction model so that expected center points of lumbar vertebrae and sacral vertebrae included in the X-ray images are extracted in response to X-ray images being input to the center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted expected center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model so that expected feature points of lumbar vertebrae and sacral vertebrae included in the crop images are detected in response to the crop images being input to the feature detection model.

[0009] The above center extraction model can be trained so that X-ray images used for training are augmented through a Random Spine Cutout (RSC) technique that sets the pixel values ​​of arbitrary square areas inside the lumbar vertebrae corresponding to L1 to L4 to zero, and confidence maps for the predicted center points of the lumbar vertebrae and sacral vertebrae included in the augmented X-ray images are extracted.

[0010] The above center extraction model can use the Mean Squared Error (MSE) between the confidence map extracted for the above-mentioned center point and the target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as a loss function.

[0011] The target confidence map generated for the above actual center point can be generated based on the actual values ​​(Ground Truth, GT) for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image.

[0012] Cropped images including each of the above lumbar vertebrae can be generated based on the difference in distance between the expected center points of the lumbar vertebrae included in the X-ray image.

[0013] A cropped image including the sacrum can be generated based on the difference in distance between the expected center point of the 4th lumbar vertebra and the expected center point of the 5th lumbar vertebra among the lumbar vertebrae included in the X-ray image.

[0014] The above feature detection model can be trained to extract a confidence map for the predicted vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images.

[0015] The above feature detection model can use the mean squared error between the confidence map extracted for the above predicted vertices and the target confidence map generated for the actual vertices of the lumbar and sacral vertebrae included in the cropped images as a loss function.

[0016] The target confidence map generated for the above actual vertices can be generated based on the actual values ​​for the actual vertices of the lumbar vertebrae and sacral vertebrae included in the above cropped images.

[0017] The above feature detection model can be trained to predict Part Affinity Fields (PAFs), which are information indicating the orientation and location of the connection (limb) between the two regions for the lumbar vertebrae and sacral vertebrae included in the cropped image.

[0018] A feature point analysis method according to one embodiment of the present invention may include: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them.

[0019] The above center extraction model can be trained by using the Mean Squared Error (MSE) between a confidence map extracted for the expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image and a target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as the loss function.

[0020] Cropped images including each of the above lumbar vertebrae can be generated based on the difference in distance between the center points of the lumbar vertebrae included in the X-ray image.

[0021] A cropped image including the sacrum can be generated based on the difference in distance between the center point of the 4th lumbar vertebra and the center point of the 5th lumbar vertebra among the lumbar vertebrae included in the X-ray image.

[0022] The above feature detection model can be trained by using the mean squared error between the confidence map extracted for the predicted vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images and the target confidence map generated for the actual vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images as the loss function.

[0023] The above feature detection model predicts Part Affinity Fields (PAFs), which are information indicating the orientation and location of the limbs between the two parts for the lumbar vertebrae and sacral vertebrae included in the cropped image, and the outputting step can map the predicted PAFs to the X-ray image.

[0024] A feature point analysis device according to one embodiment of the present invention comprises one or more processors; and a memory for loading or storing a program executed by said processors, wherein the program may include instructions for performing operations such as: extracting center points of lumbar vertebrae and sacral vertebrae included in said X-ray image by inputting an X-ray image into a center extraction model; generating crop images including each of said lumbar vertebrae and sacral vertebrae from said X-ray image using said extracted center points of lumbar vertebrae and sacral vertebrae; detecting feature points of lumbar vertebrae and sacral vertebrae included in said crop images by inputting said crop images into a feature detection model; and mapping said detected feature points of lumbar vertebrae and sacral vertebrae onto said X-ray image and outputting them.

[0025] The above center extraction model can be trained by using the Mean Squared Error (MSE) between a confidence map extracted for the expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image and a target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as the loss function.

[0026] The above feature detection model can be trained by using the mean squared error between the confidence map extracted for the predicted vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images and the target confidence map generated for the actual vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images as the loss function.

[0027] The above feature detection model predicts Part Affinity Fields (PAFs), which are information indicating the orientation and location of the limbs between the two parts for the lumbar vertebrae and sacral vertebrae included in the cropped image, and the processor can map the predicted PAFs to the X-ray image and output them. Effects of the invention

[0028] According to one embodiment of the present invention, feature points included in the X-ray region can be detected by analyzing the characteristics of the X-ray image and the morphological and positional characteristics of the spine through deep learning.

[0029] In addition, according to one embodiment of the present invention, a patient's spinal disease can be diagnosed in real time through a feature point analysis network of a relatively small size. Brief explanation of the drawing

[0030] FIG. 1 is a diagram illustrating the configuration of a feature point analysis device according to an embodiment of the present invention. FIG. 2 is a diagram showing the overall structure of a feature point analysis network according to an embodiment of the present invention. FIG. 3 is a diagram showing the process of correcting the center points of the lumbar vertebrae and sacral vertebrae through post-processing according to an embodiment of the present invention. Figure 4 is a diagram showing the structure of a center extraction model according to an embodiment of the present invention. FIG. 5 is a diagram showing an RSC technique according to an embodiment of the present invention. Figure 6 is a diagram showing the result of detecting a center point by applying the conventional cut-out technique and the RSC technique to a center extraction model according to an embodiment of the present invention. Figure 7 is a diagram showing the result of detecting a center point for a frontal X-ray image using a center extraction model that uses the RSC technique according to an embodiment of the present invention. FIG. 8 is a diagram showing the structure of a feature detection model according to an embodiment of the present invention. FIG. 9 is a diagram showing an explanation of CoordConv according to an embodiment of the present invention. FIG. 10 is a diagram showing an example of a case where the GT widths of PAFs are different according to an embodiment of the present invention. FIG. 11 is a diagram showing the result of feature point detection according to whether CoordConv and PAFs are used according to an embodiment of the present invention. FIG. 12 is a diagram showing the result of feature point detection of a feature detection model using CoordConv and PAFs according to an embodiment of the present invention. Specific details for implementing the invention

[0031] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0032] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0033] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0034] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0036] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0038] FIG. 1 is a diagram illustrating the configuration of a feature point analysis device according to an embodiment of the present invention.

[0039] As illustrated in FIG. 1, the feature point analysis device (100) may include one or more processors (110) and a memory (120) that loads or stores a program (130) executed by the processor (110). The components included in the feature point analysis device (100) of FIG. 1 are merely examples, and a person skilled in the art to which the present invention belongs will know that other general-purpose components may be included in addition to the components shown in FIG. 1.

[0040] The processor (110) controls the overall operation of each component of the feature point analysis device (100). The processor (110) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), NPU (Neural Processing Unit), DSP (Digital Signal Processor), or any other type of processor well known in the art of the present invention. Additionally, the processor (110) may perform operations for at least one application or program for executing a method / operation according to various embodiments of the present invention. The feature point analysis device (100) may be equipped with one or more processors.

[0041] The memory (120) stores one or more combinations of various data, instructions, and information used by a component (e.g., processor (110)) included in the feature point analysis device (100). The memory (120) may include volatile memory and / or non-volatile memory.

[0042] The program (130) may include one or more actions in which methods / actions according to various embodiments of the present invention are implemented, and may be stored in memory (120) in the form of software. Here, the action corresponds to an instruction implemented in the program (130). For example, the program (130) may include instructions to perform an action of extracting the center points of the lumbar vertebrae and sacral vertebrae included in an X-ray image, an action of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae, an action of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model, and an action of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae to the X-ray image and outputting them.

[0043] When the program (130) is loaded into memory (120), the processor (110) can perform methods / operations according to various embodiments of the present invention by executing a plurality of operations to implement the program (130).

[0044] The execution screen of the program (130) can be displayed through the display (140). In the case of FIG. 1, the display (140) is depicted as a separate device connected to the feature point analysis device (100), but in the case of the feature point analysis device (100) such as a terminal that a user can carry, such as a smartphone or tablet, the display (140) can be a component of the feature point analysis device (100). The screen displayed on the display (140) may be the result of the execution of the program or before information is input into the program.

[0046] FIG. 2 is a diagram showing the overall structure of a feature point analysis network according to an embodiment of the present invention.

[0047] The X-ray image (202) provided as an example in the present invention includes the lumbar and sacral regions, but this is merely an example and may include the thoracic, cervical, or spinal regions of other vertebrates. In the X-ray image of the lumbar and sacral regions covered in the present invention, bone shadows or tissues in the thoracic region may overlap and appear in a single two-dimensional image. Therefore, it may be very difficult for the feature point analysis device (100) to accurately detect feature points of the X-ray image through a simple convolutional inference process by using such an X-ray image as an input image for a deep learning model. Accordingly, the feature point analysis device (100) of the present invention can more accurately detect feature points included in the X-ray image through the following process.

[0048] First, the processor (110) of the feature point analysis device (100) can perform pre-processing (204) on an X-ray image (202) containing the patient's lumbar and sacral regions as shown in FIG. 2. The processor (110) can resize the X-ray image (202) to a size of 512 x 512. At this time, the processor (110) can perform zero padding on the X-ray image (202) so that the ratio of height to length becomes 1:1 in order to handle various resolutions of the image dataset used for learning.

[0049] Subsequently, the processor (110) can perform histogram equalization so that the cones included in the resized X-ray image (202), which has been zero-padded, are well distinguished from the background and the center point can be easily detected. For example, the processor (110) can improve the contrast of the X-ray image (202) by performing histogram equalization using Contrast Limited Adaptive Histogram Equalization (CLAHE), a method that enhances contrast by redistributing the number of pixels exceeding the threshold to brightness below the threshold.

[0050] Subsequently, the processor (110) can reduce noise present in the image by applying a blurring technique to the contrast-enhanced X-ray image (202). For example, the processor (110) can generate an X-ray image (206) with preprocessing (204) completed by applying Gaussian blurring to the contrast-enhanced X-ray image (202).

[0051] The processor (110) can extract the center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image (206) by inputting the X-ray image (206) with preprocessing (204) completed into a deep learning-based center extraction model (208) (e.g., Pose-Net). For example, the processor (110) can output confidence maps (210) for each center point of the lumbar vertebrae and sacral vertebra included in the X-ray image (206) through the center extraction model (208).

[0052] The processor (110) can determine an X-ray image (214) with possible errors in the center points of the lumbar vertebrae and sacral vertebrae corrected by performing post-processing (212) on the confidence maps (210) for the center points of the lumbar vertebrae and sacral vertebrae, respectively.

[0053] FIG. 3 is a diagram illustrating the process of correcting the center points of the lumbar vertebrae and sacral vertebrae through post-processing according to an embodiment of the present invention. First, referring to FIG. 3, the processor (110) outputs confidence maps (210) ( It is possible to determine whether outliers have occurred using ). Here, an outlier refers to a case where some feature points are clipped within the clipped image when the cone region is clipped using the predicted center point.

[0054] To do this, the processor (110) Each maximum point ( Extract the maximum point of neighboring cones can be obtained. Here, Is and distance difference It can be calculated as.

[0055] Afterwards, the processor (110) The largest value among them The average of the remaining values ​​excluding can be obtained. At this time In this situation, it may mean that the distance between two specific center points output through the center extraction model (208) is abnormally far. Therefore, in this situation, the processor (110) can perform the following post-processing (212) process.

[0056] First, the processor (110) has confidence maps (210) ) channel-wise summation result( ) can be obtained. Each pixel of , the y-coordinate value of the corresponding pixel , and Let j be the index of the value, and the corresponding The confidence maps used when extracting values Let's say.

[0057] The processor (110) The pixel value corresponding to the y-coordinate at that time can be made 0. More specifically, the processor (110) [can] the maximum point of the confidence map located at the top among the confidence maps that are farther from the nearest maximum point among the maximum points Having a y value greater than the sum of the y values You can make the pixel value of 0.

[0058] In addition, the processor (110) The pixel value corresponding to the y-coordinate at that time can be made 0. More specifically, the processor (110) [can] the maximum point of the confidence map located at the bottom among the confidence maps that are farther from the nearest maximum point among the maximum points having a y value smaller than the y value minus . You can make the pixel value of 0.

[0059] Afterwards, the processor (110) has some pixel values ​​that have become 0 At the new maximum point It can extract the maximum point set of existing confidence maps. More specifically, the processor (110) can extract the maximum point set of existing confidence maps. It is determined that the center point of T12 was mispredicted as the center point of L1. Remove and use j of the second step of post-processing (212). and Between By inserting, a new maximum point set{ It can generate.

[0060] The processor (110) can generate cropped images (216) containing each of the lumbar vertebrae and sacral vertebrae using the center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image (214) in which errors have been corrected through post-processing (212). First, the processor (110) can generate cropped images containing each of the lumbar vertebrae based on the difference in distance between the center points of the lumbar vertebrae included in the X-ray image (214). More specifically, the processor (110) can generate crop images corresponding to the lumbar vertebrae by cutting the X-ray image (214) into a square to have a length 2.5 times the y-axis distance difference between the center point of a specific lumbar vertebra and the center point of an adjacent lumbar vertebra (in the case of L1 (lumbar vertebra 1) and L5 (lumbar vertebra 5), ​​the y-axis distance difference with the predicted center point of L2 (lumbar vertebra 2) and L4 (lumbar vertebra 4), respectively, and in the case of L2, L3 (lumbar vertebra 3), and L4, the average of the y-axis distance difference with the predicted center points of the closest upper and lower lumbar vertebrae).

[0061] Meanwhile, since the variation in shape of the sacrum is very large, the processor (110) can generate a cropped image corresponding to the sacrum by cutting the X-ray image (214) into a square so that it has a length 2.5 times the difference in y-axis distance between the center point of L4 and the center point of L5 included in the X-ray image (214).

[0062] The processor (110) can resize the cropped images (216) generated in this way to 256 x 256 size and use them as input to a feature detection model (218).

[0063] The processor (110) can detect feature points of the lumbar vertebrae and sacral vertebrae included in the cropped images (216) by inputting the cropped images (216) into a feature detection model (218). For example, the processor (110) can output confidence maps (220) for four vertices of the lumbar vertebrae and the top two vertices of the sacral vertebrae included in the cropped images (216) through the feature detection model (218).

[0064] Along with this, the processor (110) can predict and output Part Affinity Fields (PAFs) (222), which are information indicating the orientation and location of the two parts of the lumbar vertebrae and sacral vertebrae included in the cropped images (216), through a feature detection model (218).

[0065] Finally, the processor (110) can output a final result image (226) by mapping (224) the detected lumbar and sacral feature points to the initial X-ray image (202).

[0066] Figure 4 is a diagram showing the structure of a center extraction model according to an embodiment of the present invention.

[0067] Referring to Fig. 4, the deep learning-based center extraction model provided in the present invention may have a kernel of 13 x 13 in the convolution layer. In addition, the center extraction model can enable the diagnosis of spondylolisthesis at the L5 (5th lumbar vertebra)-S1 (1st sacral vertebra) level of a patient by additionally detecting upper feature points of the first sacral vertebra as well as the lumbar vertebra.

[0068] The processor (110) of the present invention can output confidence maps of center points corresponding to five lumbar vertebrae and one sacral vertebra, respectively, through a center extraction model having six channels. At this time, each convolution block of the center extraction model can be used in a pre-activate form in the order of normalization, activation, and convolution. At this time, instance normalization is used in the normalization part, and a Rectified Linear Unit (ReLU) function can be used in the activation part.

[0069] The processor (110) can perform training of the center extraction model by generating Gaussian-shaped target confidence maps based on the actual value (Ground Truth, GT) coordinates of the actual center points of the lumbar vertebrae and sacral vertebrae as shown in Equation 1 below.

[0070] <Equation 1>

[0071]

[0072] Here is the position of the pixel, and is the GT coordinates for the actual center points of the lumbar and sacral vertebrae, and represents half the difference in y-axis length between the upper and lower center points of the horizontal top plate of L5.

[0073] The processor (110) can train the center extraction model by using the Mean Squared Error (MSE) between the target confidence maps generated in this way and the confidence maps for the predicted center points of the lumbar vertebrae and sacral vertebrae predicted through the center extraction model as the loss function.

[0074] At this time, the processor (110) has the expected center point of L5 as shown in FIG. 4 ( A center extraction model can be trained using a loss function between the confidence map for ) and the target confidence map of L5, and a loss function between the confidence maps for the predicted center points of all lumbar vertebrae and sacral vertebrae and the target confidence maps. Here, the processor (110) may use 1 as the weight for each of the two loss functions, but this is only one example and is not limited to the above example.

[0075] Meanwhile, the processor (110) can increase the number of X-ray images used for training by using a data augmentation technique so that the center extraction model can be robust to the position, rotation, and size of the vertebral body and the brightness and contrast of the X-ray image. Here, at least one of random translation augmentation, random scale augmentation, random rotation augmentation, or random brightness contrast augmentation may be used as the data augmentation technique.

[0076] At this time, the processor (110) can increase the number of X-ray images used for training by utilizing a Random Spine Cut-out (RSC) technique that randomly selects one of L1 to L4 and cuts out a certain portion. More specifically, the processor (110) can intensively cut out L1 to L4 as shown in FIG. 5 to make the pixel values ​​of arbitrary square areas inside the spine zero. By training a center extraction model using the increased number of training X-ray images utilizing the RSC technique in this way, the processor (110) can robustly predict the center point of the spine even if a part of the spine is obscured in the X-ray image.

[0077] Table 1 below shows the average pixel value inside the rectangle formed by all feature points of each cone in the dataset used in the experiment.

[0078] [Table 1]

[0079]

[0080] Looking at Table 1, it can be seen that the average pixel value inside L5 is significantly larger than the value of the remaining cones. Therefore, the processor (110) can augment the training X-ray image through an RSC technique that intensively cuts out only L1 to L4, excluding L5. In this way, the processor (110) can set the frequency of application of the RSC technique differently for each cone depending on the average pixel value of the image used for training.

[0081] Meanwhile, looking at Table 2 below, it can be seen that when using the RSC technique compared to the cut-out technique frequently used in the field of existing image classification, the error of the center point predicted by the center extraction model is reduced and the number of outlier cases is reduced.

[0082] The internal values ​​in Table 2 are the average of the pixel-level distance differences between the predicted center point and the GT for each cone, and the values ​​in parentheses represent the standard deviation. Additionally, an outlier refers to cases where some feature points are clipped within the cropped image when the cone region is cropped using the predicted center point.

[0083] [Table 2]

[0084]

[0085] Additionally, by looking at Fig. 6, the effect of the RSC technique can be understood more intuitively. Reference numerals 610 and 620 represent the predicted center point when neither the conventional cut-out technique nor the RSC technique is applied, reference numerals 630 and 640 represent the predicted center point when the conventional cut-out technique is applied, and reference numerals 650 and 660 represent the predicted center point when the RSC technique is applied. In Fig. 6, the blue circle and labeling represent the GT, and the red circle represents the predicted result.

[0086] Referring to reference numerals 610, 630, and 650 in Fig. 6, it can be seen that the predicted center point of L1 is shifted to the left even when using the conventional cut-out technique. In contrast, it can be seen that the center point of L1 is predicted more accurately when using the RSC technique. This is because, unlike the cut-out technique which erases an arbitrary area within the image when the center area, such as L1, is obscured in black, the RSC technique specifically erases only the lumbar vertebrae from L1 to L4; therefore, the X-ray image generated through this RSC technique can make the center extraction model more robust.

[0087] Meanwhile, referring to the reference numerals 620, 640, and 660 in Fig. 6, it can be seen that in cases where the boundary parts are difficult to identify visually, such as L2 and L3, the center point is mispredicted for all L1 to L3 in both cases where the RSC technique is not used, but when the RSC technique is used, the center point is predicted relatively accurately.

[0088] Looking at Table 3 below, it can be seen that the center extraction model using the RSC technique performs well not only in left X-ray images but also in frontal X-ray images, and Figure 7 is a diagram showing the results of detecting the center point for a frontal X-ray image using the center extraction model using the RSC technique.

[0089] The internal values ​​in Table 3 are the average of the pixel-level distance differences between the center point prediction and the GT for each cone, and the values ​​in parentheses represent the standard deviation.

[0090] [Table 3]

[0091]

[0093] FIG. 8 is a diagram showing the structure of a feature detection model according to an embodiment of the present invention.

[0094] The processor (110) of the feature point analysis device (100) can detect feature points of the lumbar spine and sacral spine by using two feature detection models with similar structures, taking into account the morphological characteristics difference between the lumbar spine and sacral spine.

[0095] The processor (110) can predict and output confidence maps and PAFs (Part Affinity Fields) for the four vertices of the lumbar spine and the top two vertices of the sacral spine by inputting cropped images for each of the lumbar spine and sacral spine into a feature detection model.

[0096] PAFs are a concept proposed in the field of pose estimation that can enable more accurate pose estimation by utilizing correlations between body structures.

[0097] The feature point analysis device (100) of the present invention can detect feature points more robustly even in situations where occlusion occurs by introducing these PAFs to learn the morphological characteristics of the vertebral body. In addition, the feature point analysis device (100) of the present invention can predict more accurate feature points by utilizing CoordConv to learn the similar positional distribution of vertebral feature points. Referring to FIG. 8, the CoordConv block represents a block that utilizes CoordConv instead of convolution in a convolution block.

[0098] For example, FIG. 9 is a diagram illustrating a description of CoordConv according to an embodiment of the present invention. When performing a convolution operation, CoordConv can concatenate an extra channel to the input feature map, in which the horizontal and vertical lengths of the input feature map are normalized to the range [-1, 1] and filled into the pixel values ​​at each location.

[0099] Therefore, when extracting information about the feature map through convolution operations, the values ​​for x and y coordinates can be utilized.

[0100] Each cone is formed in a shape similar to a square, and the position distribution of each feature point within each cropped cone region may be similar. Therefore, the processor (110) can use CoordConv in the feature detection model to utilize the fact that the position distribution of each feature point is similar within all cropped images, thereby enabling more accurate feature point detection by utilizing position information during feature point detection.

[0101] The feature point analysis device (100) and the processor (110) can perform training of the feature extraction model by generating Gaussian-shaped target confidence maps based on the actual value (Ground Truth, GT) coordinates for the actual vertices of the lumbar vertebrae and sacral vertebrae as shown in Equation 2 below.

[0102] <Equation 2>

[0103]

[0104] Here, is the position of the pixel. represents the GT coordinates for each actual vertebra of the vertebral body. Also, in the case of the lumbar spine, represents 1 / 10 of the average of the diagonal feature point distances of the input vertebrae, and in the case of the sacrum represents 1 / 6 of the average distance between the two actual vertices of the input sacrum.

[0105] Additionally, the processor (110) can generate the GT of PAFs by connecting the two actual vertices at the top and the two actual vertices at the bottom, respectively, and the two line segments connecting the two feature points, respectively. In this case, for the sacrum, since only the two actual vertices at the top exist, the single line segment connecting the two actual vertices can be generated as the GT of PAFs.

[0106] The processor (110) can train a feature detection model by using the mean squared error between the GT of the target confidence maps and PAFs generated in this way and the confidence maps and PAFs for the predicted feature points predicted through the feature detection model as a loss function.

[0107] At this time, the feature detection model for detecting feature points of the lumbar spine may use weights of 1 and 0.001 for the loss function for feature point and PAFs prediction, respectively, but this is just one example and is not limited to the above example.

[0108] In addition, a feature detection model that detects feature points of the sacrum can use weights of 1 and 0.01 for the loss function for feature point and PAF prediction, respectively, but this is just one example and is not limited to the above example.

[0110] FIG. 10 is a diagram showing an example of a case where the GT widths of PAFs are different according to an embodiment of the present invention.

[0111] The performance of the feature point prediction of the feature detection model provided in the present invention can be calculated by mapping the feature point coordinates of the cropped images to the original input image. In this case, considering that the size of each cone is different, the present invention may use not only the pixel-level distance difference (D) but also the relative distance error (RD) as an evaluation metric. Here, the relative distance error can be expressed as Equation 3 below.

[0112] <Equation 3>

[0113]

[0114]

[0115]

[0116] Here, the first and second lines of <Equation 3> are formulas for the process of calculating the RD(%) of the lumbar spine, , represents the lengths of the horizontal and vertical line segments connecting each feature point to nearby feature points, respectively. For example, assuming we are calculating the RD for the left top coordinate of the lumbar spine, It derives the length of the line segment connecting the left top feature point and the right top feature point, and represents the length of the line segment connecting the left top feature point and the left bottom feature point. Next, the third line in <Equation 3> is the formula for the process of calculating the RD(%) of the sacrum, and represents the distance between two feature points of the sacrum. Finally Is of It means coordinates.

[0117] Referring to Fig. 10, (a), (b), and (c) represent the GTs of PAFs with widths of 2 pixels, 4 pixels, and 6 pixels, respectively, and Table 4 below shows the performance of the feature detection model according to the GT width of the PAFs. Here, (D) represents the pixel-level performance of each method, and (RD) represents the performance for relative distance error. Additionally, each value represents the average value for each cone, and the value in parentheses represents the standard deviation.

[0118] [Table 4]

[0119]

[0120] Referring to Table 4, it can be seen that the performance of the feature detection model is best when the GT width of the PAFs is 4 pixels.

[0122] FIG. 11 is a diagram showing the result of feature point detection according to whether CoordConv and PAFs are used according to an embodiment of the present invention.

[0123] Table 5 below shows the feature point detection performance of vertebral bodies with and without the use of CoordConv and PAFs in lateral X-ray images. Referring to Table 5, it can be seen that using CoordConv and PAFs to predict the positional and morphological characteristics of vertebral feature points is highly effective in improving feature point detection performance.

[0124] [Table 5]

[0125]

[0126] By looking at the numerical results above and Figure 11, the effects of CoordConv and PAFs can be better understood. Figure 11 (a) shows the result of feature point detection in a lateral X-ray image with and without the use of CoordConv and PAFs, and Figure 11 (b) shows the result of feature point detection in a frontal X-ray image with and without the use of CoordConv and PAFs.

[0127] Row 1 of Figure 11 shows the result of feature detection when CoordConv and PAFs are not used. In the figure in column 1, the spine region is very faint, and L3 is asymmetrical and has an abnormal shape. In such cases, it can be seen that there is no significant performance improvement even when performing feature detection using CoordConv or PAFs individually.

[0128] As shown in row 2 of Figure 11, when only CoordConv was used, L3b was detected more accurately, but L2a, L2b, and L3a were still not properly detected. This is because the shape of L3 is abnormal, so the locations of feature points are uniquely distributed, making the use of location information ineffective.

[0129] As shown in row 3 of Figure 11, when PAFs are used alone, L2a is predicted more inaccurately, and L2b, L3a, and L3b are still not properly detected. This is because the shape of L3 is abnormal, making it difficult to predict PAFs, and the PAFs that are not accurately predicted actually have a negative impact on predicting the feature points of L3.

[0130] Meanwhile, looking at column 2, it can be seen that when using only CoordConv, L2 feature points were predicted similarly to GT, but the prediction for L2b was slightly inaccurate. Furthermore, when using only PAFs, as shown in the box in column 3, the L2 PAFs were not properly predicted, resulting in even more inaccurate feature detection results than when they were not used.

[0131] However, as shown in row 4 of Figure 11, when both CoordConv and PAFs are used, it can be seen that more accurate feature point localization performance is demonstrated, as in L2b. When both CoordConv and PAFs are used, all feature points are predicted well, which is because PAFs can be predicted more accurately based on location information, thereby enabling more accurate detection.

[0132] Through this, it can be seen that using CoordConv together with PAFs not only overcomes the problems caused by incorrect prediction of PAFs when using only PAFs, but also enables more accurate detection by utilizing the advantages of both techniques.

[0134] FIG. 12 is a diagram showing the result of feature point detection of a feature detection model using CoordConv and PAFs according to an embodiment of the present invention.

[0135] Figure 12(a) shows the result of feature point detection in a lateral X-ray image using CoordConv and PAFs, and Figure 12(b) shows the result of feature point detection in a frontal X-ray image using CoordConv and PAFs. Looking at Figures 12(a) and (b), it can be seen that the feature detection model using CoordConv and PAFs accurately detects feature points included in the patient's X-ray image.

[0137] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0138] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0139] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0140] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0141] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0142] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0144] 100 : Feature point analysis device 110 : Processor 120 : Memory 130 : Program 140 : Display

Claims

Claim 1 delete Claim 2 A feature point analysis method performed by a processor, comprising: a step of training a center extraction model so that predicted center points of lumbar vertebrae and sacral vertebrae included in the X-ray images are extracted in response to X-ray images being input to the center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted predicted center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model so that predicted feature points of lumbar vertebrae and sacral vertebrae included in the crop images are detected in response to the crop images being input to the feature detection model, wherein the center extraction model is trained such that X-ray images used for training are augmented through a Random Spine Cutout (RSC) technique that sets the pixel values ​​of arbitrary square regions within the lumbar vertebrae corresponding to L1 to L4 to zero, and confidence maps for the predicted center points of lumbar vertebrae and sacral vertebrae included in the augmented X-ray images are extracted. Claim 3 In paragraph 2, the center extraction model is a feature point analysis method that uses the Mean Squared Error (MSE) between the confidence map extracted for the predicted center point and the target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as a loss function. Claim 4 In paragraph 3, the target confidence map generated for the actual center point is a feature point analysis method generated based on the actual values ​​(Ground Truth, GT) for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image. Claim 5 A feature point analysis method performed by a processor, comprising: a step of training a center extraction model such that, in response to X-ray images being input to the center extraction model, the center extraction model is extracted with expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray images; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted expected center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model such that, in response to the crop images being input to the feature detection model, expected feature points of the lumbar vertebrae and sacral vertebrae included in the crop images are detected, wherein the crop images including each of the lumbar vertebrae are generated based on the distance difference between the expected center points of the lumbar vertebrae included in the X-ray images. Claim 6 A feature point analysis method performed by a processor, comprising: a step of training a center extraction model so that predicted center points of lumbar vertebrae and sacral vertebrae included in the X-ray images are extracted in response to X-ray images being input to the center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted predicted center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model so that predicted feature points of lumbar vertebrae and sacral vertebrae included in the crop images are detected in response to the crop images being input to the feature detection model, wherein the crop image including the sacral vertebra is generated based on the distance difference between the predicted center point of the 4th lumbar vertebra and the predicted center point of the 5th lumbar vertebra among the lumbar vertebrae included in the X-ray images. Claim 7 A feature point analysis method performed by a processor, comprising: a step of training a center extraction model such that, in response to X-ray images being input to the center extraction model, the center extraction model is extracted with expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray images; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted expected center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model such that, in response to the crop images being input to the feature detection model, expected feature points of the lumbar vertebrae and sacral vertebrae included in the crop images are detected, wherein the feature detection model is trained such that a confidence map for the expected vertices of the lumbar vertebrae and sacral vertebrae included in the crop images is extracted. Claim 8 In claim 7, the feature detection model is a feature point analysis method that uses the mean squared error between the confidence map extracted for the predicted vertices and the target confidence map generated for the actual vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images as a loss function. Claim 9 In claim 8, the target confidence map generated for the actual vertices is a feature point analysis method generated based on actual values ​​for the actual vertices of the lumbar vertebrae and sacral vertebrae included in the cropped images. Claim 10 A feature point analysis method performed by a processor, comprising: a step of training a center extraction model such that, in response to X-ray images being input to the center extraction model, the center extraction model is extracted with expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray images; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray images using the extracted expected center points of the lumbar vertebrae and sacral vertebrae; and a step of training a feature detection model such that, in response to the crop images being input to the feature detection model, expected feature points of the lumbar vertebrae and sacral vertebrae included in the crop images are detected, wherein the feature detection model is trained to predict Part Affinity Fields (PAFs), which are information that simultaneously indicates the orientation and location of the limbs between the two parts for the lumbar vertebrae and sacral vertebrae included in the crop images. Claim 11 delete Claim 12 A feature point analysis method performed by a processor, comprising: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the center extraction model is learned by using the Mean Squared Error (MSE) between a confidence map extracted for the expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image and a target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as a loss function. Claim 13 A feature point analysis method performed by a processor, comprising: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the crop images including each of the lumbar vertebrae are generated based on the distance difference between the center points of the lumbar vertebrae included in the X-ray image. Claim 14 A feature point analysis method performed by a processor, comprising: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the crop image including the sacral vertebrae is generated based on the distance difference between the center point of the 4th lumbar vertebra and the center point of the 5th lumbar vertebra among the lumbar vertebrae included in the X-ray image. Claim 15 A feature point analysis method performed by a processor, comprising: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the feature detection model is learned by using the mean squared error between a confidence map extracted for expected vertices of the lumbar vertebrae and sacral vertebrae included in the crop images and a target confidence map generated for actual vertices of the lumbar vertebrae and sacral vertebrae included in the crop images as a loss function. Claim 16 A feature point analysis method performed by a processor comprises: a step of extracting center points of lumbar vertebrae and sacral vertebrae included in an X-ray image by inputting an X-ray image into a center extraction model; a step of generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; a step of detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and a step of mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the feature detection model predicts Part Affinity Fields (PAFs), which are information indicating the orientation and location of the limbs between two parts for the lumbar vertebrae and sacral vertebrae included in the crop image, and the outputting step is a feature point analysis method in which the predicted PAFs are mapped onto the X-ray image. Claim 17 delete Claim 18 In a feature point analysis device, one or more processors; A feature point analysis device comprising a memory for loading or storing a program executed by the above processor, wherein the program includes instructions for performing operations such as: extracting center points of lumbar vertebrae and sacral vertebrae included in the X-ray image by inputting the X-ray image into a center extraction model; generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; detecting feature points of lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and mapping the detected feature points of lumbar vertebrae and sacral vertebrae to the X-ray image and outputting them. The center extraction model is learned by using the Mean Squared Error (MSE) between a confidence map extracted for the expected center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image and a target confidence map generated for the actual center points of the lumbar vertebrae and sacral vertebrae included in the X-ray image as a loss function. Claim 19 A feature point analysis device comprising: one or more processors; and a memory for loading or storing a program executed by said processors, wherein the program comprises instructions for performing operations such as: extracting center points of lumbar vertebrae and sacral vertebrae included in said X-ray image by inputting an X-ray image into a center extraction model; generating crop images including each of said lumbar vertebrae and sacral vertebrae from said X-ray image using the extracted center points of said lumbar vertebrae and sacral vertebrae; detecting feature points of said lumbar vertebrae and sacral vertebrae included in said crop images by inputting said crop images into a feature detection model; and mapping said detected feature points of said lumbar vertebrae and sacral vertebrae onto said X-ray image and outputting the result, wherein the feature detection model is learned by using the mean squared error between a confidence map extracted for expected vertices of said lumbar vertebrae and sacral vertebrae included in said crop images and a target confidence map generated for actual vertices of said lumbar vertebrae and sacral vertebrae included in said crop images as a loss function. Claim 20 In a feature point analysis device, one or more processors; A feature point analysis device comprising a memory for loading or storing a program executed by the processor, wherein the program includes instructions for performing operations such as: extracting center points of lumbar vertebrae and sacral vertebrae included in the X-ray image by inputting the X-ray image into a center extraction model; generating crop images including each of the lumbar vertebrae and sacral vertebrae from the X-ray image using the extracted center points of the lumbar vertebrae and sacral vertebrae; detecting feature points of the lumbar vertebrae and sacral vertebrae included in the crop images by inputting the crop images into a feature detection model; and mapping the detected feature points of the lumbar vertebrae and sacral vertebrae onto the X-ray image and outputting them, wherein the feature detection model predicts Part Affinity Fields (PAFs), which are information indicating the orientation and location of the limbs between the two parts for the lumbar vertebrae and sacral vertebrae included in the crop image, and the processor maps the predicted PAFs onto the X-ray image and outputs them.

Images