Hip joint X-ray film bone gap automatic segmentation measuring and grading method
Through deep learning technology, the automatic segmentation measurement and grading method of hip joint X-ray bone gap is constructed, which solves the problems of low automation and delayed diagnosis in hip joint X-ray bone gap measurement, and realizes accurate automatic measurement of hip joint bone gap and Lesion grading supports more efficient diagnosis of hip lesions.
Patent Information
- Application Number
- CN202411483439.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has problems with strong subjectivity and low degree of automation in hip joint X-ray bone space measurement, and it is difficult to accurately detect multiple types of areas of interest simultaneously, resulting in delayed diagnosis of hip joint lesions.
Using deep learning-based hip X-ray automatic segmentation measurement and grading method, the hip joint gap target recognition network and global attention segmentation network are constructed, combined with Faster R-CNN and UNet networks, the hip joint bone gap is realized automatically Identify, segment, measure and grading.
It improves the automation degree and segmentation performance of hip gap measurement, enhances measurement accuracy, provides direct basis for disease grading, and supports more efficient diagnosis of hip lesions.
Smart Images

Figure CN119006465A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of image processing, and in particular to a method for automatically segmenting, measuring and grading bone gaps in hip joint X-rays, which can be used for automated measurement and grading of bone joint gaps in diseases such as ankylosing spondylitis. Background Art
[0002] Ankylosing spondylitis (AS) includes a group of interrelated diseases characterized by inflammation of the sacroiliac joints and spine, peripheral joints, and tendon attachment points. Hip joint lesions are the main cause of disability in AS. The etiology is complex, the pathogenesis is still unclear, the early clinical manifestations of patients are atypical, and there is a lack of specific laboratory indicators. These characteristics hinder clinicians from accurately judging hip joint lesions in the early stage, leading to delayed diagnosis, irreversible bone destruction, and even lifelong disability. At present, conventional X-rays are still the main method for detecting hip joint lesions, which can show changes in bone structure and morphology, such as hip joint gap. The present invention evaluates and grades the changes in the hip joint gap and proposes a method for automatically measuring bone gaps on hip joint X-rays based on deep learning.
[0003] Traditional hip joint bone gap measurement is generally performed by professional doctors using a ruler, which is highly subjective, time-consuming, and inefficient. At present, there are also some computer measurement methods, but they require manual preprocessing, such as interactive operations such as cropping and centering, and the degree of automation is not high. Diagnosing ankylosing spondylitis (AS) by X-ray is difficult in itself, which is mainly due to the characteristics of X-ray images, such as the overlap of bones with organs, tissues, etc., the noise caused by the X-ray itself, and the unevenness of X-rays due to different bone densities of different patients, and the lack of clear edges between organs due to the similar X-ray absorption rate of adjacent organs. In addition, the shapes of bones of different patients are also different, especially the obvious differences in the pelvis of men and women. In addition, the different postures of patients in filming will also cause the bones to be in different positions in different images. Therefore, in order to solve these problems, the present disclosure proposes an advanced method for automatically extracting the distance of hip joint bone gap and grading hip joint lesions in X-ray films. Artificial intelligence methods are mostly limited to the problem of hip joint segmentation. Most of the current research focuses on the automatic segmentation of bones in X-ray films, mainly single or multiple bone structures. On the other hand, the focus is on the problem of disease grading and classification, and the present invention attempts to bridge the gap between the two and invents an automatic diagnostic tool based on deep learning for the full-process method of automatic segmentation, measurement and grading of bone gaps in hip X-rays, taking the first step towards fully automatic detection of hip joint lesions.
[0004] The research methods for detection and classification mainly include the following two methods: one is the traditional image processing method that uses artificial design and distinguishes the relevant features to be detected; the other is the deep learning method that learns and automatically extracts features through convolutional neural networks. However, most of the research directions of these methods focus on how to effectively detect a specific area, and cannot accurately detect and classify multiple types of regions of interest at the same time. In the target area segmentation part, before the booming development of deep learning technology, the commonly used image segmentation algorithms are mainly divided into pixel-based and region-based methods. For example, the filtering-based method is to denoise and enhance the image in sequence through local feature description, and then combine the processed results with other segmentation algorithms for target segmentation. However, because medical images have high complexity and lack simple linear features, traditional segmentation algorithms are easily affected by factors such as volume effects, artifacts, and small grayscale differences between soft tissues. With the development of deep learning, some new image segmentation network structures have been proposed one after another. Among them, the UNet network is widely used in the field of medical image segmentation. This is an end-to-end image segmentation network that can obtain more accurate segmentation results with only a small number of samples for training. The network structure is simple and effective. Once proposed, it has caused a research boom, and a large number of methods based on UNet have emerged. Summary of the invention
[0005] The disclosed embodiment provides a method for automatic segmentation, measurement and grading of bone gaps in hip joint X-rays, which can simultaneously complete automatic recognition, segmentation, measurement, evaluation and grading of hip joint bone gaps, and improve recognition and segmentation accuracy, and ultimately be used for automated diagnosis of ankylosing spondylitis in clinical practice.
[0006] According to a first aspect of an embodiment of the present disclosure, a method for automatically segmenting, measuring and grading bone gaps in a hip joint X-ray is provided, comprising: S1. Prepare a dataset of hip joint space recognition and segmentation in hip anteroposterior X-ray images, including: S11, obtaining hip X-ray images of ankylosing spondylitis patients, and extracting L 2D hip anteroposterior X-ray images from the hip X-ray images of ankylosing spondylitis patients; S12, converting the extracted L 2D hip anteroposterior X-ray images into L sample images of the same size by resizing, expanding the sample images by rotation and histogram equalization operations, and adding the generated new hip joint bone gap samples to the total data set; S13, selecting P sample images from the L sample images processed in step S12 as a training set and Q sample images as a test set, L=P+Q; S2. Construct a hip joint gap target recognition network F, which specifically includes: S21, marking samples of the disease with level 4 between the tested area and the bone gap and their location information in the L 2D hip anteroposterior X-ray images, and making them into a data set in VOC2007 format; S22, pre-processing the L 2D hip anteroposterior X-ray images and storing them as images in JPG format; S23, inputting the complete hip anteroposterior X-ray image and the name and position label of the sample with the fourth-level disease between the tested area and the bone gap into the existing Faster R-CNN network for training, to obtain the hip joint gap target recognition network F; S3. Construct a hip gap segmentation network U based on global attention, which includes: S31, marking the outline of the hip joint gap in the L 2D hip anteroposterior X-ray images, and storing them as a mask image; S32, introduce the global attention upsampling GAU module into the existing UNet segmentation network; S33, introducing the L 2D hip anteroposterior X-ray images and the mask image into the UNET segmentation network including the GAU module for training, and using the Adam algorithm to update the weight parameters of each layer in the UNet segmentation network to obtain the hip joint gap segmentation network U; S4, performing target recognition detection and segmentation on the 2D hip anteroposterior X-ray images in the total data set, specifically including: S41, taking the complete hip anteroposterior X-ray image and the VOC2007 format data set obtained in step S21 as input, and performing model training through the hip joint gap segmentation network U; S42, inputting the test set obtained in step S13 into the model trained in step S41 for testing, and obtaining samples of the interested region to be segmented and the bone gap reaching the fourth level of the disease and their position information; S43, using the images of the region to be segmented obtained in step S42 as a segmentation data set, and selecting M images of the region to be segmented from the images of the region to be segmented as a training set, and N images of the region to be segmented as a test set; S44, using the training set obtained in step S43 as input, adjusting the grid parameters of the hip joint gap segmentation network U, and performing iterative training to obtain a hip joint gap segmentation model; S45, inputting the hip joint gap to be segmented in the test set obtained in step S43 into the model trained in step S44 for testing, and obtaining the hip joint gap segmentation result of the 2D hip anteroposterior X-ray image of the test set; S5. Constructing a hip joint bone gap edge extraction and contour distance measurement method, specifically including: S51, performing opening and closing operations and Otsu threshold processing on the segmentation result obtained in step S45, and using an edge operator to extract contours to obtain a continuous, smooth and noise-free hip joint bone gap contour curve; S52, obtaining a distance matrix between two curves according to the hip joint bone gap contour curve and an algorithm for calculating the distance between plane curves, so as to obtain a shortest distance value; S53, converting the pixel distance obtained in step S52 into an actual distance based on the resolution information of the input image, and quantitatively evaluating the error index of the distance measured by the contours obtained by different segmentation methods; S6. Evaluate and grade the measurement results according to the BASRI-h scoring system, wherein the BASRI-h scoring system is a hip joint radiological scoring system, specifically including: S61. Evaluate the severity of hip radiographic lesions in ankylosing spondylitis according to the BASRI-h scoring system and divide it into 5 levels according to the distance of the hip joint space; S62. Obtain the results of ankylosing spondylitis grading based on the level of hip joint space distance and perform error analysis.
[0007] In one embodiment, in step S32, introducing a global attention upsampling GAU module into the existing UNet segmentation network includes: S321, adding a multi-level GAU module to the jump connection of the existing UNet segmentation network, and making each level of the GAU module receive the output of the previous level of GAU and the low-level feature map of the corresponding resolution in the encoding stage as input; S322, enabling each level of GAU module to perform global pooling processing on the high-level feature map output by the previous level of GAU, so as to perform 3×3 convolution processing on the low-level feature map of the corresponding resolution in the encoding stage, and fuse the two processed feature maps and output them; S323, concatenating the output result of each level of GAU module with the input low-level feature map and the result of upsampling the feature map output by the previous level of GAU; S324, performing convolution operation on the concatenated feature map and gradually upsampling it.
[0008] In one embodiment, in step S23, the complete hip anteroposterior X-ray image and the name and location label of the sample with the fourth-level disease between the tested area and the bone gap are input into the existing Faster R-CNN network for training, and the hip joint gap target recognition network F is obtained, which includes: S231, obtaining a feature map of a 2D anteroposterior hip X-ray image containing only the hip joint bone gap through a convolutional neural network, and generating an anchor point on the feature map for mapping each pixel point to the original X-ray image; wherein the center of each anchor point is provided with nine candidate boxes; S232, determining whether each candidate box contains a target through a binary classification network, and outputting a probability value of containing the target; S233, calculating the deviation of the target determined by the binary classifier through the regression branch of the frame regression network, and obtaining the translation amount and transformation scale required for each candidate frame; S234, calculating the softmax cross entropy loss function of the binary classification network and the smooth L1 loss function of the bounding box regression network, adding the softmax cross entropy loss function and the smooth L1 loss function of the bounding box regression network, and minimizing the addition result using the Adam optimizer; S235, back-propagating the loss function obtained in step S234 through the optimizer to optimize the binary classification network and the bounding box regression network until the loss function converges, and obtaining the probability value of each candidate box containing the target and the translation amount and transformation scale required for each candidate box; S236. Receive the result of step S235 through the extraction layer of the Faster R-CNN network, obtain a more accurate candidate box after translation and scale transformation, and eliminate the candidate boxes that overlap at the local position, and only retain the candidate box with the highest probability given by the previous binary classification network at this position, to obtain the hip joint gap target recognition network F.
[0009] In one embodiment, in step S51, the segmentation result obtained in step S45 is subjected to an opening and closing operation and an Otsu threshold processing, and an edge operator is used to extract a contour to obtain a continuous, smooth and noise-free hip joint bone gap contour curve, including: S511, performing opening and closing operations on the segmented hip joint bone gap image, the basic operations are expansion operation and corrosion operation; wherein, The dilation operation can fill the noise in the image. Suppose the image set is f and the structure element is s, then the expression of the dilation operation is: ; The erosion operation can eliminate small edges to achieve a smooth edge effect. Its expression is: ; The opening operation is to corrode first and then expand, which can smooth the contour of the image and remove the noise in the image. Its expression is: ; The closing operation is to dilate first and then corrode, which can smooth the contour and fill the holes. Its expression is: ; in, is the input image, is a structural element, and They are and The domain of S512, perform Otsu threshold processing on the hip joint bone gap image after the opening and closing operation. For image I, assume that the image size is M×N, the threshold is T, the pixels in the image whose grayscale value is lower than the threshold T are N0, and the pixels whose grayscale value is higher than the threshold T are N1. Let n pixels higher than the threshold T be the foreground, and Ln pixels lower than the threshold T be the background. Let P0 and P1 represent the corresponding probabilities of the foreground and background respectively, U0 and U1 correspond to the average grayscale of N0 and N1 respectively, X i is the number of pixels with gray level i, C i That is, the probability of a pixel with grayscale i appearing is: ; ; ; ; ; It can be deduced that when the inter-class variance g (n) reaches the maximum, it is the optimal threshold T for image segmentation, where: ; ; S513, using edge operator to extract contour, the calculation formula is: ; Using templates to represent it, we have: ; The calculation principle of Robert gradient is to replace the gradient value of the central pixel (x, y) with the sum of the absolute value calculated by template h1 and the absolute value calculated by template h2. According to this method, the image is traversed in sequence to obtain a gradient map to achieve edge extraction.
[0010] In one embodiment, in step S52, the step of obtaining a distance matrix between two curves according to the hip joint bone gap contour curve and the algorithm for calculating the distance between the plane curves to obtain the shortest distance value includes: S521, discretize the curve A, calculate the minimum distance from each discrete point to the curve B, and select a number of point pairs with large distances and satisfying the "small size" of the distances between adjacent points on the curve A and the curve B as approximate solutions; S522, according to the characteristics of the curves at each point pair, determine which of the four types of points may exist near each point pair, establish a corresponding optimization model and perform local optimization, and select the maximum distance value in the optimization result as the one-way Hausdorff distance between the two plane curves; S523: Convert the calculation of the one-way Hausdorff distance between the two plane curves into the calculation of the minimum distance from a point to a curve.
[0011] Compared with the prior art, the method for automatically segmenting, measuring and grading bone gaps in hip joint X-rays provided by the embodiment of the present disclosure has the following advantages: First, a more intelligent method for measuring hip joint gap: Considering that most existing methods for measuring joint bone gaps require manual preprocessing, such as cropping, centering and other interactive operations, and the degree of automation is not high, and the current artificial intelligence methods are mostly limited to the problem of hip joint segmentation, most of which focus on the automatic segmentation of bones in X-rays, mainly single or multiple bone structures. Therefore, the automatic segmentation, measurement and grading method of bone gaps in hip joint X-rays proposed in the present invention can integrate the steps of hip joint gap identification, bone gap segmentation, edge extraction, distance measurement and disease grading to form a complete automated measurement and diagnosis process.
[0012] Second, it has better segmentation performance and measurement accuracy: The method for automatic segmentation, measurement and grading of bone gaps in hip X-rays provided by the present invention introduces a global attention upsampling GAU module into the existing UNet network, constructs a segmentation network U for hip bone gap measurement, and can obtain accurate segmentation results. Edge extraction and bone gap distance measurement are performed on the segmentation results, avoiding interference from other bone structures and greatly improving the measurement accuracy.
[0013] Third, it has the effect of directly grading the condition, providing a more direct and efficient basis for doctors’ diagnosis: The method for automatic segmentation, measurement and grading of bone gaps in hip X-rays provided by the present invention uses the BASRI-h scoring system, which is a reliable, disease-specific, and change-sensitive method for grading hip X-ray images of ankylosing spondylitis (AS) into five severity levels.
[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0016] Figure 1 is a flowchart of the implementation of the present disclosure; Figure 2 It is a structural diagram of the global attention upsampling GAU module introduced in this disclosure; Figure 3 is a structure diagram of a liver organ segmentation network U in the present disclosure; Figure 4 is a hip joint bone gap recognition result diagram obtained by the present disclosure through a hip joint gap target recognition network and a hip joint gap segmentation network; Figure 5 This is a hip joint bone gap segmentation result diagram obtained by the present invention through the hip joint gap target recognition network and the hip joint gap segmentation network. DETAILED DESCRIPTION
[0017] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0018] Figure 1 The flow chart of the method for automatic segmentation, measurement and grading of bone gaps in hip joint X-rays provided by the embodiment of the present disclosure. Figure 1 As shown, the method includes the following steps S1 to S6: S1. Prepare a dataset of hip joint space recognition and segmentation in hip anteroposterior X-ray images, including: S11. Obtain hip X-ray images of ankylosing spondylitis patients, and extract L 2D hip AP X-ray images from the hip X-ray images of ankylosing spondylitis patients; illustratively, 120 2D hip AP X-ray images can be extracted from the hip X-ray images of ankylosing spondylitis patients.
[0019] S12, converting the extracted L 2D hip anteroposterior X-ray images into L sample images of the same size by resizing, expanding the sample images by rotation and histogram equalization operations, and adding the generated new hip joint bone gap samples to the total data set; S13. Select P sample images as a training set and Q sample images as a test set from the L sample images processed in step S12, where L=P+Q. For example, 70% of the sample images can be selected from the L sample images as a training set and 30% of the sample images can be selected as a test set.
[0020] S2. Construct a hip joint gap target recognition network F, which specifically includes: S21, marking samples of the disease with level 4 between the tested area and the bone gap and their location information in the L 2D hip anteroposterior X-ray images, and making them into a data set in VOC2007 format; S22, pre-processing the L 2D hip anteroposterior X-ray images and storing them as images in JPG format; S23, inputting the complete hip anteroposterior X-ray image and the name and position label of the sample with the fourth-level disease between the tested area and the bone gap into the existing Faster R-CNN network for training, to obtain the hip joint gap target recognition network F; In one embodiment, in step S23, the complete hip anteroposterior X-ray image and the name and location label of the sample with the fourth-level disease between the tested area and the bone gap are input into the existing Faster R-CNN network for training, and the hip joint gap target recognition network F is obtained, which includes: S231, obtaining a feature map of a 2D anteroposterior hip X-ray image containing only the hip joint bone gap through a convolutional neural network, and generating an anchor point on the feature map for mapping each pixel point to the original X-ray image; wherein the center of each anchor point is provided with nine candidate boxes; S232, determining whether each candidate box contains a target through a binary classification network, and outputting a probability value of containing the target; S233, calculating the deviation of the target determined by the binary classifier through the regression branch of the frame regression network, and obtaining the translation amount and transformation scale required for each candidate frame; S234, calculating the softmax cross entropy loss function of the binary classification network and the smooth L1 loss function of the bounding box regression network, and adding the softmax cross entropy loss function and the smooth L1 loss function of the bounding box regression network to obtain a loss function , and use the Adam optimizer to minimize the loss function ; Among them, the loss function The calculation formula is as follows: ; ; ; in, is the cross entropy loss function of the binary classification network, is the smoothL1 loss function of the bounding box regression network; is the weight parameter, which is 10. It is a parameter that controls the smoothness of the smooth L1 loss function, and its value is 3. is the number of candidate boxes, is the size of the feature map, It indicates the probability that the i-th candidate box is predicted by the binary classification network to contain the target. Indicates that the i-th candidate box only contains the real label with target 1, represents the offset of the i-th candidate box predicted by the bounding box regression network, Represents the true offset of the i-th candidate box relative to the annotated area.
[0021] S235, back-propagating the loss function obtained in step S234 through the optimizer to optimize the binary classification network and the bounding box regression network until the loss function converges, and obtaining the probability value of each candidate box containing the target and the translation amount and transformation scale required for each candidate box; S236. Receive the result of step S235 through the extraction layer of the Faster R-CNN network, obtain a more accurate candidate box after translation and scale transformation, and eliminate the candidate boxes that overlap at the local position, and only retain the candidate box with the highest probability given by the previous binary classification network at this position, to obtain the hip joint gap target recognition network F.
[0022] S3. Construct a hip gap segmentation network U based on global attention, which includes: S31, marking the outline of the hip joint gap in the L 2D hip anteroposterior X-ray images and storing them as a mask image to provide conditions for the next step of segmentation; S32, introduce the global attention upsampling GAU module into the existing UNet segmentation network; In one embodiment, in step S32, introducing a global attention upsampling GAU module into the existing UNet segmentation network includes: S321, adding a multi-level GAU module to the jump connection of the existing UNet segmentation network, and making each level of the GAU module receive the output of the previous level of GAU and the low-level feature map of the corresponding resolution in the encoding stage as input; S322, enabling each level of GAU module to perform global pooling processing on the high-level feature map output by the previous level of GAU, so as to perform 3×3 convolution processing on the low-level feature map of the corresponding resolution in the encoding stage, and fuse the two processed feature maps and output them; S323, concatenating the output result of each level of GAU module with the input low-level feature map and the result of upsampling the feature map output by the previous level of GAU; S324, performing convolution operation on the concatenated feature map and gradually upsampling it.
[0023] S33, introducing the L 2D hip X-ray images and the mask image into the UNet segmentation network including the GAU module for training, and using the Adam algorithm to update the weight parameters of each layer in the UNet segmentation network to obtain the hip joint gap segmentation network U. In this embodiment, Figure 2 As shown in the figure, the global attention upsampling GAU module includes a 3×3 convolution layer, a 1×1 convolution layer, a global pooling layer, an input low-level feature map, and a high-level feature map. The 3×3 convolution layer is used to perform channel processing on the low-level feature map, and the global pooling layer and the 1×1 convolution layer are used to process the high-level feature map. The processed high-level feature map is used to select and fuse the low-level feature map, and the result after feature fusion is output. In this example, a multi-level GAU module is added to the jump connection of the existing UNet segmentation network. The GAU module of each level receives the output of the previous level GAU and the low-level feature map of the corresponding resolution in the encoding stage as input; each level of the GAU module performs global pooling processing on the high-level feature map output by the previous level GAU, performs 3×3 convolution processing on the low-level feature map of the corresponding resolution in the encoding stage, and fuses the two processed feature maps for output; then the output result of each level of the GAU module is spliced with the input low-level feature map and the upsampling result of the feature map output by the previous level GAU, and the spliced feature map is convolved and gradually upsampled; the training set image and the mask image are input into the segmentation network introduced with the GAU module, and the weight parameters of each layer in the network are updated using the Adam algorithm to obtain the hip joint gap segmentation network U, as shown in FIG. Figure 3 shown.
[0024] S4, performing target recognition detection and segmentation on the 2D hip anteroposterior X-ray images in the total data set, specifically including: S41, taking the complete hip anteroposterior X-ray image and the VOC2007 format data set obtained in step S21 as input, and performing model training through the hip joint gap segmentation network U; S42, inputting the test set obtained in step S13 into the model trained in step S41 for testing, and obtaining samples of the interested region to be segmented and the bone gap reaching the fourth level of the disease and their position information; S43. Use the images of the region to be segmented obtained in step S42 as a segmentation data set, and select M images of the region to be segmented from the images of the region to be segmented as a training set, and N images of the region to be segmented as a test set; illustratively, select 70% of the images of the region to be segmented from the images of the region to be segmented as a training set, and 30% of the images of the region to be segmented as a test set.
[0025] S44, using the training set obtained in step S43 as input, adjusting the grid parameters of the hip joint gap segmentation network U, and performing iterative training to obtain a hip joint gap segmentation model; S45, inputting the hip joint gap to be segmented in the test set obtained in step S43 into the model trained in step S44 for testing, and obtaining the hip joint gap segmentation result of the 2D hip anteroposterior X-ray image of the test set; S5. Constructing a hip joint bone gap edge extraction and contour distance measurement method, specifically including: S51, performing opening and closing operations and Otsu threshold processing on the segmentation result obtained in step S45, and using an edge operator to extract contours to obtain a continuous, smooth and noise-free hip joint bone gap contour curve; In one embodiment, in step S51, the segmentation result obtained in step S45 is subjected to an opening and closing operation and an Otsu threshold processing, and an edge operator is used to extract a contour to obtain a continuous, smooth and noise-free hip joint bone gap contour curve, including: S511, performing opening and closing operations on the segmented hip joint bone gap image, the basic operations are expansion operation and corrosion operation; wherein, The dilation operation can fill the noise in the image. Suppose the image set is f and the structure element is s, then the expression of the dilation operation is: ; The erosion operation can eliminate small edges to achieve a smooth edge effect. Its expression is: ; The opening operation is to corrode first and then expand, which can smooth the contour of the image and remove the noise in the image. Its expression is: ; The closing operation is to dilate first and then corrode, which can smooth the contour and fill the holes. Its expression is: ; in, is the input image, is a structural element, and They are and The domain of S512, perform Otsu threshold processing on the hip joint bone gap image after the opening and closing operation. For image I, assume that the image size is M×N, the threshold is T, the pixels in the image whose grayscale value is lower than the threshold T are N0, and the pixels whose grayscale value is higher than the threshold T are N1. Let n pixels higher than the threshold T be the foreground, and Ln pixels lower than the threshold T be the background. Let P0 and P1 represent the corresponding probabilities of the foreground and background respectively, U0 and U1 correspond to the average grayscale of N0 and N1 respectively, X iis the number of pixels with gray level i, C i That is, the probability of a pixel with grayscale i appearing is: ; ; ; ; ; It can be deduced that when the inter-class variance g (n) reaches the maximum, it is the optimal threshold T for image segmentation, where: ; ; S513, using edge operator to extract contour, the calculation formula is: ; Using templates to represent it, we have: ; The calculation principle of Robert gradient is to replace the gradient value of the central pixel (x, y) with the sum of the absolute value calculated by template h1 and the absolute value calculated by template h2. According to this method, the image is traversed in sequence to obtain a gradient map to achieve edge extraction. Robert is an operator that uses a local difference operator to find edges.
[0026] Considering that the segmentation result is not fine enough, the mathematical morphology theory is used in this embodiment to improve the problems of roughness, protrusions, depressions, holes, etc. in the segmentation result.
[0027] S52, obtaining a distance matrix between two curves according to the hip joint bone gap contour curve and an algorithm for calculating the distance between plane curves, so as to obtain a shortest distance value; In step S52, the distance matrix between the two curves is obtained according to the hip joint bone gap contour curve and the algorithm for calculating the distance between the plane curves to obtain the shortest distance value, including: S521, discretize the curve A, calculate the minimum distance from each discrete point to the curve B, and select a number of point pairs with large distances and satisfying the "small size" of the distances between adjacent points on the curve A and the curve B as approximate solutions; S522, according to the characteristics of the curves at each point pair, determine which of the four types of points may exist near each point pair, establish a corresponding optimization model and perform local optimization, and select the maximum distance value in the optimization result as the one-way Hausdorff distance between the two plane curves; S523, converting the calculation of the one-way Hausdorff distance between the two plane curves into the calculation of the minimum distance from a point to a curve, the calculation process is simple and effective.
[0028] S53, converting the pixel distance obtained in step S52 into an actual distance based on the resolution information of the input image, and quantitatively evaluating the error index of the distance measured by the contours obtained by different segmentation methods; S6. Evaluate and grade the measurement results according to the BASRI-h scoring system, wherein the BASRI-h scoring system is a hip joint radiological scoring system, specifically including: S61. Evaluate the severity of hip radiographic lesions in ankylosing spondylitis according to the BASRI-h scoring system and divide it into 5 levels according to the distance of the hip joint space; S62. Obtain the results of ankylosing spondylitis grading based on the level of hip joint space distance and perform error analysis.
[0029] This embodiment can classify symptoms according to the measurement results.
[0030] Compared with the prior art, the method for automatically segmenting, measuring and grading bone gaps in hip joint X-rays provided by the embodiment of the present disclosure has the following advantages: First, a more intelligent method for measuring hip joint gap: Considering that most existing methods for measuring joint bone gaps require manual preprocessing, such as cropping, centering and other interactive operations, and the degree of automation is not high, and the current artificial intelligence methods are mostly limited to the problem of hip joint segmentation, most of which focus on the automatic segmentation of bones in X-rays, mainly single or multiple bone structures. Therefore, the automatic segmentation, measurement and grading method of bone gaps in hip joint X-rays proposed in the present invention can integrate the steps of hip joint gap identification, bone gap segmentation, edge extraction, distance measurement, disease grading and so on to form a complete automated measurement and diagnosis process.
[0031] Second, it has better segmentation performance and measurement accuracy: The method for automatic segmentation, measurement and grading of bone gaps in hip X-rays provided by the present invention introduces a global attention upsampling GAU module into the existing UNet network, constructs a segmentation network U for hip bone gap measurement, and can obtain accurate segmentation results. Edge extraction and bone gap distance measurement are performed on the segmentation results, avoiding interference from other bone structures and greatly improving the measurement accuracy.
[0032] Third, it has the effect of directly grading the condition, providing a more direct and efficient basis for doctors’ diagnosis: The method for automatic segmentation, measurement and grading of bone gaps in hip X-rays provided by the present invention uses the BASRI-h scoring system, which is a reliable, disease-specific, and change-sensitive method for grading hip X-ray images of ankylosing spondylitis (AS) into five severity levels.
[0033] It should be noted that the technical effects of the present disclosure can be further illustrated by the following simulations.
[0034] 1. Simulation conditions: This experiment was conducted on a desktop computer with Windows 10 system using Python 3.5 programming language. The network model was implemented using Tensoflow 1.6.0 and Pytorch 1.1.0, and GeForce RTX 1080 Ti, Cuda 10.0, and CuDNN 7.4 were used for computational acceleration. The simulation data contained 420 2D hip X-ray images, anteroposterior (AP) projections, with an image resolution of 2021x2021, from 420 patients.
[0035] 2. Simulation content: Simulation 1, the image of the hip bone gap area obtained in simulation 1 and its position label are input into the Faster R-CNN network in pairs for training, and iterate 4000 times; in order to take into account the accuracy and recall rate of the model, the threshold of the confidence score for judging whether a predicted box contains the target is 0.4; considering that the probability of overlap between different types is low, the non-maximum suppression strategy is adopted, and the threshold of the candidate box overlap area when removing redundant boxes is set to 0.1 to minimize overlapping redundant boxes. After each experiment is completed, the recognition model accuracy and other indicators are calculated. In recent years, the widely used evaluation annotation for target detection is "Average Precision (AP)", which was first introduced in VOC2007. AP is defined as the average detection accuracy at different recall rates, which is an evaluation under a specific category. In order to compare the performance of all target types, mean AP (mAP) is often used as the final performance metric, which is averaged over all target types. The recognition results are shown in Figure 2. Figure 4 As shown in the figure, the mAPs of the bone gap to be segmented and the four-level disease are 0.86 and 0.79, respectively.
[0036] In simulation 2, the dataset is used to test the trained model under the above conditions and calculate the relevant performance indicators.
[0037] For hip joint bone gap segmentation, the mask image and the original image of the 2D hip X-ray are input into the improved segmentation network U in the present disclosure in pairs for training. The number of iterations is set to 50 times. The results are compared with the existing segmentation networks, including the full convolutional network, UNet++, UNet+residual network and UNet+attention mechanism. Figure 5 shown.
[0038] The average Dice coefficient index of the segmentation results of the improved segmentation network U in the present disclosure and the existing segmentation networks UNet, UNet++ and AttentionUNet is calculated by the following formula: ; Among them, X is the segmentation result of the segmentation network, Y is the labeled true result, and the value range of the Dice coefficient is [0,1]. The larger the Dice coefficient, the greater the overlap between the network's segmentation result and the true result, and the better the segmentation effect. The smaller the value of the Hausdorff distance, the smaller the error between the segmentation result and the true result.
[0039] Another evaluation metric is the mean intersection over union (MIoU): ; MIoU (Mean Intersection over Union) is an evaluation indicator for semantic segmentation, which represents the average intersection over union ratio, that is, the average IoU value of each category in the data set. represents the true value, represents the predicted value, Indicates the number of samples that need to be predicted. Indicates that Prediction , Indicates that Prediction , Indicates that Prediction In fact, the denominator express All prediction results of All predictions are The value of , minus an overlapping .
[0040] This is a standard evaluation metric for segmentation problems, which calculates the overlap ratio between the intersection of two sets and their union, that is, the intersection ratio between the true segmentation and the segmentation predicted by the system. This ratio can be redefined as the number of true positive examples divided by the total number (including true positive examples, false negative examples, and false positive examples).
[0041] Table 1 is a comparison table of the indicators of the hip joint bone gap segmentation network U disclosed in the present invention and other existing network segmentation results.
[0042] Table 1 As can be seen from Table 1, compared with the existing segmentation networks, i.e., the full convolution network, UNet++, UNet+ residual network, and UNet+ attention mechanism network, the present disclosure introduces the GAU module in the UNet network, and the obtained segmentation network U has better segmentation performance. Compared with the direct splicing of low-level and high-level feature maps in the UNet network, the GAU module uses the rich semantic information of the high-level feature map to guide the low-level feature mapping of different scales, which can provide more accurate category pixel spatial positioning information, thereby improving the segmentation performance. In simulation 3, the segmented image is subjected to opening and closing operations and Otsu threshold processing in turn, and then the contour is extracted using the edge operator, thereby obtaining a continuous, smooth, and noise-free hip joint bone gap contour curve. Using the obtained hip joint bone gap contour curve, the distance matrix between the two curves is obtained by calculating the distance between the plane curves, and the value of the shortest distance is obtained; based on the resolution of the input image and other information, the obtained pixel distance is converted into the actual distance, and the error index of the distance measured by the contour obtained by different segmentation methods is quantitatively evaluated.
[0043] Table 2 is an effect evaluation table of the automatic measurement results of the hip joint gap distance disclosed in the present invention.
[0044] Table 2 Among them, SD=standard deviation, MAD=mean absolute deviation, RMSE=root mean square error, F-test=F test, joint hypothesis test.
[0045] Table 3 is the BASRI-h scoring system used in the present disclosure, which is an evaluation index for grading hip joint radiographic lesions in ankylosing spondylitis, including 5 levels of severity.
[0046] Table 3 Table 4 is a result evaluation table of ankylosing spondylitis grading obtained according to the BASRI-h scoring system after the hip joint bone gap distance is automatically measured by the present invention.
[0047] Table 4 According to Table 4, the relative errors of level 0 and level 1 are larger due to less data, while the errors of level 2 and level 3 are smaller. It should be noted that the actual relative error is the percentage of the absolute error to the fact.
[0048] Although preferred embodiments of the present disclosure have been described, additional changes and modifications may be made to these embodiments once those skilled in the art are aware of the basic inventive concepts.
[0049] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure belong to the scope of equivalent technology of the present disclosure, the present disclosure is also intended to include these modifications and variations.
Claims
1. A method for automatic segmentation, measurement and grading of bone gaps in hip joint X-rays, characterized in that: The method comprises: S1. Prepare a dataset of hip joint space recognition and segmentation in hip anteroposterior X-ray images, including: S11, obtaining hip X-ray images of ankylosing spondylitis patients, and extracting L 2D hip anteroposterior X-ray images from the hip X-ray images of ankylosing spondylitis patients; S12, converting the extracted L 2D hip anteroposterior X-ray images into L sample images of the same size by resizing, expanding the sample images by rotation and histogram equalization operations, and adding the generated new hip joint bone gap samples to the total data set; S13, selecting P sample images from the L sample images processed in step S12 as a training set and Q sample images as a test set, L=P+Q; S2. Construct a hip joint gap target recognition network F, which specifically includes: S21, marking samples of the disease with level 4 between the tested area and the bone gap and their location information in the L 2D hip anteroposterior X-ray images, and making them into a data set in VOC2007 format; S22, pre-processing the L 2D hip anteroposterior X-ray images and storing them as images in JPG format; S23, inputting the complete hip anteroposterior X-ray image and the name and position label of the sample with the fourth-level disease between the tested area and the bone gap into the existing Faster R-CNN network for training, to obtain the hip joint gap target recognition network F; S3. Construct a hip gap segmentation network U based on global attention, which includes: S31, marking the outline of the hip joint gap in the L 2D hip anteroposterior X-ray images, and storing them as a mask image; S32, introduce the global attention upsampling GAU module into the existing UNet segmentation network; S33, introducing the L 2D hip anteroposterior X-ray images and the mask image into the UNet segmentation network including the GAU module for training, and using the Adam algorithm to update the weight parameters of each layer in the UNet segmentation network to obtain the hip joint gap segmentation network U; S4, performing target recognition detection and segmentation on the 2D hip anteroposterior X-ray images in the total data set, specifically including: S41, taking the complete hip anteroposterior X-ray image and the VOC2007 format data set obtained in step S21 as input, and performing model training through the hip joint gap segmentation network U; S42, inputting the test set obtained in step S13 into the model trained in step S41 for testing, and obtaining samples of the interested region to be segmented and the bone gap reaching the fourth level of the disease and their position information; S43, using the images of the region to be segmented obtained in step S42 as a segmentation data set, and selecting M images of the region to be segmented from the images of the region to be segmented as a training set, and N images of the region to be segmented as a test set; S44, using the training set obtained in step S43 as input, adjusting the grid parameters of the hip joint gap segmentation network U, and performing iterative training to obtain a hip joint gap segmentation model; S45, inputting the hip joint gap to be segmented in the test set obtained in step S43 into the model trained in step S44 for testing, and obtaining the hip joint gap segmentation result of the 2D hip anteroposterior X-ray image of the test set; S5. Constructing a hip joint bone gap edge extraction and contour distance measurement method, specifically including: S51, performing opening and closing operations and Otsu threshold processing on the segmentation result obtained in step S45, and using an edge operator to extract contours to obtain a continuous, smooth and noise-free hip joint bone gap contour curve; S52, obtaining a distance matrix between two curves according to the hip joint bone gap contour curve and an algorithm for calculating the distance between plane curves, so as to obtain a shortest distance value; S53, converting the pixel distance obtained in step S52 into an actual distance based on the resolution information of the input image, and quantitatively evaluating the error index of the distance measured by the contours obtained by different segmentation methods; S6. Evaluate and grade the measurement results according to the BASRI-h scoring system, wherein the BASRI-h scoring system is a hip joint radiological scoring system, specifically including: S61. Evaluate the severity of hip radiographic lesions in ankylosing spondylitis according to the BASRI-h scoring system and divide it into 5 levels according to the distance of the hip joint space; S62. Obtain the results of ankylosing spondylitis grading based on the level of hip joint space distance and perform error analysis.
2. The method according to claim 1, characterized in that In step S32, the introduction of a global attention upsampling GAU module into the existing UNet segmentation network includes: S321, adding a multi-level GAU module to the jump connection of the existing UNet segmentation network, and making each level of the GAU module receive the output of the previous level of GAU and the low-level feature map of the corresponding resolution in the encoding stage as input; S322, enabling each level of GAU module to perform global pooling processing on the high-level feature map output by the previous level of GAU, so as to perform 3×3 convolution processing on the low-level feature map of the corresponding resolution in the encoding stage, and fuse the two processed feature maps and output them; S323, concatenating the output result of each level of GAU module with the input low-level feature map and the result of upsampling the feature map output by the previous level of GAU; S324, performing convolution operation on the concatenated feature map and gradually upsampling it.
3. The method according to claim 1, characterized in that In step S23, the complete hip anteroposterior X-ray image and the name and location label of the sample with the fourth-level disease between the tested area and the bone gap are input into the existing Faster R-CNN network for training, and the hip joint gap target recognition network F is obtained, which includes: S231, obtaining a feature map of a 2D anteroposterior hip X-ray image containing only the hip joint bone gap through a convolutional neural network, and generating an anchor point on the feature map for mapping each pixel point to the original X-ray image; wherein the center of each anchor point is provided with nine candidate boxes; S232, determining whether each candidate box contains a target through a binary classification network, and outputting a probability value of containing the target; S233, calculating the deviation of the target determined by the binary classifier through the regression branch of the frame regression network, and obtaining the translation amount and transformation scale required for each candidate frame; S234, calculating a softmax cross entropy loss function of the binary classification network and a smooth L1 loss function of the bounding box regression network, adding the softmax cross entropy loss function and the smooth L1 loss function of the bounding box regression network to obtain a loss function, and minimizing the loss function using an Adam optimizer; S235, back-propagating the loss function obtained in step S234 through the optimizer to optimize the binary classification network and the bounding box regression network until the loss function converges, and obtaining the probability value of each candidate box containing the target and the translation amount and transformation scale required for each candidate box; S236. Receive the result of step S235 through the extraction layer of the Faster R-CNN network, obtain a more accurate candidate box after translation and scale transformation, and eliminate the candidate boxes that overlap at the local position, and only retain the candidate box with the highest probability given by the previous binary classification network at this position, to obtain the hip joint gap target recognition network F.
4. The method according to claim 1, characterized in that: In step S51, the segmentation result obtained in step S45 is subjected to opening and closing operations and Otsu threshold processing, and contour extraction is performed using an edge operator to obtain a continuous, smooth and noise-free hip joint bone gap contour curve, including: S511, performing opening and closing operations on the segmented hip joint bone gap image, the basic operations are expansion operation and corrosion operation; wherein, The dilation operation can fill the noise in the image. Suppose the image set is f and the structure element is s, then the expression of the dilation operation is: ; The erosion operation can eliminate small edges to achieve a smooth edge effect. Its expression is: ; The opening operation is to corrode first and then expand, which can smooth the contour of the image and remove the noise in the image. Its expression is: ; The closing operation is to dilate first and then corrode, which can smooth the contour and fill the holes. Its expression is: ; in, is the input image, is a structural element, and They are and The domain of S512, perform Otsu threshold processing on the hip joint bone gap image after the opening and closing operation. For image I, assume that the image size is M×N, the threshold is T, the pixels in the image whose grayscale value is lower than the threshold T are N0, and the pixels whose grayscale value is higher than the threshold T are N1. Let n pixels higher than the threshold T be the foreground, and Ln pixels lower than the threshold T be the background. Let P0 and P1 represent the corresponding probabilities of the foreground and background respectively, U0 and U1 correspond to the average grayscale of N0 and N1 respectively, X i is the number of pixels with gray level i, C i That is, the probability of a pixel with grayscale i appearing is: ; ; ; ; ; It can be deduced that when the inter-class variance g (n) reaches the maximum, it is the optimal threshold T for image segmentation, where: ; ; S513, using edge operator to extract contour, the calculation formula is: ; Using templates to represent it, we have: ; The calculation principle of Robert gradient is to replace the gradient value of the central pixel (x, y) with the sum of the absolute value calculated by template h1 and the absolute value calculated by template h2. According to this method, the image is traversed in sequence to obtain a gradient map to achieve edge extraction.
5. The method according to claim 1, characterized in that In step S52, the distance matrix between the two curves is obtained according to the hip joint bone gap contour curve and the algorithm for calculating the distance between the plane curves to obtain the shortest distance value, including: S521, discretize the curve A, calculate the minimum distance from each discrete point to the curve B, and select a number of point pairs with large distances and satisfying the "small size" of the distances between adjacent points on the curve A and the curve B as approximate solutions; S522, according to the characteristics of the curves at each point pair, determine which of the four types of points may exist near each point pair, establish a corresponding optimization model and perform local optimization, and select the maximum distance value in the optimization result as the one-way Hausdorff distance between the two plane curves; S523: Convert the calculation of the one-way Hausdorff distance between the two plane curves into the calculation of the minimum distance from a point to a curve.
Citation Information
Patent Citations
MRI hip joint inflammation segmentation and classification automatic quantitative grading sequential method
CN111784704A
Automatic grading method for ankylosing spondylitis based on multi-task deep learning
CN113763340A
Medical image segmentation method based on u-shaped network
WO2022199143A1
Cited By
Hip joint anatomical mark point identification method and system
CN121725185A
Hip joint anatomical landmark point recognition method and system
CN121725185B
