A multi-stage integrated osteophyte detection method from coarse to fine based on Dr image
By adopting a multi-stage osteophyte integrated detection method in medical images, combined with deep learning and HRNet, the accuracy and convergence performance problems of traditional methods when detecting knee osteophyte points are solved, and high-precision bone on Dr images are achieved Repeat detection.
Patent Information
- Application Number
- CN202210142542.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-16
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-02-16
Smart Images

Figure CN114494235B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of medical image processing, and in particular to a method, device and detection system for realizing multi-stage osteophyte integration detection from coarse to fine based on Dr images. Background Art
[0002] Deep learning has achieved great success in the field of natural images. New models and architectures are proposed every year, and the accuracy of image classification, detection and segmentation is constantly being improved. The success in the field of natural images has also caused researchers to think about whether the research methods of deep learning in the field of natural images can be introduced into medical images to assist doctors in analyzing and diagnosing diseases. However, compared with natural images, medical images have a huge problem: small amount of data, high annotation cost, and complex scenes. This also poses a huge challenge to the application of deep learning in the field of medical images, and it is necessary to use more refined analysis methods than natural images to process medical images.
[0003] Based on the deep learning algorithm, the femur and tibia of the knee joint X-ray image data are segmented, and osteophyte identification, measurement and other analyses are completed at the same time. Then, based on these key reference value analyses, the equation function is obtained, and the AI model is used to finally obtain the soft tissue release value caused by the removal of osteophytes and the possible increase in the osteotomy gap. Then, a reference for adjusting the amount of osteotomy of the femur and tibia of the knee joint is given before surgery, providing doctors with preoperative soft tissue assessment and osteotomy planning. The segmentation of the knee joint and the identification of osteophytes are of great reference significance to doctors in knee surgery planning.
[0004] In medical osteophyte detection technology, osteophytes are located near the junction of the femur and tibia at the knee joint and their characteristics are not very obvious.
[0005] Therefore, based on the traditional image processing corner point extraction algorithm, the accuracy of the extracted osteophyte points is poor; at the same time, the osteophyte point targets are small, and based on the single-stage target detection algorithm, the osteophyte points will be submerged in a large amount of background noise. After the model is trained to a certain extent, the loss function will no longer decrease, the weights will no longer be optimized and updated, and the detection performance is poor; the osteophyte points of the femur and tibia are close to each other and have similar features. The features of the second and third osteophyte points are more obvious than those of the first and fourth osteophyte points. Based on the single-task deep learning detection method, all osteophyte points are detected as single targets. The model converges to the second and third osteophyte points and no longer converges further. The first and fourth osteophyte points cannot be detected and the detection fails. Summary of the invention
[0006] In view of this, the present disclosure proposes a multi-stage osteophyte integrated detection method, device and detection system from coarse to fine based on Dr images to solve the defects of the above-mentioned background technology.
[0007] According to one aspect of the present disclosure, a multi-stage osteophyte integrated detection method from coarse to fine based on Dr images is provided, characterized in that it comprises the following steps:
[0008] S100, obtaining a Dr image of osteophytes;
[0009] S200, segmenting the femur, tibia, and the connection area between the femur and tibia according to the Dr image, and extracting the corresponding foreground masks respectively;
[0010] S300, based on the foreground mask, setting a osteophyte region detection target, and using a preset deep learning model to perform multi-stage detection on the osteophyte region detection target to obtain an osteophyte prediction region; and constructing a heat map of osteophyte points in the osteophyte prediction region, and using a preset deep learning model to detect the heat map to obtain an osteophyte point prediction foreground region;
[0011] S400, presetting extraction conditions, extracting osteophyte points from the osteophyte point prediction foreground area.
[0012] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, including:
[0013] S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively;
[0014] S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia;
[0015] S303, presetting a first pixel value, taking the center point coordinates as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection.
[0016] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0017] S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point;
[0018] S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and using the minimum enclosing rectangular frame as the osteophyte region to be detected;
[0019] S330. Based on the interlaced sampling anchorbox method, the aspect ratio and area of the anchorbox are overlapped and set on different pixel values, and based on the preset deep learning model, hierarchical detection is performed on osteophyte areas of different scales; wherein the number of anchorboxes is reduced to 1 / 16 of the previous sampling amount.
[0020] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0021] S340, constructing respective Gaussian distribution heat maps with each osteophyte point of the mask coordinates as the center;
[0022] S350, respectively obtaining the centroid of the Gaussian distribution heat map and using it as the osteophyte point to be detected of the respective coordinates;
[0023] S360, constructing several sub-detection models, performing heat map detection on each corresponding osteophyte point to be detected, and obtaining the foreground area of the heat map.
[0024] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0025] S311, using a loss function based on weighted balance, for two types of inputs of the same image, using L2 and style loss functions at the prediction end and KL divergence loss function of the two types of images to guide model training;
[0026] S321, combined with HRNet as the backbone of the deep neural network and the doctor's multi-view disease diagnosis solution, based on the center of the ROI area, cropped images of different pixel values;
[0027] S331, fusing the cropped images of different pixel values into the deep learning model to guide the model to learn the characteristics of osteophytes.
[0028] In a possible implementation, optionally, in step S350, when obtaining the osteophyte point to be detected, it also includes:
[0029] S341, acquiring an osteophyte point region within a preset range of the osteophyte point to be detected;
[0030] S342, sampling image erasing mode, selecting a osteophyte area and erasing it from the Dr image;
[0031] S343, training a model on the osteophyte area after the image is erased.
[0032] In a possible implementation, optionally, in step S400, the preset extraction condition, extracting osteophyte points from the foreground area of the heat map, includes:
[0033] S401, constructing an integrated model with the same structure as the sub-detection model;
[0034] S402, incorporating the plurality of sub-detection models into the integrated model, fusing the feature graphs of the plurality of sub-detection models into the integrated model and outputting the feature graphs, thereby guiding the training of the integrated model;
[0035] S403, keeping the gradients of the plurality of sub-detection models from being returned and the weights from being updated, and performing iterative training; when the weights of the plurality of sub-detection models reach a preset weight threshold, gradually reducing the fusion weights of the feature maps output by the sub-detection models according to a reduction strategy, and stopping iteration when the integrated model reaches a preset optimal weight;
[0036] S404, based on the preset weight threshold, segment and extract a foreground area of the heat map; extract the heat map contour points of the osteophyte from the foreground area of the heat map, and take the centroid thereof as the osteophyte point.
[0037] In a possible implementation, optionally, the preset deep learning model is: an HRNet deep learning model; wherein the loss function adopts the same confidence loss function of box regression and classification as SSD.
[0038] According to another aspect of the present disclosure, a device for implementing the above-mentioned multi-stage integrated osteophyte detection method from coarse to fine based on Dr images is provided, comprising:
[0039] A femur and tibia segmentation module, for segmenting the femur and tibia according to the Dr image, extracting masks of the femur and tibia respectively, and outputting the femur and tibia as independent layers respectively;
[0040] A osteophyte region detection module, used to set a osteophyte region detection target based on the foreground mask, and perform multi-stage detection on the osteophyte region detection target using a preset deep learning model to obtain a osteophyte prediction region;
[0041] An osteophyte detection module is used to construct a heat map of osteophyte points in the osteophyte prediction area, and use a preset deep learning model to detect the heat map to obtain a foreground area for osteophyte point prediction;
[0042] The osteophyte extraction module is used to preset extraction conditions and extract osteophyte points from the foreground area of the heat map.
[0043] According to another aspect of the present disclosure, a detection system is also provided, comprising:
[0044] processor;
[0045] a memory for storing processor-executable instructions;
[0046] Wherein, the processor is configured to implement the above-mentioned multi-stage osteophyte integration detection method from coarse to fine based on Dr image when executing the executable instructions.
[0047] Technical effects of this application:
[0048] The present invention obtains a DR image of osteophytes; according to the DR image, the femur, tibia and the connection area between the femur and tibia are segmented, and the corresponding foreground masks are extracted respectively; based on the foreground mask, the osteophyte region detection target is set, and the osteophyte region detection target is multi-stage detected by a preset deep learning model to obtain the osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and the heat map is detected by a preset deep learning model to obtain the osteophyte point prediction foreground region; the extraction conditions are preset, and the osteophyte points are extracted from the osteophyte point prediction foreground region. The foreground area of the heat map can be extracted based on the threshold segmentation, and the thermal map contour points of the osteophyte can be further extracted, so as to realize the osteophyte detection integrated from coarse to fine multi-stage multi-task on the DR image.
[0049] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
[0051] Figure 1 FIG. 1 is a schematic diagram showing an implementation process of a multi-stage osteophyte integrated detection method from coarse to fine based on Dr images of the present invention;
[0052] Figure 2 Shown is a flow chart of the algorithm for segmenting the femur and tibia of the present invention;
[0053] Figure 3 Shown is a schematic diagram of the prediction results of the osteophyte prediction area (white frame area in the figure) of the present invention;
[0054] Figure 4 Shown is a schematic diagram of the detection results of osteophyte points (white points in the figure) of the present invention;
[0055] Figure 5 It is a schematic diagram showing the structure of the device composition of Example 2 of the present invention. DETAILED DESCRIPTION
[0056] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0057] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0058] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.
[0059] Example 1
[0060] like Figure 1 As shown, according to one aspect of the present disclosure, a multi-stage osteophyte integrated detection method from coarse to fine based on Dr image is provided, characterized in that it includes the following steps:
[0061] S100, obtaining a Dr image of osteophytes;
[0062] The Dr image can be obtained by using a medical image Dr detection system, and the target bone area is scanned and detected.
[0063] S200, segmenting the femur, tibia, and the connection area between the femur and tibia according to the Dr image, and extracting the corresponding foreground masks respectively;
[0064] like Figure 2 As shown, a multi-stage multi-task encoder and decoder structure is used as a segmentation model to segment the femur and tibia to obtain the foreground masks of the femur and tibia; and then the ROI region (region of interest) of the osteophyte is extracted.
[0065] First, the first stage separates the femur and tibia from the background as a whole; second, the second stage treats the femur and tibia as two independent tasks and outputs them as independent feature layers. Based on this coarse-to-fine segmentation method, gradual tuning can well solve the problem that the traditional single-stage deep learning segmentation method cannot solve the difficult segmentation of the connection between the femur and tibia; at the same time, considering that deep learning distinguishes foreground and background mainly based on the change of grayscale gradient of the image, the edge is a key factor in distinguishing foreground and background. Especially when the edge of the foreground and background is not obvious, solving the edge of the foreground can more accurately extract the foreground area.
[0066] Therefore, this application adopts a multi-task segmentation method to simultaneously perform edge segmentation and region segmentation in one model, that is, using the Dice region loss function and the FocalLoss edge loss function.
[0067] When the autoencoder downsamples the feature layer of the image, it can provide more position information at the high-resolution layer and more semantic information at the low-resolution feature layer. In order to make full use of the position information and semantic information to make decisions at the final feature layer, these feature layers are fused at the decoder end and output to the final feature layer for predicting the foreground mask.
[0068] When the deep learning model is deep, there is a risk of gradient vanishing and gradient exploding due to chain derivation when the gradient is back-propagated. That is, at the end of the training, the model's loss function no longer decreases or overflows, and the model is no longer optimized. Therefore, in the model training stage, the loss function is calculated at different scales of the decoder to impose strong supervision constraints on the model and guide the model to converge faster. The grayscale changes at the junction of the femur and tibia are relatively gentle, and some even overlap, which is a difficult area for image segmentation. When training the model, a multi-scale training strategy is adopted. While performing segmentation training on the global femur and tibia, the junction area of the femur and tibia is also cropped out for training to increase attention to this difficult area.
[0069] During training, the area where the femur and tibia are connected will be cropped out for segmentation training and auxiliary segmentation. During the prediction stage, the area at the connection will not be segmented.
[0070] This technology considers that the junction of the femur and tibia is a difficult segmentation area, and cuts out this area for further multi-scale training to increase the model's attention to this part, thereby improving the model's recognition ability for this part. When designing the model, the boundary loss function is added to the model to guide the model to pay attention to the boundary between the femur and tibia, thereby improving the segmentation performance of the model.
[0071] S300, based on the foreground mask, setting a osteophyte region detection target, and using a preset deep learning model to perform multi-stage detection on the osteophyte region detection target to obtain an osteophyte prediction region; and constructing a heat map of osteophyte points in the osteophyte prediction region, and using a preset deep learning model to detect the heat map to obtain an osteophyte point prediction foreground region;
[0072] Osteophyte points are small, and if they are detected directly on the whole image, the osteophyte points will be submerged in the background noise, making it difficult for the model to converge and the osteophyte point detection to fail; at the same time, the location of the osteophyte points has obvious regional characteristics, that is, the osteophyte points are distributed near the connection between the femur and tibia, so the foreground mask obtained based on the femur and tibia segmentation module is established, and the osteophyte area detection target is established. The preset deep learning model is used to perform multi-stage detection on the osteophyte area detection target to obtain the osteophyte prediction area, which reduces the amount of calculation while ensuring the accuracy of the prediction to the greatest extent.
[0073] The osteophyte prediction area narrows the detection range, in order to further improve the true position of the osteophyte. In this embodiment, a method based on a heat map is used to detect osteophyte points, and a heat map of a Gaussian distributed osteophyte point is constructed with the osteophyte point as the center. A heat map of the osteophyte point is constructed in the osteophyte prediction area, and the heat map is detected using a preset deep learning model to obtain the osteophyte point prediction foreground area, and the predicted heat map is used to find the center of mass of the heat map to realize the prediction of the osteophyte point.
[0074] S400, preset extraction conditions, and extract osteophyte points from the osteophyte point prediction foreground area.
[0075] The extraction model is used to extract the osteophyte points that the model focuses on from the osteophyte point prediction foreground area. According to the heat maps corresponding to different numbers of osteophyte points, the extraction models of different stages can be used to extract the osteophyte points.
[0076] The above technology will be specifically implemented below.
[0077] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, including:
[0078] S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively;
[0079] S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia;
[0080] S303, presetting a first pixel value, taking the center point coordinates as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection.
[0081] In the stage of osteophyte region detection, based on the foreground mask obtained by the femur and tibia segmentation module, the lowest edge of the femur is taken as the initial y1 point, and the x coordinate of the centroid of the femur is taken as the initial x1 point, and (x1, y1) is obtained. Similarly, for the tibia, the upper edge of the tibia mask is selected as the initial y2 point, and the x coordinate of the centroid of the tibia mask is taken as the initial x2 point, and (x2, y2) is obtained. With the center position of these two points as the center, a rectangular area with a width and height of 512 pixels, i.e., the first pixel value, is selected as the ROI area.
[0082] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0083] S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point;
[0084] S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and using the minimum enclosing rectangular frame as the osteophyte region to be detected;
[0085] S330. Based on the interlaced sampling anchorbox method, the aspect ratio and area of the anchorbox are overlapped and set on different pixel values, and based on the preset deep learning model, hierarchical detection is performed on osteophyte areas of different scales; wherein the number of anchorboxes is reduced to 1 / 16 of the previous sampling amount.
[0086] Based on this ROI area, the minimum enclosing rectangle of the osteophyte point is further detected. The existing image data is analyzed and it is found that the maximum width and height of the minimum enclosing rectangle of the osteophyte is 218 pixels, so taking the width and height of 512, i.e. the second pixel value, as the ROI area will not cause the loss of osteophyte information. Then, the minimum enclosing rectangle of the osteophyte is detected based on the anchorbox method. The minimum enclosing rectangle of the osteophyte is analyzed and it is found that 80% of the rectangular boxes have a width-to-height ratio between 0.25-0.75, and the ratio of the area to the ROI is between 0.01-0.1.
[0087] like Figure 3 As shown in the figure, the aspect ratios of (1 / 4, 1 / 3, 1 / 2, 2 / 3, 3 / 4) and the anchorboxes with areas of (0.01, 0.02, 0.03, 0.04), (0.05, 0.06, 0.07, 0.08), and (0.09, 0.1) are set at three resolutions of 512 / 256 / 128 respectively. The area ratios of the anchorboxes set between different resolutions are overlapped, and hierarchical detection is performed for osteophyte areas of different scales to maintain a balance between the number and accuracy of the anchorboxes.
[0088] After analysis, if an anchorbox is set for each pixel under the above resolution, 6717440 anchorboxes will be generated, which requires a large amount of calculation. However, if the anchorbox is set at a lower resolution, information loss will occur, and it will be difficult to detect the minimum bounding rectangle of a small osteophyte.
[0089] In order to solve the contradiction between resolution and computational complexity, this technology proposes a method of interlaced sampling anchorbbox, which reduces the number of anchorbboxes to 1 / 16 of the previous number, that is, 419840, reducing the computational complexity while maximizing the accuracy of prediction.
[0090] Here, the HRNet deep learning model is selected for osteophyte area detection to maintain the image resolution to the greatest extent and reduce information loss. The loss function uses the same confidence loss function of bbox regression and classification as SSD.
[0091] By detecting osteophyte areas, e.g. Figure 3 As shown, the detection range can be narrowed down to a osteophyte area.
[0092] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0093] S340, constructing respective Gaussian distribution heat maps with each osteophyte point of the mask coordinates as the center;
[0094] S350, respectively obtaining the centroid of the Gaussian distribution heat map and using it as the osteophyte point to be detected of the respective coordinates;
[0095] S360, constructing several sub-detection models, performing heat map detection on each corresponding osteophyte point to be detected, and obtaining the foreground area of the heat map.
[0096] Osteophyte points are small, and the background noise is large based on osteophyte point detection, which makes it difficult for the model to converge, and the model cannot predict the true location of the osteophyte; at the same time, the contribution of the local area of the osteophyte point varies with the distance from the osteophyte point, and the closer to the osteophyte point, the higher the importance. Therefore, it is necessary to further detect osteophyte points based on the osteophyte prediction area.
[0097] In this embodiment, four osteophyte points in the osteophyte prediction area are selected as detection targets. The osteophyte points are detected using a method based on a thermal map, and a thermal map of the osteophyte points with a Gaussian distribution is constructed with the osteophyte points as the center. The predicted thermal map is used to find the center of mass of the thermal map to predict the osteophyte points. Inputting thermal images of multiple fields of view into the model can enrich the contextual information of the model and guide the model to detect osteophyte points.
[0098] Since the four osteophyte points are close to each other, especially the second and third osteophyte points; at the same time, the features of the first osteophyte and the fourth point are less obvious, and they are more difficult to detect. If predicted on a model, whether in a heat map or divided into 4 channels for prediction output, the model will gradually converge to the second and third osteophyte points. In some images, the heat maps of the second and third osteophyte points will merge into one heat map, making it difficult to separate the osteophyte points.
[0099] In this technology, the detection of four osteophyte points is decomposed into four identical tasks and detected using four models. At the same time, considering the interference of adjacent osteophyte points on the current task, the adjacent osteophyte points are randomly erased from the original image to enhance the model's anti-interference ability to other osteophyte points.
[0100] To further improve the anti-interference ability of the model, based on a conclusion: when the recognition ability of the model is strong, the probability distribution of the masked and unmasked images of the surrounding neighborhood points on the predicted feature map is consistent. Therefore, in the design of the loss function, for the two types of inputs of the same image, the L2 and style loss functions and the KL divergence loss function of the two types of images are used at the prediction end to guide model training; in model design, this technology chooses HRNet as the backbone of the deep neural network; at the same time, referring to the idea of multi-view disease diagnosis by doctors, images of 512 / 256 / 128 sizes are cropped based on the center of the ROI and fused into the model to guide the model to learn the characteristics of osteophytes.
[0101] In a possible implementation, optionally, in step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising:
[0102] S311, using a loss function based on weighted balance, for two types of inputs of the same image, using L2 and style loss functions at the prediction end and KL divergence loss function of the two types of images to guide model training;
[0103] S321, combined with HRNet as the backbone of the deep neural network and the doctor's multi-view disease diagnosis solution, based on the center of the ROI area, cropped images of different pixel values;
[0104] S331, fusing the cropped images of different pixel values into the deep learning model to guide the model to learn the characteristics of osteophytes.
[0105] Since the positive samples of the points in the heat map area of the osteophyte are fewer than the background points, and in order to enhance the model's attention to positive samples and difficult negative samples, a loss function based on weighted balance is adopted in this patent, that is, based on the number of positive samples, negative sample points with higher prediction scores are selected in proportion, and weights are applied to the samples according to the gold label value of the pixel point (exponential weighting is selected in this patent) to increase the weights of positive samples and difficult negative samples.
[0106] In a possible implementation, optionally, in step S350, when obtaining the osteophyte point to be detected, it also includes:
[0107] S341, acquiring an osteophyte point region within a preset range of the osteophyte point to be detected;
[0108] S342, sampling image erasing mode, selecting a osteophyte area and erasing it from the Dr image;
[0109] S343, training a model on the osteophyte area after the image is erased.
[0110] If a single osteophyte detection model is trained, in order to reduce the interference of surrounding osteophyte points, this patent uses a data augmentation method to erase the original image in a targeted manner, which enhances the model's anti-interference ability. For single osteophyte point detection, when the model has a strong recognition ability, whether to erase other osteophyte points in the neighborhood does not affect the prediction results. The contrast learning method can be used to calculate the KL divergence, L2 loss, and perceptual loss function of the feature map with other osteophyte points randomly erased and other osteophyte points not erased, to guide the training of the model and further enhance the robustness of the model.
[0111] In a possible implementation, optionally, in step S400, the preset extraction condition, extracting osteophyte points from the foreground area of the heat map, includes:
[0112] S401, constructing an integrated model with the same structure as the sub-detection model;
[0113] S402, incorporating the plurality of sub-detection models into the integrated model, fusing the feature graphs of the plurality of sub-detection models into the integrated model and outputting the feature graphs, thereby guiding the training of the integrated model;
[0114] S403, keeping the gradients of the plurality of sub-detection models from being returned and the weights from being updated, and performing iterative training; when the weights of the plurality of sub-detection models reach a preset weight threshold, gradually reducing the fusion weights of the feature maps output by the sub-detection models according to a reduction strategy, and stopping iteration when the integrated model reaches a preset optimal weight;
[0115] S404, based on the preset weight threshold, segment and extract a foreground area of the heat map; extract the heat map contour points of the osteophyte from the foreground area of the heat map, and take the centroid thereof as the osteophyte point.
[0116] In the stage of detecting the four osteophyte points, this technology decomposes the four osteophyte points into four models as four tasks for detection, and finally fuses the four osteophyte point detection models to obtain an integrated model as the final osteophyte point monitoring model.
[0117] The detection of four osteophyte points is decomposed into four identical tasks and detected by four models. At the same time, considering the interference of adjacent osteophyte points on the current task, the adjacent osteophyte points are randomly erased on the original image to enhance the model's anti-interference ability to other osteophyte points.
[0118] Since the positive samples of the points in the thermal map area of osteophytes are fewer than the background points, and in order to enhance the model's attention to positive samples and difficult negative samples, this patent adopts a loss function based on weighted balance, that is, based on the number of positive samples, negative sample points with higher prediction scores are selected in proportion, and weights are applied to the samples according to the gold label value of the pixel points (exponential weighting is selected in this patent). Increase the weights of positive samples and difficult negative samples.
[0119] In the osteophyte point extraction stage, four models are used to detect the heat map of osteophyte points, and the sub-model focuses on the current osteophyte point, thereby reducing the interference of other background information and osteophyte points.
[0120] In order to further explore the relationship between feature points and compress the model at the same time, and reduce the hardware computing cost of the production process, this patent integrates the four models into one model, establishes a model with the same structure as the aforementioned four sub-detection models as an integrated model, incorporates the aforementioned four osteophyte detection models into the integrated model, fuses the feature maps of the four branches into the integrated model output, guides the training of the integrated model, and the gradients of the four sub-models are not returned, and the weights are not updated. As the number of iterations increases, the fusion weights of the feature maps output by the four sub-models are gradually reduced, so that the integrated model will shift its attention to the integrated model after reaching a better weight. Figure 4 As shown, the integrated model is used to predict the contour points of the heat map, and the centroid is extracted as the osteophyte point.
[0121] In this technology, the detection models of four osteophyte points are integrated into one model. As the training progresses, the weights of the feature maps of the four osteophyte point detection models are gradually reduced, the attention of the integrated model is enhanced, and the integrated model is gradually guided to converge. This not only reduces the hardware requirements of the production process, but also further utilizes the relationship between osteophyte points to improve the performance of osteophyte point detection.
[0122] In a possible implementation, optionally, the preset deep learning model is: an HRNet deep learning model; wherein the loss function adopts the same confidence loss function of box regression and classification as SSD.
[0123] It should be noted that although the above deep learning prediction method is introduced using the HRNet deep learning model as an example, those skilled in the art will understand that the present disclosure should not be limited to this. In fact, users can flexibly set the prediction learning model according to personal preferences and / or actual application scenarios, as long as it can meet the deep learning and training prediction of this technology.
[0124] Example 2
[0125] Based on the implementation of Example 1, this implementation, as Figure 5As shown, a device for realizing the above-mentioned multi-stage integrated osteophyte detection method from coarse to fine based on Dr image is provided, including:
[0126] A femur and tibia segmentation module is used to segment the femur and tibia according to the Dr image, extract masks of the femur and tibia respectively, and output the femur and tibia as independent layers respectively; and segment the connection area of the femur and tibia at the same time;
[0127] A osteophyte region detection module, used to set a osteophyte region detection target based on the foreground mask, and perform multi-stage detection on the osteophyte region detection target using a preset deep learning model to obtain a osteophyte prediction region;
[0128] An osteophyte detection module is used to construct a heat map of osteophyte points in the osteophyte prediction area, and use a preset deep learning model to detect the heat map to obtain a foreground area for osteophyte point prediction;
[0129] The osteophyte extraction module is used to preset extraction conditions and extract osteophyte points from the foreground area of the heat map.
[0130] First, the Dr image of the skeleton is obtained through the Dr image acquisition module, i.e. the medical image Dr detection system, and then detection and extraction are performed according to the above modules.
[0131] The functions and implementation principles of the above-mentioned modules / hardwares are specifically described in the above-mentioned embodiments and will not be elaborated here.
[0132] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0133] Example 3
[0134] Furthermore, according to another aspect of the present disclosure, a detection system is provided, comprising:
[0135] processor;
[0136] a memory for storing processor-executable instructions;
[0137] Wherein, the processor is configured to implement the above-mentioned multi-stage osteophyte integration detection method from coarse to fine based on Dr image when executing the executable instructions.
[0138] The control system of the disclosed embodiment includes a processor and a memory for storing processor executable instructions. The processor is configured to implement any of the above-mentioned multi-stage osteophyte integration detection methods from coarse to fine based on Dr images when executing the executable instructions.
[0139] Here, it should be noted that the number of processors can be one or more. At the same time, in the control system of the embodiment of the present disclosure, an input device and an output device may also be included. Among them, the processor, memory, input device and output device may be connected through a bus or in other ways, which are not specifically limited here.
[0140] The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs and various modules, such as: a program or module corresponding to a method for detecting multi-stage osteophyte integration from coarse to fine based on Dr images in an embodiment of the present disclosure. The processor executes various functional applications and data processing of the control system by running the software programs or modules stored in the memory.
[0141] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.
[0142] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A multi-stage integrated osteophyte detection method based on Dr images from coarse to fine, characterized in that: The steps include: S100, obtaining a Dr image of osteophytes; S200, segmenting the femur, tibia, and the connection area between the femur and tibia according to the Dr image, and extracting the corresponding foreground masks respectively; S300, based on the foreground mask, setting a osteophyte region detection target, and using a preset deep learning model to perform multi-stage detection on the osteophyte region detection target to obtain an osteophyte prediction region; and constructing a heat map of osteophyte points in the osteophyte prediction region, and using a preset deep learning model to detect the heat map to obtain an osteophyte point prediction foreground region; In step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising: S340, constructing respective Gaussian distribution heat maps with each osteophyte point of the mask coordinate as the center; S350, respectively obtaining the centroid of the Gaussian distribution heat map and using it as the osteophyte point to be detected of the respective coordinates; S360, constructing several sub-detection models, performing heat map detection on each corresponding osteophyte point to be detected, and obtaining a foreground area of the heat map; S400, presetting extraction conditions, extracting osteophyte points from the osteophyte point prediction foreground area; In step S400, the preset extraction condition is used to extract osteophyte points from the foreground area of the heat map, including: S401, constructing an integrated model with the same structure as the sub-detection model; S402, incorporating the plurality of sub-detection models into the integrated model, fusing the feature graphs of the plurality of sub-detection models into the integrated model and outputting the feature graphs, thereby guiding the training of the integrated model; S403, keeping the gradients of the plurality of sub-detection models from being returned and the weights from being updated, and performing iterative training; when the weights of the plurality of sub-detection models reach a preset weight threshold, gradually reducing the fusion weights of the feature maps output by the sub-detection models according to a reduction strategy, and stopping iteration when the integrated model reaches a preset optimal weight; S404, based on the preset weight threshold, segment and extract the foreground area of the heat map; extract the heat map contour points of the osteophyte from the foreground area of the heat map, and take the centroid thereof as the osteophyte point.
2. The multi-stage osteophyte integrated detection method based on Dr image from coarse to fine according to claim 1, characterized in that: In step S300, the osteophyte region detection target is set based on the foreground mask, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, including: S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively; S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia; S303, presetting a first pixel value, taking the center point coordinates as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection.
3. The multi-stage osteophyte integrated detection method based on Dr image from coarse to fine according to claim 2, characterized in that: In step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising: S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point; S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and using the minimum enclosing rectangular frame as the osteophyte region to be detected; S330. Based on the interlaced sampling anchorbox method, the aspect ratio and area of the anchorbox are overlapped and set on different pixel values, and based on the preset deep learning model, hierarchical detection is performed on osteophyte areas of different scales; wherein the number of anchorboxes is reduced to 1 / 16 of the previous sampling amount.
4. The multi-stage osteophyte integration detection method based on Dr image from coarse to fine according to claim 1, characterized in that: In step S300, based on the foreground mask, a osteophyte region detection target is set, and a preset deep learning model is used to perform multi-stage detection on the osteophyte region detection target to obtain a osteophyte prediction region; and a heat map of osteophyte points is constructed in the osteophyte prediction region, and a preset deep learning model is used to detect the heat map to obtain a osteophyte point prediction foreground region, further comprising: S311, using a loss function based on weighted balance, for two types of inputs of the same image, using L2 and style loss functions at the prediction end and KL divergence loss function of the two types of images to guide model training; S321, combined with HRNet as the backbone of the deep neural network and the doctor's multi-view disease diagnosis solution, based on the center of the ROI area, cropped images of different pixel values; S331, fusing the cropped images of different pixel values into the deep learning model to guide the model to learn the characteristics of osteophytes.
5. The multi-stage osteophyte integration detection method based on Dr image from coarse to fine according to claim 3, characterized in that: In step S350, when the osteophyte point to be detected is obtained, it also includes: S341, acquiring an osteophyte point region within a preset range of the osteophyte point to be detected; S342, sampling image erasing mode, selecting a osteophyte area and erasing it from the Dr image; S343, training a model on the osteophyte area after the image is erased.
6. The multi-stage osteophyte integrated detection method based on Dr image from coarse to fine according to claim 1, characterized in that: The preset deep learning model is: HRNet deep learning model; wherein the loss function adopts the same box regression and classification confidence loss function as SSD.
7. A device for realizing the multi-stage osteophyte integrated detection method from coarse to fine based on Dr image according to any one of claims 1 to 6, characterized in that: include: A femur and tibia segmentation module, for segmenting the femur and tibia according to the Dr image, extracting masks of the femur and tibia respectively, and outputting the femur and tibia as independent layers respectively; A osteophyte region detection module, used to set a osteophyte region detection target based on the foreground mask, and perform multi-stage detection on the osteophyte region detection target using a preset deep learning model to obtain a osteophyte prediction region; An osteophyte detection module is used to construct a heat map of osteophyte points in the osteophyte prediction area, and use a preset deep learning model to detect the heat map to obtain a foreground area for osteophyte point prediction; The method further comprises: setting a detection target of a osteophyte region based on the foreground mask, and performing multi-stage detection on the detection target of the osteophyte region using a preset deep learning model to obtain a predicted osteophyte region; and constructing a heat map of osteophyte points in the predicted osteophyte region, and detecting the heat map using a preset deep learning model to obtain a predicted osteophyte point foreground region. Construct respective Gaussian distribution heat maps with each osteophyte point of the mask coordinate as the center; Obtaining the centroids of the Gaussian distribution heat maps respectively, and using them as the osteophyte points to be detected at their respective coordinates; Construct several sub-detection models, perform heat map detection on each corresponding osteophyte point to be detected, and obtain the foreground area of the heat map; A osteophyte extraction module, used to preset extraction conditions and extract osteophyte points from the foreground area of the heat map; The preset extraction condition, extracting osteophyte points from the foreground area of the heat map, includes: Construct an integrated model with the same structure as the sub-detection model; Incorporating the plurality of sub-detection models into the integrated model, fusing the feature maps of the plurality of sub-detection models into the integrated model and outputting the feature maps, thereby guiding the training of the integrated model; Keep the gradients of the several sub-detection models not back-propagated and the weights not updated, and perform iterative training; when the weights of the several sub-detection models reach a preset weight threshold, gradually reduce the fusion weights of the feature maps output by the sub-detection models according to the reduction strategy, and stop iteration when the integrated model reaches a preset optimal weight; Based on the preset weight threshold, the foreground area of the heat map is segmented and extracted; the heat map contour points of the osteophyte are extracted from the foreground area of the heat map, and the centroid thereof is taken as the osteophyte point.
8. A detection system, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the multi-stage osteophyte integration detection method from coarse to fine based on Dr image as described in any one of claims 1 to 6 when executing the executable instructions.
