Multi-stage and multi-level osteophyte detection method based on deep learning

By adopting a multi-stage multi-level osteophyte detection method in medical image processing, using deep learning models to segment and detect the femur and tibia of the knee joint, establishing a father-son rectangular frame, gradually reducing the weight of the loss function, solving the osteophyte detection error in traditional methods The major problem improves detection accuracy and training efficiency, and provides an important reference for preoperative planning.

CN114511548BActive Publication Date: 2024-11-05YUNNAN BAIYAO GRP MEDICINE E-COMMERCE CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210143210.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-16
Publication Date
2024-11-05
Estimated Expiration
2042-02-16

AI Technical Summary

Technical Problem

In medical image processing, especially in the field of osteophyte detection, traditional methods have large detection errors due to the small amount of data, high annotation cost, complex scenes, and insufficient recognition ability of single-stage deep learning models to small targets, resulting in large detection errors and easy convergence in model training. When it reaches the local extreme point, the performance is poor.

Method used

Using a multi-stage multi-level osteophyte detection method based on deep learning, the femur and tibia are segmented by obtaining knee joint X-ray data, setting osteophyte detection targets, using a preset deep learning model for multi-level detection, and establishing a father-son rectangular frame , gradually reduce the loss function weight of the parent rectangle box, increase the model's attention to osteophytes, and extract osteophytes.

Benefits of technology

It improves the accuracy of osteophyte detection and model training efficiency, and can provide doctors with accurate soft tissue evaluation and osteotomy planning references before surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114511548B_ABST
    Figure CN114511548B_ABST
Patent Text Reader

Abstract

The present application relates to a multi-stage and multi-level osteophyte detection method based on deep learning, which obtains a Dr image of osteophytes; extracts masks of the femur and tibia respectively according to the Dr image, and segment the femur and tibia; sets osteophyte detection targets based on the masks of the femur and tibia, and uses a preset deep learning model to perform multi-level detection on the osteophyte detection targets to obtain osteophyte areas; preset extraction conditions are used to extract osteophyte points from the osteophyte areas. The femur and tibia can be segmented, and the ROI area near the connection between the femur and tibia can be extracted, and the osteophyte detection can be focused in this area, while improving the detection accuracy of osteophyte points. In the multi-level osteophyte detection module, a parent-child rectangular frame is established, and the weight of the loss function of the parent rectangular frame is gradually reduced, so that the model's attention to the osteophyte point sub-target is gradually increased while improving the training efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of medical image processing, and in particular to a multi-stage and multi-level osteophyte detection method, device and detection system based on deep learning. Background Art

[0002] Deep learning has achieved great success in the field of natural images. New models and architectures are proposed every year, and the accuracy of image classification, detection and segmentation is constantly being improved. The success in the field of natural images has also caused researchers to think about whether the research methods of deep learning in the field of natural images can be introduced into medical imaging to assist doctors in analyzing and diagnosing diseases.

[0003] However, compared with natural images, medical images have a huge problem: small amount of data, high annotation cost, and complex scenes. This also poses a huge challenge to the application of deep learning in the field of medical images, and it is necessary to use more refined analysis methods than natural images to process medical images.

[0004] In addition, in medical osteophyte detection technology, because osteophytes are located near the connection between the femur and tibia of the knee joint, the target is small and the features are not very obvious relative to the pixels in the neighborhood; if it is based on traditional corner point detection methods, the osteophyte points will be submerged in a large amount of background noise, resulting in large detection errors.

[0005] If a single-stage deep learning detection method is used, since osteophytes account for a small proportion of the entire image, the feature point information of osteophytes will be completely lost after the model is downsampled several times, which increases the difficulty of model feature extraction. After the model is trained to a certain stage and converges to a local extreme point, it will no longer decrease and the performance of the model is poor. Summary of the invention

[0006] In view of this, the present invention proposes a multi-stage and multi-level osteophyte detection method, device and detection system based on deep learning, which segments the femur and tibia of the knee joint X-ray image data based on a deep learning algorithm, and simultaneously completes osteophyte identification, measurement and other analyses, and then obtains an equation function based on these key reference value analyses, and finally obtains the soft tissue loosening value and the possible increase in osteotomy gap caused by the removal of osteophytes by using an AI model, and then provides a reference for adjusting the osteotomy amount of the femur and tibia of the knee joint before surgery, providing doctors with preoperative soft tissue evaluation and osteotomy planning, and having important reference significance for the segmentation of the knee joint and the identification of osteophytes in knee surgery planning.

[0007] According to one aspect of the present disclosure, a multi-stage and multi-level osteophyte detection method based on deep learning is provided, characterized in that it comprises the following steps:

[0008] S100, obtaining a Dr image of osteophytes;

[0009] S200, extracting masks of the femur and tibia respectively according to the Dr image, and segmenting the femur and tibia;

[0010] S300, setting a osteophyte detection target based on the masks of the femur and tibia, and performing multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area;

[0011] S400, presetting extraction conditions, and extracting osteophyte points from the osteophyte area.

[0012] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, including:

[0013] S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively;

[0014] S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia;

[0015] S303, presetting a first pixel value, taking the center point coordinates as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection.

[0016] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0017] S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point;

[0018] S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and establishing a parent rectangular frame based on the minimum enclosing rectangular frame; and obtaining a sub-rectangular frame established with each osteophyte point as a central pixel point;

[0019] S330: Use the parent rectangular box and the child rectangular box as targets for multi-level detection.

[0020] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0021] S340, according to the rectangular area of ​​the osteophyte, calculating the aspect ratio and the area ratio of the rectangular area of ​​the osteophyte to the entire image detected, and obtaining a preset anchor box based on this;

[0022] S350, based on the interlaced sampling method, at each pixel point, a preset number of target detection areas are established;

[0023] S360: Based on the preset deep learning model, detect the obtained target detection areas respectively.

[0024] In a possible implementation, optionally, the preset deep learning model is: an HRNet deep learning model; wherein the loss function adopts the same bbox regression and classification confidence loss function as SSD.

[0025] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0026] S311, according to the ROI area, obtaining a feature map obtained by applying the mask of the femur and the tibia to the ROI;

[0027] S321, combining the masks of the femur and tibia, the ROI region and the feature map into a three-channel image, and fusing the three-channel images into the deep learning model as the upper and lower fields of the osteophyte region;

[0028] S331. Guide model training according to the three-channel image.

[0029] In a possible implementation, optionally, in step S340, when training the model extracted from the sub-rectangular frame, the following steps are further included:

[0030] S341, obtaining a sub-region image in the model extracted by the sub-rectangular frame;

[0031] S342, a sampling image erasing method, randomly erasing a certain osteophyte point sub-region in the sub-region image;

[0032] S343, training the model after the image is erased.

[0033] In a possible implementation, optionally, in step S340, when training the parent rectangular frame, the following steps are further included:

[0034] S344, in the anchorbox of the parent rectangular box, according to the reduction strategy, gradually reduce the weight training strategy of the loss function corresponding to the parent node;

[0035] S345, performing iterative training on the model extracted by the parent rectangular box according to the reduced weight training strategy;

[0036] S346, preset a weight threshold. When the weight of the weight training strategy reaches the weight threshold, stop the iteration, and no longer extract and calculate the anchorbox corresponding to the parent node.

[0037] According to another aspect of the present disclosure, a device for implementing the above-mentioned multi-stage and multi-level osteophyte detection method based on deep learning is provided, comprising:

[0038] The femur and tibia segmentation module is used to extract the masks of the femur and tibia respectively according to the Dr image, and segment the femur and tibia;

[0039] A multi-level osteophyte detection module, used to set an osteophyte detection target based on the mask of the femur and tibia, and perform multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area;

[0040] The osteophyte extraction module is used to preset extraction conditions and extract osteophyte points from the osteophyte area.

[0041] According to another aspect of the present disclosure, a detection system is also provided, comprising:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] Among them, the processor is configured to implement the above-mentioned multi-stage and multi-level osteophyte detection method based on deep learning when executing the executable instructions.

[0045] Technical effects of this application:

[0046] The present invention obtains a Dr image of osteophytes; extracts masks of the femur and tibia respectively according to the Dr image, and segments the femur and tibia; sets osteophyte detection targets based on the masks of the femur and tibia, and uses a preset deep learning model to perform multi-level detection on the osteophyte detection targets to obtain osteophyte regions; preset extraction conditions, and extracts osteophyte points from the osteophyte regions. The femur and tibia can be segmented, and the ROI region near the connection between the femur and the tibia can be extracted, and the osteophyte detection is focused in this region, while improving the detection accuracy of the osteophyte points. In the multi-level osteophyte detection module, a parent-child rectangular frame is established, and the weight of the loss function of the parent rectangular frame is gradually reduced. The training of the parent rectangular frame is stopped at a certain stage, and the model's attention to the osteophyte point sub-target is gradually increased while improving the training efficiency.

[0047] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0049] Figure 1 Shown is an implementation flow chart of the multi-stage and multi-level osteophyte detection method based on deep learning of the present invention;

[0050] Figure 2 Shown is a schematic diagram of the algorithm flow of the femur and tibia segmentation of the present invention;

[0051] Figure 3 FIG. 1 is a schematic diagram showing the detection results of the osteophyte detection module of the present invention;

[0052] Figure 4 Shown is a schematic diagram of osteophyte prediction results of the present invention (white dots are detected osteophyte points);

[0053] Figure 5 It is a schematic diagram showing the structure of the device system of the second embodiment of the present invention. DETAILED DESCRIPTION

[0054] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0055] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0056] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0057] Example 1

[0058] like Figure 1 As shown, according to one aspect of the present disclosure, a multi-stage and multi-level osteophyte detection method based on deep learning is provided, characterized in that it includes the following steps:

[0059] S100, obtaining a Dr image of osteophytes;

[0060] Dr images can be obtained through a medical DR image system. By scanning and detecting osteophytes through the image system, a Dr image of the target osteophyte can be obtained.

[0061] S200, extracting masks of the femur and tibia respectively according to the Dr image, and segmenting the femur and tibia;

[0062] like Figure 2 As shown, the femur and tibia segmentation modules are used to extract the masks of the femur and tibia respectively, and the image regions of different femurs and tibias are obtained. As a preferred method, a deep learning model with an encoder and a decoder as the basic structure can be used, and Dice is used as the loss function to extract the masks of the femur and tibia from the Dr image. In the following implementation, the mask image points of the segmented femur and tibia will be extracted by the parent-child rectangular frame trained by the deep learning model, and the model's attention to the osteophyte point sub-target will be gradually improved through a step-by-step contraction strategy, and finally accurate osteophyte points will be extracted.

[0063] S300, setting a osteophyte detection target based on the masks of the femur and tibia, and performing multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area;

[0064] After segmenting the femur and tibia imaging data, it is necessary to set the target area for osteophyte identification, measurement and other analyses, so as to use the AI ​​model to ultimately obtain the soft tissue release value caused by osteophyte removal and the possible increase in osteotomy gap, and then provide a reference for adjusting the femoral and tibia osteotomy amount of the knee joint before surgery.

[0065] First, based on the foreground mask obtained by the femur and tibia segmentation module, a rectangular area is cropped from the original image according to preset conditions (user-defined) as the ROI for osteophyte detection, i.e., the osteophyte detection target (region of interest). The mask obtained by extracting the femur and tibia and the feature map formed by the mask acting on the original image are used as prior knowledge to guide the model to focus on the area where osteophytes exist.

[0066] Secondly, the existing image data is analyzed to obtain the minimum bounding rectangle for detecting osteophyte points, and the required OI area is constructed based on this, that is, the area near the connection between the femur and the tibia, so that the information of the osteophyte can be retained without loss.

[0067] As attached Figure 3 As shown, the construction is completed, and the parent rectangular box of the minimum enclosing rectangular box of the selected osteophyte point and the sub-rectangular box established with each osteophyte point as the central pixel point are detected by the deep learning model to obtain the osteophyte area as the target of osteophyte detection.

[0068] By first detecting the osteophyte area and then extracting the osteophyte points from the osteophyte area, it is possible to achieve multi-stage and multi-level osteophyte detection from coarse to fine on the DR image.

[0069] S400, presetting extraction conditions, and extracting osteophyte points from the osteophyte area.

[0070] After the osteophyte area is detected, the osteophyte points can be extracted from the small area.

[0071] The conventional method is to directly extract the center point of the predicted sub-target rectangle, but this method does not take advantage of the model's attention to osteophyte points.

[0072] As attached Figure 4 The osteophyte prediction results shown in the present application adopt a balanced method, extracting the feature map at the position of the rectangular box corresponding to the osteophyte point on the feature map before the multi-level osteophyte detection module prediction rectangular box, and selecting the focus point of the model in the area (that is, the point with the largest value on the feature map) and the center of the rectangular box of the sub-target to calculate the average as the coordinates of the osteophyte point. At the same time, refer to the focus point of the model and the center point of the predicted rectangular box to improve the accuracy of osteophyte point extraction.

[0073] The above steps will be described in detail below.

[0074] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, including:

[0075] S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively;

[0076] S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia;

[0077] S303, presetting a first pixel value, taking the center point coordinates as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection.

[0078] The mask reference point is a reference coordinate that helps users select mask coordinates. This can be selected and set by the user.

[0079] In this embodiment, based on the foreground mask obtained by the femur and tibia segmentation module, the bottom edge of the femur mask is taken as the initial y1 point, and the x coordinate of the centroid of the femur mask is taken as the initial x1 point to obtain the point (x1, y1). Similarly, for the tibia, the top edge of the tibia mask is selected as the initial y2 point, and the x point of the centroid of the tibia mask is taken as the initial x2 point to obtain (x2, y2). With the center of these two points as the center, a rectangular area with a width and height of 512 pixels is cut from the original image as the ROI (region of interest) for osteophyte detection. The mask obtained by extracting the femur and tibia and the feature map formed by the mask acting on the original image are used as prior knowledge to guide the model to pay attention to the area where osteophytes exist.

[0080] Furthermore, the minimum enclosing rectangle for detecting osteophyte points: based on the analysis of existing image data, the maximum width and height of the minimum enclosing rectangle of osteophyte is 128 pixels. Therefore, the width and height of 512 pixels, that is, the first pixel value, is taken as the ROI area, that is, the area near the connection between the femur and the tibia, that is, the ROI area for osteophyte detection, which can retain the information of osteophyte without losing it.

[0081] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0082] S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point;

[0083] S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and establishing a parent rectangular frame based on the minimum enclosing rectangular frame; and obtaining a sub-rectangular frame established with each osteophyte point as a central pixel point;

[0084] S330: Use the parent rectangular box and the child rectangular box as targets for multi-level detection.

[0085] In this embodiment, a multi-stage and multi-level osteophyte detection scheme from coarse to fine is adopted, and the regional target of osteophyte detection is set as: the parent rectangular box of the minimum enclosing rectangular box of four osteophyte points, and the sub-rectangular box established with each osteophyte point as the central pixel point.

[0086] In this embodiment, the sub-target width and height are selected to be 20 pixels to construct a rectangular frame. Here, it is also feasible to select other smaller rectangular frames, as long as each minimum rectangular frame can isolate the osteophyte (the IOU between the minimum rectangular frames of the osteophyte is less than 0.5), but at the same time it is not too small, and the target information will be lost during the downsampling process, causing the osteophyte point detection to fail. It can be selected by the user without restriction.

[0087] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0088] S340, according to the rectangular area of ​​the osteophyte, calculating the aspect ratio and the area ratio of the rectangular area of ​​the osteophyte to the entire image detected, and obtaining a preset anchor box based on this;

[0089] S350, based on the interlaced sampling method, at each pixel point, a preset number of target detection areas are established;

[0090] S360: Based on the preset deep learning model, detect the obtained target detection areas respectively.

[0091] The parent-child rectangular box is constructed and will be used as a bone spur detection model later. Therefore, a deep learning model is needed to train the parent-child rectangular box.

[0092] In this embodiment, the anchorbox method is used to train the parent-child rectangular box extraction model.

[0093] The analysis of the parent rectangle of the minimum bounding rectangle of the osteophyte shows that the aspect ratio of 80% of the rectangles is between 0.25-0.75, and the area ratio of the area to the ROI is between 0.01-0.1. Therefore, the aspect ratio of the anchorbox (1 / 4, 1 / 3, 1 / 2, 2 / 3, 3 / 4) and the area ratio of (0.01, 0.02, 0.03, 0.04), (0.05, 0.06, 0.07, 0.08), (0.09, 0.1) are respectively taken on 512 / 256 / 128 to detect the parent rectangle. Considering that if an anchorbox is set at each pixel point on 512 / 256 / 128, 6717440 anchorboxes will be generated, which is a large amount of calculation. If the anchorbox is set at a lower resolution, information loss will occur, and the detection of the minimum bounding rectangle of the smaller osteophyte will fail.

[0094] In order to solve the contradiction between resolution and computational complexity, this application proposes a method for setting anchorboxes by interlaced sampling. The number of anchorboxes is reduced to 1 / 16 of the previous number, which is 419840, greatly reducing the computational complexity while ensuring the accuracy of prediction to the greatest extent. At the same time, since the target of the minimum enclosing rectangular box of the osteophyte point is small, the rectangular box of the sub-target, i.e., the sub-rectangular box, is only detected at a resolution of 512. Similar to the detection idea of ​​the parent rectangular box, the sub-target uses a preset 65536 anchorboxes with an aspect ratio of 1 and an area ratio of 0.0015 at 512 resolution. The labels of the four osteophyte sub-targets are set to 2 / 3 / 4 / 5 respectively, and the parent target label is 1. In the multi-level osteophyte detection module, a parent-child rectangular box is established, and the weight of the loss function of the parent rectangular box is gradually reduced. The training of the parent rectangular box is stopped at a certain stage after iteration, and the model's attention to the osteophyte sub-target is gradually increased while improving the training efficiency.

[0095] In a possible implementation, optionally, the preset deep learning model is: an HRNet deep learning model; wherein the loss function adopts the same bbox regression and classification confidence loss function as SSD.

[0096] The HRNet deep learning model is used to detect osteophyte areas to maintain the image resolution to the greatest extent and reduce information loss. The loss function uses the same confidence loss function of bbox regression and classification as SSD.

[0097] In a possible implementation, optionally, in step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and a preset deep learning model is used to perform multi-level detection on the osteophyte detection target to obtain the osteophyte area, and further includes:

[0098] S311, according to the ROI area, obtaining a feature map obtained by applying the mask of the femur and the tibia to the ROI;

[0099] S321, combining the masks of the femur and tibia, the ROI region and the feature map into a three-channel image, and fusing the three-channel images into the deep learning model as the upper and lower fields of the osteophyte region;

[0100] S331. Guide model training according to the three-channel image.

[0101] During the detection process, referring to the doctor's multi-field disease diagnosis ideas, 256 / 128 images were cut out from the original image as the upper and lower fields of the osteophyte area and fused into the model to enhance the model's recognition of osteophyte points; since the osteophyte points are located on the femur and tibia, the ROI near the connection between the femur and tibia. Therefore, this application combines the ROI area extracted from the femur and tibia, the corresponding mask, and the feature map obtained by the mask acting on the ROI into a three-channel image and inputs it into the model to guide model training.

[0102] In a possible implementation, optionally, in step S340, when training the model extracted from the sub-rectangular frame, the following steps are further included:

[0103] S341, obtaining a sub-region image in the model extracted by the sub-rectangular frame;

[0104] S342, a sampling image erasing method, randomly erasing a certain osteophyte point sub-region in the sub-region image;

[0105] S343, training the model after the image is erased.

[0106] Since the features of various osteophyte points are similar, during the training process of the sub-goal of osteophyte point detection, the interference of adjacent osteophyte points will make it difficult to reduce the loss function and the model optimization will be poor.

[0107] Therefore, in this application, an image erasing method is used to randomly erase a certain bone spur point area to train the model and increase the robustness of the model; the parent target is a relatively large target and is easier to train.

[0108] In a possible implementation, optionally, in step S340, when training the parent rectangular frame, the following steps are further included:

[0109] S344, in the anchorbox of the parent rectangular box, according to the reduction strategy, gradually reduce the weight training strategy of the loss function corresponding to the parent node;

[0110] S345, performing iterative training on the model extracted by the parent rectangular box according to the reduced weight training strategy;

[0111] S346, preset a weight threshold. When the weight of the weight training strategy reaches the weight threshold, stop the iteration, and no longer extract and calculate the anchorbox corresponding to the parent node.

[0112] The training of the parent target is also to guide the model to pay attention to the osteophyte sub-region, thereby improving the detection performance of a single osteophyte point. Therefore, a weight training strategy is adopted to gradually reduce the loss function corresponding to the parent node, and the training is performed to a certain number of iterations. In this embodiment, 300 iterations are selected to reduce the corresponding weight to 0. During the forward calculation, the anchorbox corresponding to the parent node is no longer extracted and calculated, which improves the training speed of the model, increases the attention to the sub-osteophyte point at a better weight of the model, and improves the performance of the model in detecting the sub-target of the osteophyte point. The prediction results of the final osteophyte area are shown in the attached figure. Figure 3 shown.

[0113] After the osteophyte area is detected, the feature map at the position of the rectangular box corresponding to the osteophyte point is extracted from the feature map before the rectangular box predicted by the multi-level osteophyte detection module, and the focus point of the model in the area (that is, the point with the largest value on the feature map) and the center of the rectangular box of the sub-target are selected to calculate the average as the coordinates of the osteophyte point. At the same time, the focus point of the model and the center point of the predicted rectangular box are referenced to extract the osteophyte point.

[0114] It should be noted that although the HRNet deep learning model is used as an example to introduce the above father-child training model, those skilled in the art will understand that the present disclosure should not be limited to this. In fact, users can flexibly set the training model according to personal preferences and / or actual application scenarios, as long as the father-child rectangular frame training in this technology can be implemented.

[0115] In this way, by establishing parent-child rectangular boxes, the weight of the loss function of the parent rectangular box is gradually reduced, and the training of the parent rectangular box is stopped at a certain stage, the model's attention to the osteophyte point sub-target is gradually increased while the training efficiency is improved; the mask obtained by extracting the femur and tibia and the feature map formed by the mask acting on the original image, as prior knowledge, guide the model to pay attention to the area where osteophytes exist, and realize osteophyte detection from coarse to fine multi-stage and multi-level relationships on DR images.

[0116] Example 2

[0117] Based on the implementation of Example 1, this implementation, as Figure 5 As shown, according to another aspect of the present disclosure, a device for implementing the above-mentioned multi-stage and multi-level osteophyte detection method based on deep learning is provided, comprising:

[0118] The femur and tibia segmentation module is used to extract the masks of the femur and tibia respectively according to the Dr image, and segment the femur and tibia;

[0119] A multi-level osteophyte detection module, used to set an osteophyte detection target based on the mask of the femur and tibia, and perform multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area;

[0120] The osteophyte extraction module is used to preset extraction conditions and extract osteophyte points from the osteophyte area.

[0121] First, you need to obtain the Dr image, which can be obtained through the medical imaging system.

[0122] Secondly, the functions and implementation principles of each module / hardware are specifically described in the above embodiments and will not be repeated here.

[0123] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0124] Example 3

[0125] Furthermore, according to another aspect of the present disclosure, a detection system is provided, comprising:

[0126] processor;

[0127] a memory for storing processor-executable instructions;

[0128] Among them, the processor is configured to implement the above-mentioned multi-stage and multi-level osteophyte detection method based on deep learning when executing the executable instructions.

[0129] The detection system of the embodiment of the present disclosure includes a processor and a memory for storing processor executable instructions. The processor is configured to implement any of the above-mentioned multi-stage and multi-level osteophyte detection methods based on deep learning when executing the executable instructions.

[0130] Here, it should be noted that the number of processors can be one or more. At the same time, the detection system of the embodiment of the present disclosure may also include an input device and an output device. The processor, memory, input device and output device may be connected via a bus or in other ways, which are not specifically limited here.

[0131] The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs and various modules, such as the program or module corresponding to the multi-stage and multi-level osteophyte detection method based on deep learning in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the detection system by running the software programs or modules stored in the memory.

[0132] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0133] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multi-stage and multi-level osteophyte detection method based on deep learning, characterized in that: The steps include: S100, obtaining a Dr image of osteophytes; S200, extracting masks of the femur and tibia respectively according to the Dr image, and segmenting the femur and tibia; S300, setting a osteophyte detection target based on the masks of the femur and tibia, and performing multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area; S400, presetting extraction conditions, extracting osteophyte points from the osteophyte area; In step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and the osteophyte detection target is multi-level detected using a preset deep learning model to obtain the osteophyte area, including: S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively; S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia; S303, presetting a first pixel value, taking the center point coordinate as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection; In step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and the osteophyte detection target is multi-level detected using a preset deep learning model to obtain the osteophyte area, which also includes: S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point; S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and establishing a parent rectangular frame based on the minimum enclosing rectangular frame; and obtaining a sub-rectangular frame established with each osteophyte point as a central pixel point; S330: Use the parent rectangular box and the child rectangular box as targets for multi-level detection.

2. The multi-stage and multi-level osteophyte detection method based on deep learning according to claim 1, characterized in that: In step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and the osteophyte detection target is multi-level detected using a preset deep learning model to obtain the osteophyte area, which also includes: S340, according to the rectangular area of ​​the osteophyte, calculating the aspect ratio and the area ratio of the rectangular area of ​​the osteophyte to the entire image detected, and obtaining a preset anchor box based on this; S350, based on the interlaced sampling method, at each pixel point, a preset number of target detection areas are established; S360: Based on the preset deep learning model, detect the obtained target detection areas respectively.

3. The multi-stage and multi-level osteophyte detection method based on deep learning according to claim 1, characterized in that: The preset deep learning model is: HRNet deep learning model; wherein the loss function adopts the same bbox regression and classification confidence loss function as SSD.

4. The multi-stage and multi-level osteophyte detection method based on deep learning according to claim 1, characterized in that: In step S300, the osteophyte detection target is set based on the mask of the femur and the tibia, and the osteophyte detection target is multi-level detected using a preset deep learning model to obtain the osteophyte area, which also includes: S311, according to the ROI area, obtaining a feature map obtained by applying the mask of the femur and the tibia to the ROI; S321, combining the masks of the femur and tibia, the ROI region and the feature map into a three-channel image, and fusing the three-channel images into the deep learning model as the upper and lower fields of the osteophyte region; S331. Guide model training according to the three-channel image.

5. The multi-stage and multi-level osteophyte detection method based on deep learning according to claim 2, characterized in that: In step S340, when training the model extracted from the sub-rectangular frame, it also includes: S341, obtaining a sub-region image in the model extracted by the sub-rectangular frame; S342, a sampling image erasing method, randomly erasing a certain osteophyte point sub-region in the sub-region image; S343, training the model after the image is erased.

6. The multi-stage and multi-level osteophyte detection method based on deep learning according to claim 4, characterized in that: In step S340, when training the parent rectangular frame, it also includes: S344, in the anchorbox of the parent rectangular box, according to the reduction strategy, gradually reduce the weight training strategy of the loss function corresponding to the parent node; S345, performing iterative training on the model extracted by the parent rectangular box according to the reduced weight training strategy; S346, preset a weight threshold. When the weight of the weight training strategy reaches the weight threshold, stop the iteration, and no longer extract and calculate the anchorbox corresponding to the parent node.

7. A device for implementing the multi-stage and multi-level osteophyte detection method based on deep learning as described in any one of claims 1 to 6, characterized in that: include: The femur and tibia segmentation module is used to extract the masks of the femur and tibia respectively according to the Dr image, and segment the femur and tibia; A multi-level osteophyte detection module, used to set an osteophyte detection target based on the mask of the femur and tibia, and perform multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area; A osteophyte extraction module, used to preset extraction conditions and extract osteophyte points from the osteophyte area; The method of setting a osteophyte detection target based on the mask of the femur and the tibia, and performing multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area includes: S301, preset mask reference points, and obtain mask coordinates of the femur and tibia respectively; S302, calculating and obtaining the coordinates of the center point between the two mask coordinates according to the mask coordinates of the femur and the tibia; S303, presetting a first pixel value, taking the center point coordinate as the center, obtaining a rectangular area in the Dr image that matches the first pixel value as the ROI area for osteophyte detection; The method of setting a osteophyte detection target based on the mask of the femur and the tibia, and performing multi-level detection on the osteophyte detection target using a preset deep learning model to obtain an osteophyte area, further includes: S310, presetting a second pixel value, and obtaining a rectangular area satisfying the second pixel value in the ROI area as a minimum enclosing rectangular frame of the osteophyte point; S320, obtaining the minimum enclosing rectangular frame of each osteophyte point represented by the mask coordinates, and establishing a parent rectangular frame based on the minimum enclosing rectangular frame; and obtaining a sub-rectangular frame established with each osteophyte point as a central pixel point; S330: Use the parent rectangular box and the child rectangular box as targets for multi-level detection.

8. A detection system, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the multi-stage and multi-level osteophyte detection method based on deep learning as described in any one of claims 1 to 6 when executing the executable instructions.

Citation Information

Patent Citations

  • Method and system for automatic detection of spinal bone lesions in 3d medical image data

    CN102737250A

  • Osteophyte recognition method and device, electronic equipment and storage medium

    CN113076987A