Segmentation and key point detection method and system suitable for lower limb X-ray full-length film, terminal and medium

Abstract

According to the segmentation and key point detection method and system suitable for the lower limb X-ray full-length film, the terminal and the medium provided by the invention, the model provided by the invention not only can automatically segment areas such as femur, tibia and fibula in the lower limb X-ray full-length film, but also can utilize the segmented areas to assist in identifying anatomical mark points of the lower limb; therefore, intelligent detection of the key points of the two-dimensional screen of the HTO operation and intelligent planning of the HTO operation are realized. According to the invention, operators can be copied, key points can be automatically detected, and the diagnosis efficiency of doctors can be improved. According to the method, the accuracy of the model in segmentation and detection is improved by optimizing a post-processing method of nnU-Net and a data enhancement and threshold screening method in YOLOv5.

Description

Technical Field

[0001] The present application relates to a field, and in particular to a segmentation and key point detection method, system, terminal and medium suitable for full-length X-ray films of lower limbs. Background Art

[0002] Knee osteoarthritis (KOA) is a complex degenerative joint disease. Its lesions involve damage to the knee cartilage, subchondral bone, synovium and surrounding tissues. It is one of the most common joint diseases in the elderly. At the end of the disease, it can lead to severe joint deformities, causing great psychological and physical pain to patients. With the aging of the global population and the increase in obesity, the impact of KOA on humans is getting worse year by year. According to statistics, more than 250 million people suffer from osteoarthritis worldwide, of which KOA is one of the most common types. Even patients with early and middle-stage KOA often have varus deformities such as medial cartilage wear and increased tibial or femoral varus angles. This deformity will lead to increased medial load and increased cartilage wear, which will make the deformity more serious and form a vicious circle.

[0003] At present, there is no cure for KOA: non-surgical treatments such as drugs and physical therapy can only relieve joint pain to a certain extent, but cannot stop or reverse the course of KOA. When the patient's joint function is significantly impaired, total knee arthroplasty (TKA) is currently an effective procedure for treating end-stage KOA. However, in addition to being expensive and not permanently durable, artificial joints still cause serious problems such as prosthetic discomfort, limited mobility, and even infection after replacement. Therefore, exploring new treatment methods to intervene in the progression of early and mid-stage KOA and delay or even avoid TKA is of great significance for alleviating patient pain, reducing medical costs, and improving quality of life.

[0004] Since Coventry first reported high tibial osteotomy (HTO), HTO has been used in the early and middle treatment of KOA for more than 50 years. After long-term follow-up observation of patients after surgery, the efficacy of HTO has been widely recognized, and it has now developed into a safe, accurate and effective treatment method. The operation aims to correct the force line of the lower limb by osteotomy of the proximal tibia, translate the lower leg bone, and transfer the force line from the severely worn and worn knee compartment to the relatively normal compartment on the other side, thereby reducing the load on the affected compartment of the knee joint, relieving pain in the knee joint and delaying the course of KOA, and even allowing the cartilage and meniscus on the damaged compartment to repair themselves. Among the many factors that affect the efficacy of HTO, how to achieve precise control of the force line of the lower limb after surgery is a key issue to be solved in HTO. Insufficient and excessive force correction are the most common complications after HTO surgery. Once the force correction is insufficient, it is difficult to effectively reduce the mechanical load of the damaged compartment and prevent the development of KOA. If the force correction is excessive, it will cause excessive load on the normal compartment and cause problems such as patellar subluxation, low patella, and accelerated degeneration of the lateral cartilage. Therefore, achieving precise force control is one of the keys to HTO surgery.

[0005] The surgical planning of HTO is usually based on X-ray and CT images of the lower limbs. The surgeon manually marks the anatomical landmarks of the lower limbs on the full-length film of the lower limbs and manually measures the force lines of the lower limbs to plan the key surgical parameters such as the distraction angle and osteotomy point during the operation. However, this planning method is not only time-consuming and labor-intensive, but also highly subjective. In addition, the manual error is too large, and the accuracy of quantitative analysis is seriously insufficient. In recent years, the use of computer-aided design and production technology, such as three-dimensional reconstruction of the lower limbs and finite element simulation technology, to achieve automation and precision in HTO surgical planning has become a research hotspot. Among them, computer-assisted surgery technology based on artificial intelligence and deep learning has broad application prospects in realizing the automatic analysis and processing of preoperative imaging data.

[0006] Artificial intelligence is a technology and theory that simulates human intelligence, aiming to enable computers to think, learn, judge and make decisions like humans. Its core idea is to establish mathematical models through learning and analyzing large amounts of data, so that it can autonomously identify, classify, reason and predict. Deep learning, relying on its complex multi-layer neural network structure, can automatically represent and learn large amounts of data, and is a powerful promoter of the rapid progress of artificial intelligence technology. Medical image analysis based on deep learning is currently a hot topic in the field of intelligent medical research. Using huge image data sets, deep learning technology has shown unlimited potential in the fields of lesion detection, image segmentation, image configuration and image reconstruction. The nnU-Net proposed by Isensee et al. is an automated medical image segmentation framework based on U-Net and 3D U-Net, which achieves adaptation to a variety of medical segmentation tasks. By focusing on data preprocessing and training strategy optimization rather than network structure, the model achieves dynamic adaptation to different medical image data sets and has achieved very good results in most medical image segmentation tasks. The YOLO series detection model proposed by Redmon et al. is one of the most popular algorithms in the current one-stage target detection model. This model directly generates anchor frames based on the grid, locates and classifies all targets in the entire image at one time, and achieves the best balance between detection accuracy and speed. Therefore, how to better apply artificial intelligence to KOA analysis is a technical challenge facing this field. Summary of the invention

[0007] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a segmentation and key point detection method, system, terminal and medium suitable for lower limb full-length X-ray films, so as to solve the technical problem of how to better apply artificial intelligence to KOA analysis.

[0008] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a segmentation and key point detection system suitable for full-length X-ray films of lower limbs, including: a regional segmentation module, used to segment the full-length X-ray film to extract the ROI area of ​​the bony structure involved in the HTO surgical planning from the full-length X-ray film; a regional fusion module, used to reorganize the extracted ROI areas of each bony structure according to the position information, and fuse the reorganized image with the full-length X-ray film of the lower limb to generate a full-length X-ray film of the lower limb with regional extraction information; a key point detection module, used to input the full-length X-ray film of the lower limb with regional extraction information into a key point detection model to output a variety of lower limb key point information.

[0009] In some embodiments of the first aspect of the present application, the regional segmentation module divides the full-length X-ray film of the lower limbs symmetrically along the longitudinal direction, and inputs the sub-images obtained by the division into the femur extraction network and the tibia and fibula extraction network respectively to extract the surgical area of ​​interest; the extracted surgical area of ​​interest includes the tibia area, the femur area and the fibula area.

[0010] In some embodiments of the first aspect of the present application, the femur extraction network is based on a full-length X-ray of the lower limbs and the femur segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model; the tibia and fibula extraction network is based on a full-length X-ray of the lower limbs and the tibia segmentation and annotation data and the fibula segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model.

[0011] In some embodiments of the first aspect of the present application, the regional segmentation module also executes a post-processing strategy after extracting the bony structure area involved in the HTO surgical planning from the full-length X-ray film; the post-processing strategy is based on the analysis of the connected areas on the predicted segmentation result image, first binarizing the predicted result image, retaining the two largest connected areas based on the binarization results, and removing other components.

[0012] In some embodiments of the first aspect of the present application, the key point detection model is based on key point annotation data integrated with ROI information and is obtained by model training using a YOLOv5 network.

[0013] In some embodiments of the first aspect of the present application, a gamma transform data enhancement strategy is added to the YOLOv5 network to reduce the detection errors of key points by the YOLOv5 network caused by the differences in grayscale distribution between full-length X-ray films.

[0014] In some embodiments of the first aspect of the present application, the gamma transform method used by the gamma transform data enhancement strategy is as follows: new =i old γ ; where i new Represents the gray value of the original image, i old represents the grayscale value of the transformed image, and γ represents the parameter that controls the transformation intensity; when γ>1, the grayscale of the brighter area in the image is stretched, the grayscale of the darker area is compressed, and the image becomes darker as a whole; when γ<1, the grayscale of the brighter area in the image is compressed, the grayscale of the darker area is stretched, and the image becomes brighter as a whole.

[0015] In some embodiments of the first aspect of the present application, the key point detection model is screened based on an optimized threshold screening strategy; the threshold screening strategy includes: setting a preset low threshold so that the YOLOv5 detection model can detect at least two detection boxes in each category; calculating the confidence of all detection boxes corresponding to each category, retaining the two detection boxes with the largest confidence and deleting the remaining detection boxes.

[0016] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a segmentation and key point detection method suitable for a full-length X-ray film of lower limbs, comprising: segmenting the full-length X-ray film to extract the ROI area of ​​the bony structure involved in the HTO surgical planning from the full-length X-ray film; reorganizing the extracted ROI areas of each bony structure according to the position information, and fusing the reorganized image with the full-length X-ray film of the lower limbs to generate a full-length X-ray film of the lower limbs with regional extraction information; inputting the full-length X-ray film of the lower limbs with regional extraction information into a key point detection model to output a variety of lower limb key point information.

[0017] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the segmentation and key point detection method applicable to full-length X-ray films of lower limbs.

[0018] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the segmentation and key point detection method applicable to full-length X-ray films of lower limbs.

[0019] As described above, the segmentation and key point detection method, system, terminal and medium applicable to the full-length X-ray film of lower limbs of the present application have the following beneficial effects:

[0020] (1) The present invention provides a comprehensive and efficient detection model: Traditional detection models usually only focus on the analysis of characteristics, while the model proposed in the present invention can not only automatically segment the femur, tibia and fibula in the full-length X-ray film of the lower limb, but also use the segmented areas to assist in the identification of anatomical landmarks of the lower limb, thereby realizing the intelligent detection of key points on the two-dimensional screen of HTO surgery and the intelligent planning of HTO surgery.

[0021] (2) Reduce labor costs: Traditional HTO surgery planning usually requires the surgeon to manually mark the key anatomical points of the lower limb on the full-length X-ray film of the lower limb, and then conduct further parameter measurement and analysis. However, the present invention can replicate the surgeon, realize automatic detection of key points, and assist in improving the doctor's diagnostic efficiency.

[0022] (3) Algorithm optimization to improve detection accuracy: The present invention improves the accuracy of the model in segmentation and detection by optimizing the post-processing method of nnU-Net and the data enhancement and threshold screening methods in YOLOv5.

[0023] In summary, the present invention uses deep learning to develop an intelligent segmentation and key point detection model for full-length X-ray films of lower limbs, which can effectively assist doctors in HTO surgery planning, improve planning efficiency, and reduce labor costs, and has certain clinical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Shown is a structural schematic diagram of a segmentation and key point detection system suitable for lower limb full-length X-ray films in one embodiment of the present application.

[0025] Figure 2A Shown is the original image of the full-length CT scan of the lower limbs in one embodiment of the present application.

[0026] Figure 2B Shown is a prediction diagram without post-processing of a full-length CT scan of the lower limbs in one embodiment of the present application.

[0027] Figure 2C Shown is a post-processed prediction image of a full-length CT scan of the lower limbs in one embodiment of the present application.

[0028] Figure 3 Shown is a flow chart of a method for segmentation and key point detection applicable to full-length X-ray films of lower limbs in one embodiment of the present application.

[0029] Figure 4 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0032] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0033] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0034] In order to solve the problems in the above-mentioned background technology, the present invention provides a segmentation and key point detection method, system, terminal and medium suitable for lower limb full-length X-ray films, aiming to build an intelligent segmentation and key point detection model for lower limb full-length X-ray films based on nnU-Net and YOLOv5, and the model is mainly composed of two sub-models, namely a segmentation model and a key point detection model.

[0035] The segmentation model realizes accurate segmentation of HTO surgical areas such as femur, tibia and fibula in full-length X-ray films of lower limbs based on nnU-Net. The present invention optimizes the post-processing algorithm on the basis of the basic model, so that the model can process the prediction results, retain the key areas, and remove possible wrong segmentation areas, thereby improving the robustness and accuracy of the basic model on this model. The key point detection model realizes intelligent detection of key points of HTO surgery in full-length X-ray films of lower limbs based on YOLOv5. The present invention optimizes the data enhancement and threshold screening algorithms on the basis of the basic model, introduces Gamma enhancement to improve the generalization ability of the model for grayscale value differences of full-length X-ray films, and uses the uniformity of the detection task to add threshold screening while reducing the detection confidence threshold to solve the problems of model misdetection and false detection, effectively improving the detection ability of the model.

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0037] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0038] <1> Full-length X-ray of lower limbs: used to examine lesions of the hip joint, femur, knee joint, etc. Full-length weight-bearing X-ray of both lower limbs is an AP X-ray of both lower limbs taken when the patient is standing.

[0039] <2> ROI (Region of interest): In machine vision and image processing, the area to be processed is outlined in the image in the form of a box, circle, irregular polygon, etc. This is called the ROI. In machine vision software such as Halcon, OpenCV, and Matlab, various operators and functions are often used to obtain the ROI of the region of interest and perform the next step of image processing.

[0040] The embodiments of the present invention provide a segmentation and key point detection method applicable to a full-length X-ray film of a lower limb, a system applicable to the segmentation and key point detection method for a full-length X-ray film of a lower limb, and a storage medium storing an executable program for implementing the segmentation and key point detection method applicable to a full-length X-ray film of a lower limb. As for the implementation of the segmentation and key point detection method applicable to a full-length X-ray film of a lower limb, the embodiments of the present invention will describe an exemplary implementation scenario of the segmentation and key point detection method applicable to a full-length X-ray film of a lower limb.

[0041] like Figure 1As shown, a schematic diagram of the structure of a segmentation and key point detection system for lower limb full-length X-ray films in an embodiment of the present invention is shown. The segmentation and key point detection system for lower limb full-length X-ray films in this embodiment uses a model based on a deep learning algorithm, which can automatically extract surgical areas and key points from full-length X-ray films, providing doctors with more accurate and reliable surgical planning and implementation support.

[0042] In the embodiment of the present invention, the segmentation and key point detection system 100 for lower limb full-length X-ray films specifically includes: a region segmentation module 101, a region fusion module 102, and a key point detection module 103. Hereinafter, the implementation process and principle of each module will be explained in detail with reference to specific examples.

[0043] The region segmentation module 101 is used to segment the full-length X-ray film to extract the ROI region of the bone structure involved in the HTO surgical planning from the full-length X-ray film. It should be understood that the full-length X-ray film of the lower limbs can fully display the overall picture of the patient's lower limb skeletal system, including important parts such as the hip joint, femur, tibia, fibula, knee and ankle joints. However, since the HTO surgical planning usually only involves the three regions of femur, tibia and fibula, the region segmentation module 101 extracts the three regions of femur, tibia and fibula from the full-length X-ray film to allow the subsequent detection network to pay more attention to the ROI (Region of Interest).

[0044] Specifically, the region segmentation module 101 symmetrically divides the full-length X-ray film of the lower limbs in the longitudinal direction, and inputs the sub-images obtained by the division into the femur extraction network and the tibia and fibula extraction network respectively to extract the region of interest for the surgery; the extracted region of interest for the surgery includes but is not limited to the tibia region, the femur region and the fibula region. It is worth noting that after the region extraction, the redundant information in the image can be effectively reduced, and the region of interest ROI of the surgery can be used as a mask to provide guidance information for subsequent detection.

[0045] In an embodiment of the present invention, the femur extraction network is based on the full-length X-ray film of the lower limbs and the femur segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model; the tibia and fibula extraction network is based on the full-length X-ray film of the lower limbs and the tibia segmentation and annotation data and the fibula segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model.

[0046] It is worth noting that the nnU-Net segmentation model used in the embodiment of the present invention is different from the usual U-Net segmentation model. Codec networks represented by UNet are commonly used in medical image segmentation. Although image segmentation can be achieved, the complexity and diversity of medical images themselves also affect the generalization and versatility of the U-Net segmentation model, which is reflected in the following aspects: (1) There are large differences between medical images of various modalities, such as the size of the research cohort, image size and dimension, resolution and voxel intensity; (2) The semantic labels of segmentation are extremely unbalanced; compared with the normal tissue in the image, the lesion area generally occupies only a very small part, which results in an extreme difference between the voxel labels of normal tissue and the voxel labels of lesion tissue; (3) There are large differences in expert annotations between different image data, and the annotation results of some images are ambiguous; (4) Some data sets have obvious differences in attributes such as image geometry and shape, and the problems of slice misalignment and anisotropy are also very serious.

[0047] In view of this, nnU-Net (no new Net) pays more attention to image preprocessing, can automatically determine the image modality and perform corresponding normalization operations, and resample different image voxel spacings according to cubic spline interpolation. In terms of network structure, nnU-Net proposes three networks based on the original U-Net, namely 2D U-Net, 3DU-Net and two cascaded 3D U-Nets. The first of the two cascaded 3D U-Nets is used to generate coarse segmentation results, and the second is used to refine the coarse segmentation results. Compared with the original U-Net network, the nnU-Net model has made certain structural adjustments, including but not limited to: using padded convolution to achieve the same output and input shapes, using Leaky ReLU instead of ReLU, and using instance normalization instead of batch normalization. In addition, the nnU-Net model can automatically set hyperparameters, such as training batch size, image block size, number of downsampling, etc. All U-Net architectures have passed 5-fold cross validation, using cross entropy loss and Dice loss as loss functions during training. The optimizer can use Adam, and set a dynamic learning rate adjustment strategy. The online data enhancement strategy is also used during training. Since the nnU-Net model itself is already available, it will not be described in detail.

[0048] Preferably, the region segmentation module 101 further executes a post-processing strategy after extracting the bony structure region involved in the HTO surgical planning from the full-length X-ray film; the post-processing strategy is based on the analysis of the connected regions on the predicted segmentation result map, first binarizing the predicted result map, retaining the two largest connected regions based on the binarization result, and removing other components, so as to achieve the purpose of not belonging to the surgical region of interest ROI in the effective region segmentation prediction result. The embodiment of the present invention redesigns the post-processing strategy in the model based on the prior knowledge that the human lower limbs contain two instances of the target structure. To facilitate understanding by those skilled in the art, the following is provided: Figures 2A to 2C To compare the effects with and without post-processing; Figure 2A It's the original picture. Figure 2B is the prediction map without post-processing, Figure 2C It is the prediction map after post-processing. It can be seen that the prediction map after post-processing is more focused on the ROI area after removing irrelevant components.

[0049] The regional fusion module 102 is used to reorganize the extracted ROI regions of each bone structure according to the position information, and fuse the reorganized image with the lower limb X-ray full-length film to generate a lower limb X-ray full-length film with regional extraction information.

[0050] Specifically, the regional fusion module 102 reassembles the ROI regions of the femur, tibia and fibula according to the position information, and then fuses the reassembled image with the lower limb X-ray full-length film, thereby obtaining a complete two-dimensional lower limb X-ray full-length film containing regional extraction information. Since the X-ray film and the extracted ROI image are both single-channel grayscale images, when performing fusion, the method used is to splice the X-ray film and the ROI image in the channel direction.

[0051] The key point detection module 103 is used to input the lower limb X-ray full-length film with region extraction information into a key point detection model to output a variety of lower limb key point information.

[0052] For example, a full-length X-ray film with ROI information is input into the key point detection network, which outputs the location information of 18 key points such as the greater trochanter landmark point and the lateral tibial malleolus point.

[0053] In an embodiment of the present invention, the key point detection model is obtained by training the model using the YOLOv5 network based on the key point annotation data fused with ROI information. It should be understood that the YOLOv5 network can generally be divided into four modules, namely, the input module, the reference network module, the Neck network module and the Head output module. The versions of the YOLOv5 algorithm include but are not limited to YOLOv5s, YOLOv5m, YOLOv5I, and YOLOv5x. YOLOv5s is used as an example to illustrate as follows: the input module is used to input pictures. For example, the input image size of the network is 608*608. This stage usually includes an image preprocessing stage, that is, scaling the input image to the input size of the network and performing normalization operations. The reference network module is a classifier network with excellent performance. This module is used to extract some common feature representations; YOLOv5 not only uses the CSPDarknet53 structure, but also uses the Focus structure as the reference network. The Neck network module is usually located in the middle of the baseline network and the head network, and it can be used to further improve the diversity and robustness of features. The Head output module is used to complete the output of target detection results.

[0054] Furthermore, although other classifiers (such as VGGNet or Inception classifiers) are used in some scenarios, an object detector is formed by traversing the image through a small sliding window. The classifier is run once at each step to classify the object in the current window. Although this detection method is feasible, it is inefficient and the detection process is very slow because the classifier needs to be run many times. The YOLO (You Only Look Once) series of models adopt a completely different approach. For example, the YOLO model divides the image into 13*13 grid cells, each grid cell is responsible for predicting 5 detection boxes, and a detection box contains a rectangular area of ​​​​an object. YOLO also gives a confidence level, which describes the degree of certainty that a certain detection box does contain a certain object. Generally, the higher the confidence level, the thicker the detection box. For each detection box, the corresponding grid cell also gives a classification prediction. The confidence level and classification prediction of the detection box are finally integrated into a final score to obtain the probability that a specific type of object is contained in the detection box.

[0055] More preferably, the embodiment of the present invention adds a gamma transform data enhancement strategy in the YOLOv5 network to reduce the detection errors of the YOLOv5 network on key points caused by the grayscale distribution differences between the full-length X-ray films. Specifically, since the various data enhancement strategies adopted by YOLOv5 during the training process are mostly for natural images of color, they cannot be applied to single-channel grayscale images such as full-length X-ray films. In addition, the grayscale value of the full-length X-ray film is determined by the different absorbances produced by the X-ray passing through the body tissue. The full-length X-ray films of different hospitals may have certain differences in grayscale distribution between the full-length X-ray films due to the differences in the grayscale range and resolution of the X-ray machine, and the shooting conditions such as exposure time, tube voltage, filter, etc., and this difference is bound to affect the detection effect of YOLOv5 on key points. Therefore, the present invention adds a gamma transform data enhancement method to the original data enhancement.

[0056] Gamma transform, also known as exponential transform or power transform, is a commonly used data enhancement method for nonlinear transformation of the grayscale value of the input image. The transformation formula is as follows:

[0057] i new =i old γ ;Formula (1)

[0058] Among them, i new Represents the gray value of the original image, i old represents the grayscale value of the transformed image, and γ represents the parameter that controls the transformation strength. When γ>1, the grayscale of the brighter area in the image is stretched, the grayscale of the darker area is compressed, and the image becomes darker overall. Conversely, when γ<1, the grayscale of the brighter area in the image is compressed, the grayscale of the darker area is stretched, and the image becomes brighter overall. Therefore, performing a random gamma transformation within a certain range on the entire data set can improve the model's generalization ability for grayscale differences in full-length X-ray films.

[0059] More preferably, an embodiment of the present invention proposes an optimized threshold screening method, which includes: setting a preset low threshold so that the YOLOv5 detection model can detect at least two detection frames for each category; calculating the confidence of all detection frames corresponding to each category, retaining the two detection frames with the largest confidence and deleting the remaining detection frames.

[0060] In the YOLOv5 network, threshold screening and non-maximum suppression are usually used to ensure that the final prediction box corresponds one-to-one with the target. For most target detection tasks, the type and number of targets in each image to be detected are unpredictable, but in the data set to be detected in the embodiment of the present invention, each image to be detected is in a unified form, and the types and numbers of key points expected to be detected on these images to be detected are fixed, that is, 18 left-right symmetrical key points, a total of 36 key points. In view of this, the embodiment of the present invention optimizes the threshold screening method in YOLOv5.

[0061] Since the threshold set during detection is fixed and cannot be dynamically adjusted according to the difficulty of different types of detection, there are theoretically two ways to set the threshold: the first is to set the threshold high enough to ensure that there will be no false detections, but this will also cause difficult-to-detect targets to be easily missed; the second is to set the threshold low to ensure that each target key point can be detected, but this method may result in multiple detection boxes corresponding to easy-to-detect points.

[0062] Therefore, the embodiment of the present invention analyzes the expected number and characteristics of key points, assuming that when the detection threshold is low enough, YOLOv5 can detect at least two boxes for each type of the 18 key points expected to be detected. Although this may also result in multiple detection boxes for points that are easy to detect, this problem can be solved by adding an algorithm. The algorithm only retains the top two boxes in each type of all detection boxes and deletes other boxes, so that difficult-to-detect targets can be detected while removing duplicate detection boxes, thereby achieving the purpose of improving the overall detection accuracy and recall rate.

[0063] It should be noted that the segmentation and key point detection system for lower limb full-length X-ray films provided in the above embodiment only uses the division of the above-mentioned program modules as an example when performing segmentation and key point detection on lower limb full-length X-ray films. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the system can be divided into different program modules to complete all or part of the processing described above.

[0064] like Figure 3 As shown, a flow chart of a method for segmentation and key point detection of a full-length X-ray film of lower limbs according to an embodiment of the present invention is shown. The method for segmentation and key point detection of a full-length X-ray film of lower limbs according to an embodiment of the present invention is executed as follows:

[0065] Step S31: segmenting the full-length X-ray film to extract the ROI region of the bone structure involved in the HTO surgery planning from the full-length X-ray film.

[0066] Step S32: reorganize the extracted ROI regions of the bony structures according to the position information, and fuse the reorganized image with the lower limb X-ray full-length film to generate a lower limb X-ray full-length film with the region extraction information.

[0067] Step S33: input the lower limb X-ray full-length film with region extraction information into a key point detection model to output a variety of lower limb key point information.

[0068] In some examples, the method also includes: symmetrically dividing the full-length lower limb X-ray film along the longitudinal direction, and inputting the sub-images obtained by the division into the femur extraction network and the tibia and fibula extraction network respectively to extract the surgical area of ​​interest; the extracted surgical area of ​​interest includes the tibia area, the femur area and the fibula area.

[0069] Furthermore, the femur extraction network is obtained after training with the nnU-Net segmentation model based on the full-length X-ray film of the lower limbs and the femur segmentation and annotation data thereon; the tibia and fibula extraction network is obtained after training with the nnU-Net segmentation model based on the full-length X-ray film of the lower limbs and the tibia segmentation and annotation data and the fibula segmentation and annotation data thereon.

[0070] In some examples, the method also includes: executing a post-processing strategy after extracting the bony structure area involved in the HTO surgical planning from the full-length X-ray film; the post-processing strategy is based on the analysis of the connected areas on the predicted segmentation result image, first binarizing the predicted result image, retaining the two largest connected areas based on the binarization processing results, and removing other components.

[0071] In some examples, the key point detection model is based on key point annotation data fused with ROI information and is obtained by model training using a YOLOv5 network.

[0072] In some examples, a gamma transform data enhancement strategy is added to the YOLOv5 network to reduce the YOLOv5 network's key point detection errors caused by differences in grayscale distribution between full-length X-ray films.

[0073] Furthermore, the gamma transform method used by the gamma transform data enhancement strategy is as follows: new =i old γ ; where i new Represents the gray value of the original image, i oldrepresents the grayscale value of the transformed image, and γ represents the parameter that controls the transformation intensity; when γ>1, the grayscale of the brighter area in the image is stretched, the grayscale of the darker area is compressed, and the image becomes darker as a whole; when γ<1, the grayscale of the brighter area in the image is compressed, the grayscale of the darker area is stretched, and the image becomes brighter as a whole.

[0074] In some examples, the key point detection model is screened based on an optimized threshold screening strategy; the threshold screening strategy includes: setting a preset low threshold so that the YOLOv5 detection model can detect at least two detection boxes for each category; calculating the confidence of all detection boxes corresponding to each category, retaining the two detection boxes with the highest confidence and deleting the remaining detection boxes.

[0075] It should be noted that the segmentation and key point detection method for lower limb full-length X-ray films provided in the above embodiment and the segmentation and key point detection system embodiment for lower limb full-length X-ray films belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0076] The segmentation and key point detection method for the full-length X-ray film of the lower limbs provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the segmentation and key point detection terminal for the full-length X-ray film of the lower limbs, please refer to Figure 4 , is an optional hardware structure diagram of an electronic terminal 400 provided in an embodiment of the present invention. The terminal 400 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 400 suitable for segmentation and key point detection of full-length X-ray films of lower limbs includes: at least one processor 401, a memory 402, at least one network interface 404 and a user interface 406. The various components in the device are coupled together through a bus system 405. It can be understood that the bus system 405 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 405 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 4 In the specification, various buses are labeled as bus systems.

[0077] The user interface 406 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0078] It is understood that the memory 402 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0079] The memory 402 in the embodiment of the present invention is used to store various categories of data to support the operation of the terminal 400 for segmentation and key point detection of full-length X-ray films of lower limbs. Examples of these data include: any executable program used to operate on the terminal 400 for segmentation and key point detection of full-length X-ray films of lower limbs, such as an operating system 4021 and an application 4022; the operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 4022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The segmentation and key point detection method for full-length X-ray films of lower limbs provided in the embodiment of the present invention may be included in the application 4022.

[0080] The method disclosed in the above embodiment of the present invention can be applied to the processor 401, or implemented by the processor 401. The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 401 or the instruction in the form of software. The above processor 401 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 401 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 401 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0081] In an exemplary embodiment, the segmentation and key point detection terminal 400 suitable for full-length X-ray films of lower limbs can be used by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the above-mentioned method.

[0082] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0083] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0084] In summary, the present application provides a method, device, terminal and medium for segmentation and key point detection of full-length X-ray films of lower limbs. The present invention provides a comprehensive and efficient detection model: traditional detection models usually only focus on the analysis of characteristics, while the model proposed in the present invention can not only automatically segment the femur, tibia and fibula in the full-length X-ray film of the lower limb, but also use the segmented areas to assist in identifying the anatomical landmarks of the lower limbs, thereby realizing the intelligent detection of key points on the two-dimensional screen of HTO surgery and the intelligent planning of HTO surgery. Reduce labor costs: Traditional HTO surgery planning usually requires the surgeon to manually mark the anatomical key points of the lower limbs on the full-length X-ray film of the lower limb, and then perform further parameter measurement and analysis. However, this model can copy the surgeon to realize automatic detection of key points, which helps to improve the diagnostic efficiency of doctors and improve clinical efficacy. Algorithm optimization improves detection accuracy: The present invention improves the accuracy of the model in segmentation and detection by optimizing the post-processing method of nnU-Net and the data enhancement and threshold screening methods in YOLOv5. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0085] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A segmentation and key point detection system for lower limb full-length X-ray films, characterized in that: include: A region segmentation module, used to segment the full-length X-ray film to extract the ROI region of the bone structure involved in the HTO surgery planning from the full-length X-ray film; A regional fusion module, used for reorganizing the extracted ROI regions of each bone structure according to the position information, and fusing the reorganized image with the lower limb X-ray full-length film to generate a lower limb X-ray full-length film with regional extraction information; The key point detection module is used to input the lower limb X-ray full-length film with region extraction information into the key point detection model to output a variety of lower limb key point information.

2. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 1, characterized in that: The regional segmentation module divides the full-length X-ray film of the lower limbs symmetrically along the longitudinal direction, and inputs the sub-images obtained by the division into the femur extraction network and the tibia and fibula extraction network respectively to extract the surgical regions of interest; the extracted surgical regions of interest include the tibia region, the femur region and the fibula region.

3. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 2, characterized in that: The femur extraction network is based on the full-length X-ray film of the lower limbs and the femur segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model; the tibia and fibula extraction network is based on the full-length X-ray film of the lower limbs and the tibia segmentation and annotation data and the fibula segmentation and annotation data thereon, and is obtained after training using the nnU-Net segmentation model.

4. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 1, characterized in that: The regional segmentation module also executes a post-processing strategy after extracting the bone structure area involved in the HTO surgical planning from the full-length X-ray film; the post-processing strategy is based on the analysis of the connected areas on the predicted segmentation result map, first binarizing the predicted result map, retaining the two largest connected areas based on the binarization processing results, and removing other components.

5. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 1, characterized in that: The key point detection model is based on the key point annotation data integrated with ROI information and is obtained by model training using the YOLOv5 network.

6. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 5, characterized in that: A gamma transform data enhancement strategy is added to the YOLOv5 network to reduce the detection errors of the YOLOv5 network on key points caused by the grayscale distribution differences between full-length X-ray films.

7. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 6, characterized in that: The gamma transform method used by the gamma transform data enhancement strategy is as follows: i new =i old γ ; Among them, i new Represents the gray value of the original image, i old represents the grayscale value of the transformed image, and γ represents the parameter that controls the transformation intensity; when γ>1, the grayscale of the brighter area in the image is stretched, the grayscale of the darker area is compressed, and the image becomes darker as a whole; when γ<1, the grayscale of the brighter area in the image is compressed, the grayscale of the darker area is stretched, and the image becomes brighter as a whole.

8. The segmentation and key point detection system for lower limb full-length X-ray films according to claim 5, characterized in that: The key point detection model is screened based on an optimized threshold screening strategy; The threshold screening strategy includes: setting a preset low threshold so that the YOLOv5 detection model can detect at least two detection frames for each category; calculating the confidence of all detection frames corresponding to each category, retaining the two detection frames with the largest confidence and deleting the remaining detection frames.

9. A method for segmentation and key point detection of full-length X-ray films of lower limbs, characterized in that: include: Segmenting the full-length X-ray film to extract the ROI region of the bone structure involved in the HTO surgical planning from the full-length X-ray film; Recombining the extracted ROI regions of each bone structure according to the position information, and fusing the recombined image with the lower limb X-ray full-length film to generate a lower limb X-ray full-length film with the region extraction information; The full-length X-ray film of the lower limbs with the region extraction information is input into the key point detection model to output a variety of lower limb key point information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the segmentation and key point detection method applicable to full-length X-ray films of lower limbs as described in claim 10 is implemented.

11. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the terminal executes the segmentation and key point detection method applicable to full-length X-ray films of lower limbs as described in claim 10.

Images