Lower limb coronal view parameter automatic measurement system and method based on deep learning, terminal and medium

Abstract

The invention provides a lower limb coronal parameter automatic measurement system and method based on deep learning, a terminal and a medium, and the system comprises an image collection module which is used for collecting a lower limb X-ray image of a patient; the image preprocessing module is used for preprocessing the acquired lower limb X-ray image; the preprocessing comprises image noise reduction and / or image enhancement; the deep learning model module is used for performing feature extraction and classification by adopting a deep learning model and outputting anatomical mark points and thighbone anatomical axes of the lower limb X-ray images; and the parameter calculation module is used for carrying out visual data processing and rendering so as to convert the anatomical mark points of the lower limb X-ray image and the anatomical axis of the thighbone into corresponding lower limb coronal morphology parameters. The method has the advantages that the accuracy and efficiency of measurement are improved, the labor cost is reduced, convenient services are provided, and the method is applied to large-scale epidemiological research.

Description

Technical Field

[0001] The present application relates to a field, and in particular to a system, method, terminal and medium for automatically measuring lower limb coronal parameters based on deep learning. Background Art

[0002] Osteoarthritis (OA) is a complex degenerative disease of all joints. Pathological changes include cartilage wear, subchondral bone remodeling, osteophyte formation and secondary periarticular muscle atrophy. It can affect multiple parts of the body, such as the knee, hip, finger joints and spine, with the knee being the most common. Knee osteoarthritis (KOA) is one of the main causes of pain and disability in the elderly, affecting approximately 250 million people worldwide. Among people over 60 years old, the incidence of KOA in my country is 60.1%, and the prevalence gradually increases with age. In addition to causing knee pain, swelling, deformity and dysfunction, the long-term disease state of KOA can also affect the patient's mental health, increase the risk of cardiovascular and cerebrovascular accidents, the risk of falls and all-cause mortality, and bring a heavy burden to society, families and individuals.

[0003] The risk factors for KOA have been extensively studied, and modifiable risk factors are a hot topic of research. Among them, the relationship between the overall morphological deformity of the lower limb coronal plane and the prevalence of KOA has been clarified. The overall poor force alignment of the lower limb coronal plane can cause abnormal load distribution in the knee joint, ultimately leading to the progression of KOA in the lateral compartment with greater load. For patients with early KOA with poor overall force alignment of the lower limb coronal plane, safe and non-invasive intervention measures, such as knee orthopedic braces, are recommended. When KOA progresses or conservative treatment is ineffective, surgical treatment is usually recommended. The choice of surgical procedure is related to the location of the coronal morphological deformity of the lower limb: when the deformity is mainly from the joint, unicompartmental arthroplasty is recommended; when the deformity is mainly from the periarticular plane, periarticular knee osteotomy is a better choice. Therefore, accurately evaluating the coronal morphological parameters of the lower limb related to KOA is a prerequisite for early screening and reasonable treatment.

[0004] The coronal morphological parameters of the lower limbs include global parameters and local parameters. The global parameters are mainly the hip-knee-ankle angle (HKAA) and the weight-bearing line rate (WBLR), which represent the overall force alignment of the lower limb. When the overall force alignment is poor, the pressure of the medial and lateral compartments of the knee joint will be unbalanced, resulting in excessive pressure load in the affected compartment, which in turn accelerates the pathological process of KOA, such as cartilage wear and subchondral bone sclerosis. The subchondral trabecular microstructure is related to the severity of HKAA and KOA. With the increase of knee joint alignment deviation and KOA severity, the subchondral trabecular bone volume, trabecular number and trabecular thickness of the affected tibial plateau increase, and the trabecular separation decreases. In the non-KOA population, 180° is considered to be the neutral position, greater than 180° is valgus, and less than 180° is varus. However, many studies have found that the HKAA of normal people is slightly varus, so a HKAA of 180° does not represent normal. Correction surgeries include total knee replacement, unicompartmental knee replacement, and osteotomy around the knee.

[0005] There are two views on the degree of alignment in total knee replacement:

[0006] (1) Correction to the neutral position. Scholars who support this view believe that when the force line is corrected to 180°±3°, the local stress distribution of the implant is more uniform, which can effectively reduce implant wear and loosening, extend the life of the implant, and reduce the revision rate. However, many patients are naturally in a varus force line state, and the neutral force line state is abnormal for them. If you want to achieve neutral force line correction, a certain degree of medial soft tissue release must be performed during the operation.

[0007] (2) Correction to the natural force alignment state before the disease occurs, that is, anatomical reconstruction. This group of scholars advocates personalized reconstruction, usually using the healthy lower limb as a reference, with the goal of restoring the anatomical contour of the affected side before the disease occurs. Although the patient's natural varus force alignment state after surgery may affect the long-term survival of the implant, clinically, in patients with preoperative varus force alignment deformity, residual varus force alignment after surgery does not lead to an increase in the revision rate.

[0008] Unicompartmental arthroplasty is a good choice for patients with unicompartmental KOA without ligament laxity. Compared with total knee replacement, unicompartmental arthroplasty has many advantages, including more natural knee kinematics, more knee function retention, and more bone mass retention. The concept of unicompartmental arthroplasty is also personalized reconstruction to restore the force line state before the onset of the disease. Osteotomy around the knee is another idea to reconstruct the force line of the lower limb, represented by high tibial osteotomy. This surgery is mainly suitable for medial compartment KOA. By changing the morphology of the tibia, the force line of the lower limb is shifted outward, and the pressure on the medial compartment is reduced, which can delay the progression of KOA. The latest animal experiments have shown that after high tibial osteotomy, the cartilage of the affected compartment can be restored to a certain extent, which provides an important theoretical basis for high tibial osteotomy.

[0009] WBLR is mainly used in the field of high tibial osteotomy as an indicator of surgical goal and correction effect evaluation. Therefore, the normal range of WBLR in non-KOA population has not been clearly defined. There are many local parameters, including those of the hip, knee, and ankle joints. The mechanical lateral distal femoral angle (mLDFA), mechanical medial-proximal-tibial angle (mMPTA), and joint line convergence angle (JLCA) are local parameters of the knee joint. mLDFA and mMPTA reflect the morphology of the distal femur and proximal tibia, respectively, and JLCA reflects the relative relationship between the femur and tibia.

[0010] The local parameters of the knee joint are closely related to the overall parameter HKAA, and there is a clear mathematical relationship: HKAA = mMPTA + (180°-mLDFA)-JLCA. Another local parameter of the knee joint, knee joint line orientation (KJLO), is related to the efficacy of high tibial osteotomy. Excessive KJLO will affect the prognosis of patients undergoing high tibial osteotomy. Since the lower limbs are a whole, the local parameters of the hip and ankle joints are also related to the local parameters and overall parameters of the knee joint.

[0011] The mechanical axis of hip joint parameters, the mechanical lateral proximal femoral angle (mLPFA), can reflect the relationship between the proximal femur and the femoral shaft to a certain extent, and therefore can affect HKAA and mLDFA. The mechanical axis of ankle joint parameters, the mechanical lateral distal tibial angle (mLDTA), can reflect the relative relationship between the talus and tibia, and the ankle joint line orientation (AJLO) reflects the relationship between the talus and the horizontal plane. Changes in the force line of the lower limbs will cause changes in mLDTA and AJLO, which is one of the important reasons for ankle symptoms caused by KOA surgery.

[0012] In view of this, the field urgently needs a technical solution that can automatically and accurately measure the coronal morphological parameters of the lower limbs. Summary of the invention

[0013] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a system, method, terminal and medium for automatic measurement of lower limb coronal parameters based on deep learning, so as to solve the technical problems of large measurement errors and complex operation in traditional lower limb coronal parameter measurement methods.

[0014] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a deep learning-based automatic measurement system for lower limb coronal parameters, comprising: an image acquisition module for acquiring lower limb X-ray images of patients; an image preprocessing module for preprocessing the acquired lower limb X-ray images; the preprocessing includes image denoising and / or image enhancement; a deep learning model module for extracting and classifying features using a deep learning model and outputting anatomical landmarks and femoral anatomical axes of the lower limb X-ray images; a parameter calculation module for performing visual data processing and rendering to convert the anatomical landmarks and femoral anatomical axes of the lower limb X-ray images into corresponding lower limb coronal morphological parameters.

[0015] In some embodiments of the first aspect of the present application, the deep learning model module includes a marker point detection module and an image segmentation module; the marker point detection module is used to detect anatomical marker points in the X-ray image; the image segmentation module is used to segment the cortical bone and medullary cavity parts in the input image and calculate the femoral anatomical axis through a fitting algorithm.

[0016] In some embodiments of the first aspect of the present application, the marker detection module performs target detection based on a multi-cascade detection algorithm of image spatial features.

[0017] In some embodiments of the first aspect of the present application, the marker point detection module performs target detection based on a multi-cascade detection algorithm of image spatial features, which includes the following: counting the spatial coordinate information of all anatomical key points and performing cluster analysis on them, generating corresponding regional bounding box information based on the clustered spatial coordinate point set; training a coarse positioning model based on the yolov5 model based on the regional bounding box information; using the coarse positioning model to extract the spatial features of the X-ray image and identify each of the spatial features separately; and converting the identified spatial features back to the original image coordinate system through coordinate conversion.

[0018] In some embodiments of the first aspect of the present application, the image segmentation module first uses the U-net segmentation network to segment the medullary cavity, and then calculates the femoral anatomical axis based on the segmented image; wherein the femoral anatomical axis includes the proximal axis of the lower limb, the mid-axis of the lower limb and the distal axis of the lower limb.

[0019] In some embodiments of the first aspect of the present application, the U-net segmentation network segmentation process includes the following: according to the femur image of the anatomical region and the anatomical landmarks identified thereon, the femur is corrected to be perpendicular to the image coordinate system, and input into the U-net segmentation network to obtain segmented images of the cortical bone, medullary cavity and background respectively; the segmented image is processed row by row to extract the mutation inner and outer points of the cortical bone and medullary cavity according to the segmented image value, thereby obtaining the medullary cavity midpoint of each row of pixels; all medullary cavity midpoint axes are divided into four segments, and each group of spatial coordinates is fitted to obtain the most approximate axis of each group of medullary cavity focus.

[0020] In some embodiments of the first aspect of the present application, the segmentation model is further provided with a spatial attention mechanism module and a channel attention mechanism module in the deep learning-based feature extraction module.

[0021] In some embodiments of the first aspect of the present application, the spatial attention mechanism module is used to perform the following steps: for each input image of size W*H*C, arbitrarily divide it into m*n input blocks along the two axes of width and height; wherein W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; m and n are natural numbers greater than or equal to 1; each input block becomes a feature vector of size (1*m) / (1*n) after passing through a fully connected layer; each feature vector is calculated similarity with each other to obtain the correlation relationship between each input block and the complete input image, thereby obtaining m*n similarity matrices of size m*n; the similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0022] In some embodiments of the first aspect of the present application, the channel attention mechanism module is used to perform the following steps: for each input image of size W*H*C, divide it into Q input blocks along the channel; wherein W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; Q is a natural number greater than or equal to 1; each input block becomes a 1*Q feature vector after passing through a fully connected layer; each feature vector is similarity calculated with each other to obtain the correlation relationship between the channel and all channels, thereby obtaining Q similarity matrices of size 1*Q; the similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0023] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a method for automatic measurement of lower limb coronal parameters based on deep learning, comprising: acquiring a patient's lower limb X-ray image and preprocessing the acquired lower limb X-ray image; the preprocessing includes image denoising and / or image enhancement; using a deep learning model to extract and classify features and output the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image; performing visual data processing and rendering to convert the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image into corresponding lower limb coronal morphological parameters.

[0024] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the method when executed by a processor.

[0025] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory so that the terminal executes the method.

[0026] As described above, the system, method, terminal and medium for automatic measurement of lower limb coronal parameters based on deep learning of the present application have the following beneficial effects:

[0027] (1) Improve measurement accuracy and efficiency: Traditional full-length X-ray film evaluation of both lower limbs requires professional doctors to manually measure multiple parameters, which is time-consuming, labor-intensive and subjective. The automated lower limb parameter measurement software of the present invention can accurately measure multiple parameters and quickly output evaluation results, greatly improving the accuracy and efficiency of measurement.

[0028] (2) Reducing labor costs: Traditional manual measurement requires doctors to spend a lot of time and energy to operate, while the automated measurement software of the present invention can complete the measurement without human intervention, thereby reducing labor costs.

[0029] (3) Providing convenient services: The automated lower limb parameter measurement software of the present invention can provide outpatients with fast, convenient and reliable lower limb parameter measurement services, allowing patients to undergo medical examinations more conveniently.

[0030] (4) Application in large-scale epidemiological studies: The automated lower limb parameter measurement software of the present invention can also be used as an early screening tool in large-scale epidemiological studies to improve the efficiency and accuracy of the research and provide strong support for the early detection and prevention of diseases.

[0031] In summary, the automated lower limb parameter measurement software of the present invention has broad application prospects in clinical applications and large-scale epidemiological studies. It can greatly improve the accuracy and efficiency of measurement, reduce labor costs, provide convenient services for patients, and provide strong support for early detection and prevention of diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Shown is a structural schematic diagram of a system for automatic measurement of lower limb coronal parameters based on deep learning in one embodiment of the present application.

[0033] Figure 2 Shown is a flowchart of a multi-cascade detection algorithm based on image spatial features in one embodiment of the present application.

[0034] Figure 3 Shown is a schematic diagram of the spatial feature recognition effect in one embodiment of the present application.

[0035] Figure 4 Shown is a schematic diagram of the process of using the U-net segmentation network to perform image segmentation in one embodiment of the present application.

[0036] Figure 5 It is a schematic diagram showing the effect of dividing the axis of the midpoint of the medullary cavity into four sections in one embodiment of the present application.

[0037] Figure 6 Shown is a schematic diagram of the effects of VTK rendering, display and interaction in one embodiment of the present application.

[0038] Figure 7 Shown is a flowchart of a method for automatic measurement of lower limb coronal parameters based on deep learning in one embodiment of the present application.

[0039] Figure 8 Shown is a flowchart of a method for automatic measurement of lower limb coronal parameters based on deep learning in one embodiment of the present application.

[0040] Fig. 9 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0043] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0044] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0045] In order to solve the problems in the above-mentioned background technology, the present invention provides a system, method, terminal and medium for automatic measurement of lower limb coronal parameters based on deep learning, which can accurately measure the coronal morphological parameters of the lower limbs, including overall parameters and local parameters. Through this automatic measurement scheme, doctors can more accurately evaluate the coronal deformity of the lower limbs of patients with knee joint lesions, so as to provide personalized treatment plans in the end. The present invention can quickly and accurately measure the coronal morphological parameters of the lower limbs by learning massive image data, avoiding the problems of large measurement errors and complex operations in traditional measurements.

[0046] At the same time, in order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution in the embodiment of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the invention.

[0047] The embodiments of the present invention provide a method for automatically measuring lower limb coronal parameters based on deep learning, a system for automatically measuring lower limb coronal parameters based on deep learning, and a storage medium storing an executable program for implementing the method for automatically measuring lower limb coronal parameters based on deep learning. As for the implementation of the method for automatically measuring lower limb coronal parameters based on deep learning, the embodiments of the present invention will illustrate an exemplary implementation scenario of automatically measuring lower limb coronal parameters based on deep learning.

[0048] like Figure 1 As shown, a schematic diagram of the structure of a system for automatically measuring lower limb coronal parameters based on deep learning in an embodiment of the present invention is shown. The system for automatically measuring lower limb coronal parameters based on deep learning in this embodiment mainly includes the following modules: an image acquisition module 11, an image preprocessing module 12, a deep learning model module 13 and a parameter calculation module 14. The functions of each module and the interaction process between modules will be further described below in conjunction with specific embodiments.

[0049] The image acquisition module 11 is used to acquire X-ray images of the patient's lower limbs. For example, the lower limb X-ray images can be acquired by ultrasound equipment, CT imaging equipment, DAS imaging equipment, MR imaging equipment, SPETC imaging equipment or PTE imaging equipment.

[0050] The image preprocessing module 12 is used to preprocess the collected lower limb X-ray images; the preprocessing includes image noise reduction and / or image enhancement.

[0051] Image denoising refers to the process of reducing noise in digital images. Noise is an important cause of image interference. In practical applications, an image may contain various noises, which may be generated during transmission or during quantization, such as additive noise, multiplicative noise, quantization noise, etc. In the embodiment of the present invention, image denoising can be performed by any one of a mean filter, an adaptive Wiener filter, a median filter, a morphological noise filter, and a wavelet denoising filter.

[0052] Image enhancement refers to the purposeful enhancement of the overall or local characteristics of an image, making an originally unclear image clear or emphasizing certain features of interest, expanding the differences between the features of different objects in the image, and suppressing features of no interest, so as to improve image quality, enrich the amount of information, and enhance image interpretation and recognition effects.

[0053] The deep learning model module 13 is used to extract and classify features using a deep learning model and output feature values. For example, a convolutional neural network (CNN) can be used for feature extraction and classification. A convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure. It has representation learning capabilities and can perform translation-invariant classification of input information according to its hierarchical structure.

[0054] In an embodiment of the present invention, the deep learning model module 13 specifically includes a marker point detection module 131 and an image segmentation module 132; wherein the marker point detection module 131 is used to detect anatomical marker points in the X-ray image; the image segmentation module 132 is used to segment the cortical bone and medullary cavity parts in the input image and calculate the femoral anatomical axis through a fitting algorithm.

[0055] In some examples, the anatomical landmarks detected by the landmark detection module 131 include, but are not limited to, the center of the femoral head, the center of the knee joint, the greater trochanter, etc.

[0056] In some examples, the marker detection module 131 may use YOLO V5 as a target detection module. Compared with other detection modules, the target detection modules of the YOLO series have the characteristics of fast speed and strong versatility. Among them, YOLO (You Only Look Once) is a target detection model. Different from the two-stage target detection of the R-CNN series model that first extracts image features, the YOLO model can perform end-to-end detection of the target in just one step.

[0057] Preferably, the marker point detection module 131 in the embodiment of the present invention performs target detection based on a multi-cascade detection algorithm of image spatial features. This is because: in the lower limb X-ray detection task, it is necessary to detect some anatomical marker points whose features are not obvious and whose target accounts for a very small proportion of the overall image. These points usually have relatively obvious image features, but contain obvious spatial features, such as the medial and lateral concave points of the tibial plateau, the medial and lateral points of the distal femoral condylar line, etc.

[0058] Furthermore, the execution steps of the multi-cascade detection algorithm based on image spatial features are as follows: Figure 2 As shown:

[0059] Step S21: Count the spatial coordinate information of all anatomical key points and perform cluster analysis on them, and generate corresponding regional bounding box information based on the clustered spatial coordinate point set.

[0060] Specifically, all anatomical key points are counted, including but not limited to: anterior superior iliac spine, iliac spine, greater trochanter of femur, lesser trochanter of femur, medial femoral condyle, lateral femoral condyle, medial femoral epicondyle, lateral femoral epicondyle, fibular head, medial ankle condyle, lateral ankle condyle and other bony landmarks. Cluster analysis is performed on all the anatomical key points counted, for example, they can be divided into three groups with a total of six regions, namely, left hip region, right hip region, left knee region, right knee region, left ankle region and right ankle region.

[0061] Step S22: According to the region bounding box information, a coarse positioning model is trained based on the yolov5 model.

[0062] YOLO, short for You Only Look Once, is a series of single-stage deep learning-based object detectors that can detect objects at speeds faster than real time and with state-of-the-art accuracy. YOLOv5 is a deep learning model for object detection that uses the PyTorch framework and includes a variety of models of different sizes and accuracy, suitable for a variety of scenarios and devices.

[0063] Specifically, the process of training the coarse positioning model based on the yolov5 model includes the following:

[0064] First, prepare the dataset, including the annotated images and the corresponding label files. Second, clone the YOLOv5 repository and obtain the YOLOv5 code and pre-trained model from the software project hosting platform GitHub. Then, run the training and execute the training code to start model training. You can choose to use GPU or CPU for training according to your needs. Finally, check the performance and compare the mAP (mean Average Precision), FPS (Frames Per Second) and inference time of different models to evaluate the training results.

[0065] Step S23: Use the coarse positioning model to extract the spatial features of the X-ray image and identify each of the spatial features respectively, for example Figure 3 The white rectangular box in the middle represents the identified spatial features, and the spatial features identified from top to bottom are the acetabulum region, knee joint region, and ankle joint region.

[0066] Step S24: convert the identified spatial features back to the original image coordinate system through coordinate conversion.

[0067] Specifically, the result of local image control price recognition is converted back to the original image coordinate system to complete the recognition.

[0068] At this point, through a set of multi-cascade detection algorithms based on image spatial features provided by an embodiment of the present invention, X-ray detection of some points whose image features are not obvious but contain obvious spatial features (such as the medial and lateral concave points of the tibial plateau, the medial and lateral points of the distal femoral ankle line, etc.) is achieved.

[0069] In the embodiment of the present invention, the image segmentation module 132 is used to segment the cortical bone and medullary cavity in the input image and calculate the femoral anatomical axis through a fitting algorithm. The femoral anatomical axis is divided into the following three sections: the proximal axis of the lower limb, the middle axis of the lower limb, and the distal axis of the lower limb; in the process of calculating the femoral anatomical axis, the segmentation model uses the inner wall of the medullary cavity as a reference, first uses a feature extraction module based on deep learning to segment the medullary cavity, and then calculates the medullary cavity axis based on the segmented image. It should be understood that the anatomical axis of the femur and tibia is the axis of their medullary cavity.

[0070] Preferably, the deep learning-based feature extraction module uses a U-net segmentation network, that is, the medullary cavity is first segmented using the U-net segmentation network, and then the medullary cavity axis is calculated based on the segmented image. The specific process is as follows: Figure 4 The steps shown include the following:

[0071] Step S41: According to the femoral image of the anatomical region and the anatomical landmarks identified thereon, the femur is rectified to be perpendicular to the image coordinate system, and input into the U-net segmentation network to obtain segmented images of the cortical bone, medullary cavity and background respectively.

[0072] Specifically, the femur image is extracted according to the anatomical region obtained by the marker point detection module, the femur is corrected to be perpendicular to the image coordinate system according to the anatomical marker points (such as from the greater trochanter to the center of the knee joint), and the image is input into the U-net segmentation network to obtain a segmented image of the cortical bone, medullary cavity and background.

[0073] It should be understood that the U-net segmentation network uses splicing to fuse deep and shallow features. The convolution of the U-net segmentation network does not use zero padding, so the size of the feature map changes after each convolution, and the size of the final output segmentation prediction result is smaller than the original image. The U-net segmentation network is essentially a fully convolutional neural network model. Its architecture is shaped like the letter U. The backbone of the network is divided into two symmetrical left and right parts. The left side is the feature extraction network (i.e., encoder). The original input image is downsampled four times through convolution-max pooling to obtain a four-level feature map; the right side is the feature fusion network (i.e., decoder). The feature maps of each level are fused with the feature maps obtained by deconvolution through jump connection; the last layer predicts the semantic map by calculating the loss with the label.

[0074] Step S42: Process the segmented image row by row to extract the inner and outer points of the cortical bone and the medullary cavity according to the segmented image values, thereby obtaining the midpoint of the medullary cavity of each row of pixels.

[0075] Step S43: Divide all medullary cavity midpoint axes into four segments, perform fitting on each group of spatial coordinates, and obtain the most approximate axis of each group of medullary cavity points.

[0076] For example, Figure 5 For example, the midpoint axis of the medullary cavity is divided into four equal sections, and the figure shows the four equal division points of the femoral axis and the midpoint of the femoral axis respectively.

[0077] In some examples, each set of spatial coordinates can be fitted by a fitting algorithm, which refers to finding a function or curve through a given set of data points so that the function / curve can pass through as many data points as possible. The fitting algorithm in the embodiments of the present invention includes, but is not limited to, a linear regression fitting algorithm, a polynomial fitting algorithm, an interpolation fitting algorithm, a spline curve fitting algorithm, a parameter fitting algorithm, etc.

[0078] It should be noted that the embodiments of the present invention can not only perform calculation and analysis on the axis, but also analyze the overall structure of the femur, such as curvature, medullary cavity morphology change trend, etc., and can also be applied to scenarios with similar image features, and the present invention is not limited to this.

[0079] Preferably, the segmentation model is further provided with a spatial attention mechanism module and a channel attention mechanism module in the deep learning-based feature extraction module.

[0080] The spatial attention mechanism originates from the study of human vision. In cognitive science, due to the bottleneck of information processing, humans selectively focus on part of all information while ignoring other visible information. The above mechanism is usually called the attention mechanism.

[0081] In computer vision, the channel attention mechanism pays more attention to the relationship between channels in the feature map, while ordinary convolution will perform channel fusion on channels, such as SENet, GSOP-Net, etc. In convolutional neural networks, convolution operations pay more attention to the receptive field, and the default channel is the fusion of all channels, while the channel attention mechanism focuses on the channels and can learn the weights between different channels.

[0082] In an embodiment of the present invention, the spatial attention mechanism module is used to perform the following steps:

[0083] Step A1: For each input image of size W*H*C, arbitrarily divide it into m*n input blocks along the two-axis directions of width and height; where W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; m and n are natural numbers greater than or equal to 1.

[0084] Step A2: Each input block is transformed into a feature vector of size (1*m) / (1*n) after passing through a fully connected layer.

[0085] Step A3: Calculate the similarity between each feature vector to obtain the correlation between each input block and the complete input image, thereby obtaining m*n similarity matrices of size m*n.

[0086] Step A4: assigning the similarity matrix to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0087] In an embodiment of the present invention, the channel attention mechanism module is used to perform the following steps:

[0088] Step B1: For each input image of size W*H*C, divide it into Q input blocks along the channel; wherein W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; and Q is a natural number greater than or equal to 1.

[0089] Step B2: Each input block is converted into a 1*Q feature vector after passing through a fully connected layer.

[0090] Step B3: perform similarity calculation (such as vector product) between each feature vector to obtain the correlation relationship between the channel and all channels, thereby obtaining Q similarity matrices with a size of 1*Q.

[0091] Step B4: assigning the similarity matrix to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0092] It is worth noting that the above-mentioned spatial attention mechanism module and channel attention mechanism module will not change the size of input and output in the algorithm application, so the two attention mechanism modules can be added at the appropriate stage in the design of any image feature extraction module.

[0093] In an embodiment of the present invention, the parameter calculation module 14 uses the open source software library VTK (Visualization toolkit) to perform visual data processing and rendering to convert the feature values ​​output by the deep learning model module 13 into corresponding coronal morphological parameters of the lower limbs.

[0094] Specifically, the parameter calculation module 14 selects valuable combined point sequences based on the anatomical landmarks and femoral anatomical axis obtained by the deep learning model module 13, analyzes and calculates the Euclidean distance, spatial angle, and bilateral difference, obtains the corresponding analysis data, and renders and interacts with it through VTK. The effect can be referred to Figure 6 shown.

[0095] It should be noted that: when the automatic measurement system of lower limb coronal parameters based on deep learning provided in the above embodiment performs automatic measurement of lower limb coronal parameters based on deep learning, only the division of the above-mentioned program modules is used as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the system is divided into different program modules to complete all or part of the processing described above.

[0096] like Figure 7 As shown, a flowchart of a method for automatic measurement of lower limb coronal parameters based on deep learning in an embodiment of the present invention is shown. The method in the embodiment of the present invention mainly includes the following steps:

[0097] Step S71: Acquire the patient's lower limb X-ray image and pre-process the acquired lower limb X-ray image; the pre-processing includes image noise reduction and / or image enhancement.

[0098] Step S72: Use a deep learning model to extract and classify features and output the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image.

[0099] Step S73: Perform visual data processing and rendering to convert the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image into corresponding coronal morphological parameters of the lower limb.

[0100] In some examples, the method includes detecting anatomical landmarks in an X-ray image based on a landmark detection algorithm; and segmenting the cortical bone and medullary cavity in the input image based on an image segmentation model and calculating the femoral anatomical axis through a fitting algorithm.

[0101] In some examples, the method includes performing target detection using a multi-cascade detection algorithm based on image spatial features.

[0102] Furthermore, the multi-cascade detection algorithm based on image spatial features performs target detection, and its process includes: counting the spatial coordinate information of all anatomical key points and performing cluster analysis on them, generating corresponding regional bounding box information according to the clustered spatial coordinate point set; training a coarse positioning model based on the yolov5 model according to the regional bounding box information; using the coarse positioning model to extract the spatial features of the X-ray image and identify each of the spatial features respectively; and converting the identified spatial features back to the original image coordinate system through coordinate conversion.

[0103] In some examples, the image segmentation process includes: first using a U-net segmentation network to segment the medullary cavity, and then calculating the femoral anatomical axis based on the segmented image; wherein the femoral anatomical axis includes the proximal axis of the lower limb, the mid-axis of the lower limb, and the distal axis of the lower limb.

[0104] Furthermore, the U-net segmentation network segmentation process includes the following: according to the femur image of the anatomical region and the anatomical landmarks identified thereon, the femur is corrected to be perpendicular to the image coordinate system, and input into the U-net segmentation network to obtain the segmented images of the cortical bone, the medullary cavity and the background respectively; the segmented image is processed row by row to extract the mutation inner and outer points of the cortical bone and the medullary cavity according to the segmented image value, so as to obtain the medullary cavity midpoint of each row of pixels; all the medullary cavity midpoint axes are divided into four segments, and each group of spatial coordinates is fitted to obtain the most approximate axis of each group of medullary cavity focus.

[0105] In some examples, the segmentation model adds a spatial attention mechanism and a channel attention mechanism in the deep learning-based feature extraction process.

[0106] Furthermore, the process of adding the spatial attention mechanism includes the following: for each input image of size W*H*C, it is arbitrarily divided into m*n input blocks along the two axes of width and height; wherein W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; m and n are natural numbers greater than or equal to 1; each input block becomes a feature vector of size (1*m) / (1*n) after passing through a fully connected layer; each feature vector is similarity calculated with each other to obtain the correlation relationship between each input block and the complete input image, thereby obtaining m*n similarity matrices of size m*n; the similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0107] In some examples, the process of adding the channel attention mechanism includes the following: for each input image of size W*H*C, divide it into Q input blocks along the channel; wherein W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; Q is a natural number greater than or equal to 1; each input block becomes a 1*Q feature vector after passing through a fully connected layer; each feature vector is similarity calculated with each other to obtain the correlation relationship between the channel and all channels, thereby obtaining Q similarity matrices of size 1*Q; the similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

[0108] In order to facilitate those skilled in the art to better understand the technical solution of the present invention, Figure 8 The process of an automatic measurement method of lower limb coronal parameters based on deep learning is explained.

[0109] exist Figure 8 In the method, starting from the original image, the clustering markers are counted, and the corresponding results are output as regional clustering bounding boxes. The regional clustering bounding boxes are used as labels to train the regional coarse positioning recognition model. The trained regional coarse positioning recognition model is used for anatomical recognition, and the corresponding acetabular regional anatomical recognition, knee joint regional anatomical recognition, and ankle joint regional anatomical recognition are obtained respectively. Subsequently, the femur image is extracted, the medullary cavity femoral segmentation model is input, and the medullary cavity axis fitting algorithm is used to fit the medullary cavity axis point. Finally, VTK software is used to render based on the recognition results of the medullary cavity axis point and the anatomical marker points of the lower limb image, so as to obtain the lower limb parameters on the full-length X-ray film of both lower limbs.

[0110] It should be noted that the method for automatic measurement of lower limb coronal parameters based on deep learning provided in the above embodiment and the embodiment of automatic measurement system for lower limb coronal parameters based on deep learning belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0111] The method for automatic measurement of lower limb coronal parameters based on deep learning provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Fig. 9, is an optional hardware structure diagram of an electronic terminal 900 provided in an embodiment of the present invention. The terminal 900 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The electronic terminal 900 includes: at least one processor 901, a memory 902, at least one network interface 904 and a user interface 906. The various components in the device are coupled together through a bus system 905. It can be understood that the bus system 905 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 905 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Fig. 9 In the specification, various buses are labeled as bus systems.

[0112] The user interface 906 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0113] It is understood that the memory 902 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0114] The memory 902 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 900. Examples of these data include: any executable program for operating on the electronic terminal 900, such as an operating system 9021 and an application 9022; the operating system 9021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 9022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The method for automatically measuring lower limb coronal parameters based on deep learning provided in the embodiment of the present invention may be included in the application 9022.

[0115] The method disclosed in the above embodiment of the present invention can be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 901 or the instruction in the form of software. The above processor 901 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 901 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0116] In an exemplary embodiment, the electronic terminal 900 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0117] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0118] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0119] In summary, the present application provides a system, method, terminal and medium for automatic measurement of lower limb coronal parameters based on deep learning. The present application improves the accuracy and efficiency of measurement: the traditional full-length X-ray film evaluation of both lower limbs requires professional doctors to manually measure multiple parameters, which is time-consuming, labor-intensive and subjective, while the automated lower limb parameter measurement software of the present invention can accurately measure multiple parameters and quickly output the evaluation results, greatly improving the accuracy and efficiency of measurement; reducing labor costs: traditional manual measurement requires doctors to spend a lot of time and energy to operate, while the automated measurement software of the present invention can complete the measurement without manual intervention, thereby reducing labor costs; providing convenient services: the automated lower limb parameter measurement software of the present invention can provide outpatients with fast, convenient and reliable lower limb parameter measurement services, so that patients can more conveniently undergo medical examinations; applied to large-scale epidemiological studies: the automated lower limb parameter measurement software of the present invention can also be used as an early screening tool, applied to large-scale epidemiological studies, improve the efficiency and accuracy of the research, and provide strong support for the early detection and prevention of diseases. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0120] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A system for automatic measurement of lower limb coronal parameters based on deep learning, characterized in that: include: An image acquisition module, used for acquiring X-ray images of the patient's lower limbs; An image preprocessing module is used to preprocess the collected lower limb X-ray images; The preprocessing includes image noise reduction and / or image enhancement; A deep learning model module, used for extracting and classifying features using a deep learning model and outputting anatomical landmarks and femoral anatomical axes of the lower limb X-ray image; The parameter calculation module is used for visual data processing and rendering to convert the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image into corresponding coronal morphological parameters of the lower limb.

2. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 1 is characterized in that: The deep learning model module includes a marker point detection module and an image segmentation module; the marker point detection module is used to detect anatomical marker points in the X-ray image; the image segmentation module is used to segment the cortical bone and medullary cavity parts in the input image and calculate the femoral anatomical axis through a fitting algorithm.

3. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 2 is characterized in that: The marker detection module performs target detection based on a multi-cascade detection algorithm of image spatial features.

4. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 3 is characterized in that: The marker detection module performs target detection based on a multi-cascade detection algorithm of image spatial features, which includes the following: Count the spatial coordinate information of all anatomical key points and perform cluster analysis on them, and generate the corresponding regional bounding box information based on the clustered spatial coordinate point set; According to the bounding box information of the region, a coarse positioning model is trained based on the yolov5 model; Extracting spatial features of the X-ray image using the coarse positioning model and identifying each of the spatial features respectively; The identified spatial features are converted back to the original image coordinate system through coordinate transformation.

5. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 2 is characterized in that: The image segmentation module first uses the U-net segmentation network to segment the medullary cavity, and then calculates the femoral anatomical axis based on the segmented image; wherein the femoral anatomical axis includes the proximal axis of the lower limb, the mid-end axis of the lower limb and the distal axis of the lower limb.

6. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 5 is characterized in that: The process of U-net segmentation network segmentation includes the following: According to the femoral image of the anatomical region and the anatomical landmarks identified thereon, the femur is rectified to be perpendicular to the image coordinate system and input into the U-net segmentation network to obtain the segmented images of the cortical bone, medullary cavity and background respectively; The segmented image is processed line by line to extract the inner and outer points of the cortical bone and the medullary cavity according to the segmented image value, thereby obtaining the medullary cavity midpoint of each line of pixels; The axes of all medullary cavity midpoints were divided into four segments, and each group of spatial coordinates was fitted to obtain the most approximate axis of each group of medullary cavity points.

7. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 2, characterized in that: The segmentation model is further provided with a spatial attention mechanism module and a channel attention mechanism module in the deep learning-based feature extraction module.

8. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 7, characterized in that: The spatial attention mechanism module is used to perform the following steps: For each input image of size W*H*C, it is arbitrarily divided into m*n input blocks along the two axes of width and height; where W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; m and n are natural numbers greater than or equal to 1; Each input block becomes a feature vector of size (1*m) / (1*n) after passing through a fully connected layer; The similarity between each feature vector is calculated to obtain the correlation between each input block and the complete input image, thereby obtaining m*n similarity matrices of size m*n; The similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

9. The automatic measurement system for lower limb coronal parameters based on deep learning according to claim 7, characterized in that: The channel attention mechanism module is used to perform the following steps: For each input image of size W*H*C, divide it into Q input blocks along the channel; where W represents the width of the input image; H represents the height of the input image; C represents the number of channels of the input image; Q is a natural number greater than or equal to 1; Each input block becomes a 1*Q feature vector after passing through a fully connected layer; The similarity between each feature vector is calculated to obtain the correlation between the channel and all channels, and Q similarity matrices with a size of 1*Q are obtained; The similarity matrix is ​​assigned to the input image; the assignment method includes any one of corresponding addition, direct superposition and corresponding multiplication.

10. A method for automatic measurement of lower limb coronal parameters based on deep learning, characterized in that: include: Acquire the patient's lower limb X-ray images and pre-process the acquired lower limb X-ray images; The preprocessing includes image noise reduction and / or image enhancement; Using a deep learning model to extract and classify features and output anatomical landmarks and femoral anatomical axis of the lower limb X-ray image; Visual data processing and rendering are performed to convert the anatomical landmarks and femoral anatomical axis of the lower limb X-ray image into corresponding coronal morphological parameters of the lower limb.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically measuring lower limb coronal parameters based on deep learning as described in claim 10 is implemented.

12. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the terminal executes the automatic measurement method of lower limb coronal parameters based on deep learning as described in claim 10.

Images