Segmentation and key point detection method and system suitable for lower limb CT full-length film, terminal and medium

Abstract

The invention provides a segmentation and key point detection method and system suitable for a lower limb CT full-length film, a terminal and a medium, and aims to construct a full-process segmentation and key point detection model for the lower limb CT full-length film based on nnU-Net and nnDesection, the model can automatically process original lower limb CT full-length film data, segmentation data and key point labeling data, and the segmentation and key point detection efficiency is improved. And the functions of automatically segmenting each bony structure of the lower limb and automatically marking each key point of the lower limb are realized. According to the method, convenient operation is provided, the accuracy and efficiency of tasks are improved, the accuracy of key point detection is improved, codes are optimized, various defects in the prior art are effectively overcome, and the method has high industrial utilization value.

Description

Technical Field

[0001] The present application relates to the technical field of image analysis and processing, and in particular to a segmentation and key point detection method, system, terminal and medium applicable to full-length CT films of lower limbs. Background Art

[0002] Osteoarthritis (OA) is one of the joint diseases that affects human health. Its incidence rate in my country is about 18%, and its impact on the population increases with age, especially on people over 50 years old. It is one of the main causes of pain and disability in the elderly. In addition to causing pain and reducing the quality of life for patients, osteoarthritis also imposes a heavy economic burden on society, becoming a major problem in social development.

[0003] At present, the pathogenesis of knee osteoarthritis (KOA) is unclear, and there is no specific treatment: non-surgical treatments such as drugs and physical therapy can only relieve joint pain to a certain extent, but they cannot stop or reverse the development of KOA, which eventually leads to patients undergoing knee replacement surgery; but knee replacement surgery is expensive, has large intraoperative trauma, and causes severe pain to patients after surgery. In addition, artificial joints cannot completely replace and restore knee joint function, and "worry-free knees" after knee replacement surgery are still a luxury. Therefore, it is urgent to explore new treatment methods, intervene in the progression of early and mid-stage KOA, and delay or even avoid artificial knee replacement surgery.

[0004] High tibial osteotomy (HTO) is one of the effective means to treat KOA. HTO transfers the force line from the inflamed and worn medial compartment of the knee joint to the relatively normal lateral compartment through proximal tibia osteotomy, thereby relieving arthritis symptoms and prolonging the life of the knee joint. HTO can effectively prevent cartilage wear, relieve pain symptoms, and even allow the worn cartilage and injured meniscus to self-repair, thereby maintaining knee joint function and even restoring the ability of some patients to do heavy physical labor, thereby prolonging the natural life of the patient's knee joint. However, the efficacy of HTO still has shortcomings such as large individual differences, low satisfaction, and high complication rate. Among the many factors that affect the efficacy, how to achieve precise control of the force line of the lower limb is a key issue to be solved in high tibial osteotomy.

[0005] HTO surgical planning is usually based on lower limb X-rays and CT images. At present, there are many defects in the surgical planning and intraoperative force line judgment of HTO, which seriously limits the surgeon's precise control of the lower limb force line. For example, 3D printed personalized osteotomy guides can improve the accuracy of high tibial osteotomy to a certain extent, but its surgical planning based on supine CT cannot truly judge the patient's lower limb force line in standing position, and the incidence of postoperative undercorrection or overcorrection of force line can be as high as 20%. In detail, when reading the full-length film of the lower limb on the traditional supine CT, there are often problems such as unintuitive tomographic images and difficulty in achieving three-dimensional spatial measurement, which easily leads to large preoperative planning errors. Therefore, artificial intelligence technologies such as deep learning should be combined to develop intelligent modeling algorithms for the full-length CT films of the lower limbs, automatically segment and reconstruct the bony structure of the lower limbs, and identify three-dimensional bony landmarks, so as to provide intuitive preoperative model references for HTO and automatically calculate the spatial parameters related to surgery.

[0006] Deep learning is a machine learning method that aims to imitate the working mode of human brain neural network and process and interpret complex data by constructing and training multi-level neural networks. The core idea of ​​deep learning is to learn the abstract representation of data through multiple processing levels, so as to realize automatic feature extraction and pattern recognition. In the field of medical imaging, deep learning has achieved great success and has performed well in tasks such as segmentation, key point detection, and registration. nnU-Net proposed by Isensee et al. is a full-process automated segmentation framework in the field of medical image segmentation. Users only need to provide the original image and segmentation annotations, and nnU-Net can complete the automated model training process. Users do not need to design other processes such as preprocessing, postprocessing, and hyperparameter design. nnDetection proposed by Baumgartner et al. is a full-process automated framework for target detection in medical datasets. The network structure uses the Retina U-Net model. Data preprocessing, postprocessing, data enhancement and other links follow the idea of ​​nnU-Net. It also extracts data fingerprints and pipeline fingerprints from the input dataset. There is no need for users to manually adjust parameters, and the network training can be completed automatically, and the location of the target box and object classification are finally output.

[0007] In the prior art, the key point detection method for the full-length CT film of the lower limbs is usually still based on its 2D slices, and the segmentation and annotation are not integrated, which makes the process complicated and inefficient. In terms of intelligent segmentation and key point detection of the full-length CT film of the lower limbs, there is no full-process lower limb CT full-length film segmentation and key point detection model based on artificial intelligence, which can perform key point detection on the full-length 3DCT film, and cannot automatically process the original full-length CT film data of the lower limbs, segmentation data and key point annotation data, and cannot realize the automatic segmentation of the bony structures of the lower limbs, and the automatic annotation function of the key points of the lower limbs. Summary of the invention

[0008] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a segmentation and key point detection method, system, terminal and medium suitable for lower limb CT full-length films, so as to solve the technical problems that the existing key point detection process for lower limb CT full-length films is complicated and inefficient.

[0009] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a segmentation and key point detection method suitable for lower limb full-length CT films, including: obtaining CT image data, segmentation annotation data and key point annotation data of lower limb case images, and preprocessing the CT image data and key point annotation data to convert them into a data format suitable for neural networks, thereby generating corresponding lower limb full-length CT films and their segmentation annotation data; based on the lower limb full-length CT films and their segmentation annotation data generated after preprocessing, using the nnU-Net segmentation model to perform model training to obtain a medical image segmentation model capable of performing bone structure segmentation on lower limb case images; based on the lower limb full-length CT films, segmentation annotation data and key point annotation data generated after preprocessing, using the nnDetection detection model to perform model training to obtain a key point detection model capable of performing key point detection on lower limb case images.

[0010] In some embodiments of the first aspect of the present application, the process of preprocessing the CT image data and key point annotation data includes: integrating the original DICOM data slices of each case into corresponding NIFTI format images, and converting the key point annotation data in JSON format into NIFTI format images.

[0011] In some embodiments of the first aspect of the present application, the process of converting key point annotation data in JSON format into an image in NIFTI format includes: reading the coordinate data of each key point in the world coordinate system from a file in JSON format, converting the world coordinate system to the corresponding pixel coordinate system based on affine transformation, and obtaining the coordinate data of each key point in the pixel coordinate system; reducing the length, width and height of each key point to a preset size; and creating key point annotations in NIFTI format based on the key point information in JSON format after coordinate conversion and size compression.

[0012] In some embodiments of the first aspect of the present application, it includes: using nnU-Net to perform model training to obtain an image segmentation model that can segment the full-length CT film of the lower limbs into corresponding pelvis, femur, patella, tibia, fibula, inner part of the femur, inner part of the tibia, and foot.

[0013] In some embodiments of the first aspect of the present application, after using the medical image segmentation model to segment the lower limb case images, the method also performs a step of adding connected domain post-processing to the segmentation results, which includes: extracting the parts of the femur, tibia, and fibula from the segmentation prediction results output by the medical image segmentation model, retaining the two largest connected domains of these three types of bone structures to remove discrete points.

[0014] In some embodiments of the first aspect of the present application, the method further includes improving the running process of nnU-Net by deleting a process pool module during the verification process.

[0015] In some embodiments of the first aspect of the present application, the method further performs the following before training the key point detection model: classifying and cropping the lower limb CT full-length film, segmentation annotation data and key point annotation data generated after preprocessing.

[0016] In some embodiments of the first aspect of the present application, the classification and cropping process includes: dividing the key point annotation data into three categories and the key points in each category do not overlap; two of the three categories of key points are all located at the end of the femur and below, and the other category of key points are all located at the end of the femur and above; let the original image height be h, then the image height h′ retained at the end of the femur and below is: Cut off the pixel points from 0 to h' on the z-axis of the original image; let the image height h" of the femur end and above be: Intercept the pixels from h″ to h on the z-axis of the original image.

[0017] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a segmentation and key point detection system suitable for full-length CT films of lower limbs, including: a preprocessing module, used to obtain CT image data, segmentation annotation data and key point annotation data of lower limb case images, and preprocess the CT image data and key point annotation data to convert them into a data format suitable for neural networks, thereby generating corresponding full-length CT films of lower limbs and their segmentation annotation data; an image segmentation module, used to use the nnU-Net segmentation model to perform model training based on the full-length CT films of lower limbs and their segmentation annotation data generated after preprocessing to obtain a medical image segmentation model capable of performing bone structure segmentation on lower limb case images; a key point detection module, used to use the nnDetection detection model to perform model training based on the full-length CT films of lower limbs, segmentation annotation data and key point annotation data generated after preprocessing to obtain a key point detection model capable of performing key point detection on lower limb case images.

[0018] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the segmentation and key point detection method applicable to full-length CT films of lower limbs.

[0019] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the segmentation and key point detection method applicable to full-length CT films of lower limbs.

[0020] As described above, the segmentation and key point detection method, system, terminal and medium of the present application applicable to the full-length CT film of the lower limbs have constructed an intelligent segmentation and key point detection system for the full-length CT film of the lower limbs, which can automatically process the original data of the hospital, and respectively train and reason the segmentation network and the key point detection network, and perform data post-processing and effect evaluation on the reasoning results, which has the following beneficial effects:

[0021] (1) Providing convenient operation: The present invention constructs an automated medical image preprocessing process, automatically processes the lower limb CT full-length films and their annotation data in the medical data set, converts the lower limb full-length film slices in DICOM format into NIfTI format CT full-length films, and converts the key point annotation data in JSON format into NIfTI format images through methods such as coordinate system conversion, target box volume reduction, grouping multiple key points, and filling key point information in an empty array. Through this preprocessing process, the original data from the hospital can be conveniently processed into a data format that can be processed and analyzed by a neural network, greatly improving the processing efficiency of the neural network for medical data.

[0022] (2) Improve the accuracy and efficiency of tasks: This paper uses the powerful medical segmentation architecture nnU-Net and the medical key point detection architecture nnDetection to build a segmentation and key point detection model suitable for full-length CT films of the lower limbs. This model can directly process 3D CT data for model training, avoiding the lengthy operation of converting data into 2D and then merging the results into 3D in traditional methods, greatly improving the efficiency and performance of segmentation and key point detection of 3D medical data.

[0023] (3) Improving the accuracy of key point detection: The present invention introduces segmentation annotation reference into the key point detection model, and uses semantic segmentation information to guide the orientation and category of key points. Compared with the traditional key point detection model without segmentation information, the accuracy of key point detection is greatly improved.

[0024] (4) Code optimization: The present invention solves the problem of nnU-Net model training stuck due to Python process pool deadlock, and optimizes the running continuity of nnU-Net on the server.

[0025] In summary, the present invention uses deep learning technology to construct a new segmentation and key point detection model for lower limb CT full-length films. The research results of the present invention will provide effective assistance for accurate preoperative planning and assist in the design of HTO surgical guides. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Shown is a flow chart of a method for segmentation and key point detection applicable to full-length CT films of lower limbs in one embodiment of the present application.

[0027] Figure 2 Shown is a flowchart of converting key point annotation data in JSON format into NIFTI format images in one embodiment of the present application.

[0028] Figure 3 Shown are schematic diagrams of target frames in different coordinate systems in one embodiment of the present application.

[0029] Figure 4A Shown is a horizontal plane schematic diagram of a key point annotated image converted into NIFTI format in one embodiment of the present application.

[0030] Figure 4B Shown is a sagittal schematic diagram of a key point annotated image converted into NIFTI format in one embodiment of the present application.

[0031] Figure 4C Shown is a coronal plane schematic diagram of a key point annotated image converted into NIFTI format in one embodiment of the present application.

[0032] Figure 5A Shown is a schematic diagram of a segmented image without connected domain post-processing in one embodiment of the present application.

[0033] Figure 5B Shown is a schematic diagram of a segmented image after connected domain post-processing in one embodiment of the present application.

[0034] Fig. 6A Shown is a schematic diagram of key point prediction values ​​performed by a key point detection model in one embodiment of the present application.

[0035] Figure 6B Shown is a schematic diagram of the key point truth value performed by the key point detection model in one embodiment of the present application.

[0036] Figure 7Shown is a structural schematic diagram of a segmentation and key point detection system suitable for full-length CT films of lower limbs in one embodiment of the present application.

[0037] Figure 8 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0039] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0040] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0041] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0042] In order to solve the problems in the above-mentioned background technology, the present invention provides a segmentation and key point detection method, system, terminal and medium suitable for lower limb full-length CT films, aiming to build a full-process segmentation and key point detection model for lower limb full-length CT films based on nnU-Net and nnDetection. The model can automatically process the original lower limb full-length CT film data, segmentation data and key point annotation data, and realize the automated segmentation of various bony structures of the lower limbs, as well as the automated annotation function of various key points of the lower limbs.

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0044] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0045] <1> Full-length X-ray of lower limbs: used to examine lesions of the hip joint, femur, knee joint, etc. Full-length weight-bearing X-ray of both lower limbs is an AP X-ray of both lower limbs taken when the patient is standing.

[0046] <2> DICOM (Digital Imaging and Communications in Medicine): Digital Imaging and Communications in Medicine is an international standard for medical images and related information. It defines a medical image format that can be used for data exchange with quality that meets clinical needs. DICOM data is usually a storage format for medical images, such as CT images and X-ray images.

[0047] <3> NIFTI (Neuroimaging Informatics Technology Initiative): An image storage format that adds directional information and left-right position information of the patient compared to the earlier Analyze format without the need for additional files.

[0048] <4> JSON (JavaScript Object Notation) is a lightweight data exchange format that uses a text format that is completely independent of the programming language to store and represent data. It is easy to read and write, easy to parse and generate by machines, and can effectively improve network transmission efficiency.

[0049] The embodiments of the present invention provide a segmentation and key point detection method applicable to a full-length CT image of a lower limb, a system applicable to the segmentation and key point detection method for a full-length CT image of a lower limb, and a storage medium storing an executable program for implementing the segmentation and key point detection method for a full-length CT image of a lower limb. As for the implementation of the segmentation and key point detection method applicable to a full-length CT image of a lower limb, the embodiments of the present invention will describe an exemplary implementation scenario of the segmentation and key point detection method applicable to a full-length CT image of a lower limb.

[0050] like Figure 1 As shown, a flow chart of a method for segmentation and key point detection of a full-length CT image of a lower limb in an embodiment of the present invention is shown. The method for segmentation and key point detection of a full-length CT image of a lower limb in this embodiment mainly includes the following steps:

[0051] Step S1: Acquire CT image data, segmentation annotation data and key point annotation data of lower limb case images, and pre-process the CT image data and key point annotation data to convert them into a data format suitable for a neural network, thereby generating a corresponding lower limb CT full-length film and its segmentation annotation data.

[0052] The above step S1 is a process of preprocessing data for the case, which includes: integrating the original DICOM data slices of each case into corresponding NIFTI format images, and converting the key point annotation data in JSON format into NIFTI format images.

[0053] It should be understood that DICOM (Digital Imaging and Communications in Medicine) refers to digital imaging and communications in medicine, and is an international standard for medical images and related information, which defines a medical image format that can be used for data exchange with quality that meets clinical needs. DICOM data is usually a medical image storage format, such as CT images and X-ray images.

[0054] Compared with the earlier Analyze format, the NIFTI (Neuroimaging Informatics Technology Initiative) format adds directional information and the patient's left-right position information without the need for additional files.

[0055] In some examples, the conversion of DICOM data format into NIFTI format can be completed by any of the following image conversion tools: dcm2niix conversion tool, diNIfTI conversion tool, dicom2NIfTI conversion tool, dcmstack conversion tool, mrconvert conversion tool, etc., which is not specifically limited in the embodiments of the present invention.

[0056] In some examples, the key point annotation data in JSON format is converted into NIFTI format images. The process is as follows Figure 2 The specific examples shown are as follows:

[0057] Step S11: read the coordinate data of each key point in the world coordinate system from the file in JSON format, convert the world coordinate system to the corresponding pixel coordinate system based on affine transformation, and obtain the coordinate data of each key point in the pixel coordinate system.

[0058] Specifically, the affine transformation process from the world coordinate system to the pixel coordinate system includes: transforming each segmentation point and key point from the world coordinate system to the corresponding camera coordinate system, then transforming from the camera coordinate system to the corresponding image coordinate system, and then transforming from the image coordinate system to the corresponding pixel coordinate system.

[0059] The conversion process from the world coordinate system to the camera coordinate system requires the use of the rotation matrix R and the translation matrix T. The specific conversion method is as follows:

[0060]

[0061] Among them, X c , Y c , Z c Represents the camera coordinate system; X w , Y w , Z w represents the world coordinate system; R represents the orthogonal unit rotation matrix, and t represents the three-dimensional translation vector.

[0062] The conversion process from the camera coordinate system to the image coordinate system is a central projection process, that is, the perspective relationship from the camera coordinate system to the image coordinate system, which is calculated using similar triangles. The calculation method is as follows:

[0063]

[0064] in, represents a point in the image coordinate system, represents the camera focal length matrix, Represents a point in the camera coordinate system.

[0065] The conversion process from the image coordinate system to the pixel coordinate system is a discretization process. In fact, the pixel coordinate system and the image coordinate system are both on the imaging plane, but their origins and measurement units are different. The origin of the image coordinate system is usually the midpoint of the imaging plane, and its unit is mm, which is a physical unit. The unit of the pixel coordinate system is pixel, and a pixel point is usually described by several rows and columns. Therefore, the conversion method from the image coordinate system to the pixel coordinate system is as follows:

[0066]

[0067]

[0068]

[0069] in, refers to a point in the pixel coordinate system, It refers to the point in the image coordinate system. dx and dy indicate how many mm each column and row represent respectively, that is, 1 pixel = dx mm.

[0070] Combining the above formulas (1) to (5), the complete formula for converting from the world coordinate system to the pixel coordinate system can be obtained as follows:

[0071]

[0072]

[0073] in, a x 、a y , u0, v0 represent the camera's internal parameters; K represents the camera's internal parameter matrix; M1 represents the camera's external parameter matrix; R and t represent the camera's external parameters; M represents the projection matrix.

[0074] Step S12: reducing the length, width and height of each key point to a preset size.

[0075] It is worth noting that since nnDetection does not accept overlapping key point inputs, in order to avoid overlapping key points, the embodiment of the present invention reduces the length, width and height of each key point to a preset size. For example, the experience-based preset size obtained by the embodiment of the present invention after repeated model tests is 14 voxels, that is, the length, width and height of each key point are reduced to 14 voxels. It should be understood that nnDetection is a self-configuring framework for 3D (volumetric) medical object detection, which can be applied to new data sets without manual intervention. Voxel is the abbreviation of volume pixel. The three-dimensional containing voxels can be expressed by three-dimensional rendering or extracting polygonal isosurfaces with given threshold contours. Voxel is the smallest unit of digital data in three-dimensional space segmentation.

[0076] Step S13: Create key point annotations in NIFTI format based on the key point information in JSON format after coordinate conversion and size compression.

[0077] For example, you can use the dcm2niix command to convert from JSON format to NIFTI format. The parameter to be filled in the dcm2niix command is the location of the DICOM folder to be converted. For example, dcm2niix~ / dicomdir will convert all DICOM files in the folder "dicomdir".

[0078] To facilitate understanding by those skilled in the art, the following description is given by taking compression to a size of 14 voxels as an example: Figure 3 The target frame on the far left is the target frame in the world coordinate system. Figure 3 The target box in the middle is the target box in the pixel coordinate system. Figure 3 The target box on the far right is the reduced target box. It can be seen that the length, width and height coordinates of the two diagonal corners of the reduced target box are (x, y, z) = (121, 228, 684) and (x, y, z) = (135, 242, 698) respectively. The length x, width y and height z differ by 24 voxels, which ensures that there is no overlap between key points.

[0079] In addition, the key point annotation images converted into NIFTI format are as follows Figures 4A to 4C As shown, Figure 4A It shows the horizontal plane. Figure 4B The sagittal plane is shown. Figure 4C The coronal plane is shown. The horizontal plane is also called the transverse plane, which is a plane parallel to the ground and divides the body into upper and lower parts. The sagittal plane is a plane that divides the body into left and right parts along the front-back direction of the human body. The coronal plane is also called the frontal plane, which is a plane that divides the body into front and back parts along the left-right direction of the human body.

[0080] Step S2: Based on the lower limb CT full-length film and its segmentation annotation data generated after preprocessing, the nnU-Net segmentation model is used for model training to obtain a medical image segmentation model that can perform bone structure segmentation on lower limb case images.

[0081] In an embodiment of the present invention, after model training using nnU-Net, an image segmentation model is obtained that can segment the full-length CT film of the lower limbs into the corresponding pelvis, femur, patella, tibia, fibula, inner part of the femur, inner part of the tibia, and foot.

[0082] It is worth noting that the nnU-Net segmentation model used in the embodiment of the present invention is different from the usual U-Net segmentation model. Codec networks represented by UNet are commonly used in medical image segmentation. Although image segmentation can be achieved, the complexity and diversity of medical images themselves also affect the generalization and versatility of the U-Net segmentation model, which is reflected in the following aspects: (1) There are large differences between medical images of various modalities, such as the size of the research cohort, image size and dimension, resolution and voxel intensity; (2) The semantic labels of segmentation are extremely unbalanced; compared with the normal tissue in the image, the lesion area generally occupies only a very small part, which results in an extreme difference between the voxel labels of normal tissue and the voxel labels of lesion tissue; (3) There are large differences in expert annotations between different image data, and the annotation results of some images are ambiguous; (4) Some data sets have obvious differences in attributes such as image geometry and shape, and the problems of slice misalignment and anisotropy are also very serious.

[0083] In view of this, nnU-Net (no new Net) pays more attention to the preprocessing of images, can automatically determine the image modality and perform corresponding normalization operations, and resample different image voxel spacings according to cubic spline interpolation. In terms of network structure, nnU-Net proposes three networks based on the original U-Net, namely 2D U-Net, 3DU-Net and two cascaded 3D U-Nets. The first of the two cascaded 3D U-Nets is used to generate coarse segmentation results, and the second is used to refine the coarse segmentation results. Compared with the original U-Net network, the nnU-Net network has made certain structural adjustments, including but not limited to: using padded convolution to achieve the same output and input shapes, using Leaky ReLU instead of ReLU, and using instance normalization instead of batch normalization. In addition, the nnU-Net network can automatically set hyperparameters, such as training batch size, image block size, number of downsampling, etc. All U-Net architectures have passed 5-fold cross validation, using cross entropy loss and Dice loss as loss functions during training. The optimizer can use Adam, and set a dynamic learning rate adjustment strategy. The online data enhancement strategy is also used during training. Since the nnU-Net network itself is already available, it will not be repeated here.

[0084] More preferably, the method provided in the embodiment of the present invention, after using the medical image segmentation model to segment the lower limb case images, also performs the step of adding connected domain post-processing to the segmentation results, and the specific contents include the following: extracting the parts of the femur, tibia, and fibula in the segmentation prediction results output by the medical image segmentation model, retaining the two largest connected domains of these three types of bone structures to remove discrete points.

[0085] It should be understood that the connected domain of an image refers to the area composed of pixels with the same pixel value and adjacent positions in the image. Extracting connected domains in binary images is the core of many automatic image analysis applications. Connected domain analysis is a technology based on image segmentation. It can connect two or more connected components in a visual way, thereby enhancing the features and details of the image, making it easier to extract objects in the image. It detects objects in the image by analyzing local and global structural features in the image, as well as information such as color space, thereby achieving segmentation of objects and their structural features.

[0086] In the original segmentation prediction results obtained based on the nnU-Net model, in addition to the correct predictions on the left and right sides of the human body, there are also discrete points of incorrect segmentation for the femur, tibia, and fibula. Therefore, the segmentation prediction results for each case retain the two largest connected domains of the three types of bone structures to achieve the effect of removing discrete points, and merge the bone structures after removing the discrete points into the same segmented image. After the connected domain processing, the erroneous points in the original image are removed, and the accuracy of the segmentation is improved. For example, Figure 5A and 5B Take this as an example to illustrate: Figure 5A is the case where it contains an erroneous segmentation point. Figure 5B This is the segmented image after removing the segmentation points. It can be seen that the image segmentation is more accurate after connected domain processing.

[0087] Preferably, the method provided in the embodiment of the present invention further includes improving the operation process of nnU-Net by deleting the process pool module during the verification process.

[0088] Specifically, when using the original nnU-Net code for training, the program will get stuck after a cycle of training verification is completed. By checking the source code, it is found that in nnUNetTrainer.py, in order to improve the efficiency of the program, the Python built-in process pool Pool is set during verification, and the starmap_async() function is used to asynchronously process the request to convert the segmentation results from softmax form to NIfTI format during verification. Due to problems with Python itself, the introduction of the process pool will cause the program to deadlock and cannot continue to run. Therefore, the process pool module in the program is deleted to solve the problem of training stuck.

[0089] Step S3: Based on the lower limb CT full-length film, segmentation annotation data and key point annotation data generated after preprocessing, the nnDetection detection model is used for model training to obtain a key point detection model that can perform key point detection on lower limb case images.

[0090] Preferably, the method provided in the embodiment of the present invention further performs the following before training the key point detection model: classifying and cutting the lower limb CT full-length film, segmentation annotation data and key point annotation data generated after preprocessing. The classification and cutting process is as follows:

[0091] First, the key point annotation data are divided into three categories and the key points within each category do not overlap with each other; two of the three categories of key points are all located at the end of the femur and below, and the other category of key points are all located at the end of the femur and above.

[0092] Secondly, let the original image height be h, then the image height h′ of the femur end and the following parts is retained as:

[0093]

[0094] Intercept the pixels from 0 to h' on the z-axis of the original image.

[0095] Finally, the image height h″ of the femur end and above is retained as:

[0096]

[0097] Intercept the pixels from h″ to h on the z-axis of the original image.

[0098] In the above, in order to save computing resources, all input images of each computing task are intercepted on the z-axis.

[0099] In an embodiment of the present invention, after the key point detection model is obtained by model training using the nnDetection detection model, the key point results outputted by it are restored on the z-axis, and non-maximum suppression is performed on each type of key point so that each type of key point retains the prediction with the highest confidence.

[0100] It should be noted that the nnDetection model is a medical detection model. The simultaneous localization and classification of objects in medical images, also known as medical object detection, has a high degree of clinical inertia. nnDetection can adapt to any medical detection problem without any human intervention.

[0101] For ease of understanding, the following is combined with Fig. 6A and 6B To demonstrate the key point prediction and the true value performed by the key point detection model, a comparison between the two shows that the key point prediction of the key point detection model in the present invention is extremely close to the true value.

[0102] like Figure 7 , a schematic diagram of a segmentation and key point detection system for lower limb CT full-length films according to an embodiment of the present invention is shown. The segmentation and key point detection system 700 according to an embodiment of the present invention includes the following modules: a preprocessing module 701, an image segmentation module 702, and a key point detection module 703.

[0103] The preprocessing module 701 is used to obtain CT image data, segmentation annotation data and key point annotation data of lower limb case images, and preprocess the CT image data and key point annotation data to convert them into a data format suitable for a neural network, thereby generating a corresponding lower limb CT full-length film and its segmentation annotation data.

[0104] The image segmentation module 702 is used to perform model training using the nnU-Net segmentation model based on the lower limb CT full-length film and its segmentation annotation data generated after preprocessing to obtain a medical image segmentation model that can perform bone structure segmentation on lower limb case images.

[0105] The key point detection module 703 is used to perform model training using the nnDetection detection model based on the lower limb CT full-length film, segmentation annotation data and key point annotation data generated after preprocessing to obtain a key point detection model that can perform key point detection on lower limb case images.

[0106] In some examples, the process of preprocessing the CT image data and key point annotation data by the preprocessing module 701 includes: integrating the original DICOM data slices of each case into corresponding NIFTI format images, and converting the key point annotation data in JSON format into NIFTI format images.

[0107] In some examples, the process of converting the key point annotation data in JSON format into an image in NIFTI format by the preprocessing module 701 includes: reading the coordinate data of each key point in the world coordinate system from a file in JSON format, converting the world coordinate system to the corresponding pixel coordinate system based on affine transformation, and obtaining the coordinate data of each key point in the pixel coordinate system; reducing the length, width and height of each key point to a preset size; and creating key point annotations in NIFTI format based on the key point information in JSON format after coordinate conversion and size compression.

[0108] In some examples, the image segmentation module 702 uses nnU-Net to perform model training to obtain an image segmentation model that can segment the full-length CT film of the lower limbs into corresponding pelvis, femur, patella, tibia, fibula, inner femur, inner tibia, and foot.

[0109] In some examples, after using the medical image segmentation model to segment the lower limb case image, the image segmentation module 702 also performs a step of adding connected domain post-processing to the segmentation result, which includes: extracting the femur, tibia, and fibula from the segmentation prediction results output by the medical image segmentation model, retaining the two largest connected domains of these three types of bone structures to remove discrete points.

[0110] In some examples, the key point detection module 703 is also used to perform the following: improving the running process of nnU-Net by deleting the process pool module during the verification process.

[0111] In some examples, the key point detection module 703 further performs the following before training to obtain the key point detection model: classifying and cropping the lower limb CT full-length film, segmentation annotation data and key point annotation data generated after preprocessing.

[0112] In some examples, the classification and cropping process of the key point detection module 703 includes: dividing the key point annotation data into three categories and the key points in each category do not overlap; two of the three categories of key points are all located at the end of the femur and below, and the other category of key points are all located at the end of the femur and above; let the original image height be h, then the image height h′ retained at the end of the femur and below is: Cut off the pixel points from 0 to h' on the z-axis of the original image; let the image height h" of the femur end and above be: Intercept the pixels from h″ to h on the z-axis of the original image.

[0113] It should be noted that: the segmentation and key point detection system for lower limb full-length CT films provided in the above-mentioned embodiment is only illustrated by the division of the above-mentioned program modules when performing segmentation and key point detection for lower limb full-length CT films. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the system can be divided into different program modules to complete all or part of the processing described above. In addition, the segmentation and key point detection device for lower limb full-length CT films provided in the above-mentioned embodiment and the segmentation and key point detection method embodiment for lower limb full-length CT films belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0114] The segmentation and key point detection method for the full-length CT film of the lower limbs provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 8 , which is an optional hardware structure diagram of an electronic terminal 800 provided in an embodiment of the present invention. The terminal 800 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 800 suitable for segmentation and key point detection of full-length CT films of lower limbs includes: at least one processor 801, a memory 802, at least one network interface 804 and a user interface 806. The various components in the device are coupled together through a bus system 805. It can be understood that the bus system 805 is used to realize connection and communication between these components. In addition to the data bus, the bus system 805 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 8 In the specification, various buses are labeled as bus systems.

[0115] The user interface 806 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0116] It is understood that the memory 802 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0117] The memory 802 in the embodiment of the present invention is used to store various categories of data to support the operation of the segmentation and key point detection terminal 800 applicable to the full-length CT film of the lower limbs. Examples of these data include: any executable program used to operate on the segmentation and key point detection terminal 800 applicable to the full-length CT film of the lower limbs, such as an operating system 8021 and an application 8022; the operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 8022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The segmentation and key point detection method applicable to the full-length CT film of the lower limbs provided in the embodiment of the present invention may be included in the application 8022.

[0118] The method disclosed in the above embodiment of the present invention can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 801 or the instruction in the form of software. The above processor 801 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 801 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0119] In an exemplary embodiment, the segmentation and key point detection terminal 800 suitable for the full-length CT film of the lower limbs can be used by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the above-mentioned method.

[0120] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0121] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0122] In summary, the present invention provides a segmentation and key point detection method, system, terminal and medium suitable for lower limb CT full-length films, aiming to build a full-process segmentation and key point detection model for lower limb CT full-length films based on nnU-Net and nnDetection, which can automatically process original lower limb CT full-length film data, segmentation data and key point annotation data, and realize the automatic segmentation of various bony structures of the lower limbs, as well as the automatic annotation function of various key points of the lower limbs. The present invention provides convenient operation, improves the accuracy and efficiency of tasks, improves the accuracy of key point detection and optimizes the code. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0123] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A segmentation and key point detection method for full-length CT films of lower limbs, characterized in that: include: Acquire CT image data, segmentation annotation data and key point annotation data of lower limb case images, and pre-process the CT image data and key point annotation data to convert them into a data format suitable for a neural network, thereby generating a corresponding lower limb CT full-length film and its segmentation annotation data; According to the lower limb CT full-length films and their segmentation annotation data generated after preprocessing, the nnU-Net segmentation model is used for model training to obtain a medical image segmentation model that can perform bone structure segmentation on lower limb case images; According to the lower limb CT full-length films, segmentation annotation data and key point annotation data generated after preprocessing, the nnDetection detection model is used for model training to obtain a key point detection model that can perform key point detection on lower limb case images.

2. The segmentation and key point detection method for lower limb CT full-length films according to claim 1, characterized in that: The process of preprocessing the CT image data and key point annotation data includes: integrating the original DICOM data slices of each case into corresponding NIFTI format images, and converting the key point annotation data in JSON format into NIFTI format images.

3. The segmentation and key point detection method for lower limb CT full-length films according to claim 2, characterized in that: The process of converting JSON format keypoint annotation data into NIFTI format images includes: Read the coordinate data of each key point in the world coordinate system from the file in JSON format, convert the world coordinate system to the corresponding pixel coordinate system based on affine transformation, and obtain the coordinate data of each key point in the pixel coordinate system; Reduce the length, width and height of each key point to the preset size; Create key point annotations in NIFTI format based on key point information in JSON format after coordinate conversion and size compression.

4. The segmentation and key point detection method for lower limb CT full-length films according to claim 1, characterized in that: include: After model training using nnU-Net, we obtained an image segmentation model that can segment the full-length CT film of the lower limbs into the corresponding pelvis, femur, patella, tibia, fibula, inner part of the femur, inner part of the tibia, and foot.

5. The segmentation and key point detection method for lower limb CT full-length films according to claim 1, characterized in that: After using the medical image segmentation model to segment the lower limb case images, the method also performs a step of adding connected domain post-processing to the segmentation results, which includes: extracting the parts of the femur, tibia, and fibula in the segmentation prediction results output by the medical image segmentation model, retaining the two largest connected domains of these three types of bone structures to remove discrete points.

6. The segmentation and key point detection method for lower limb CT full-length films according to claim 1, characterized in that: The method also includes improving the operation process of nnU-Net by deleting the process pool module during the verification process.

7. The segmentation and key point detection method for lower limb CT full-length films according to claim 1, characterized in that: Before training to obtain the key point detection model, the method further performs the following: classifying and cutting the lower limb CT full-length slice, segmentation annotation data and key point annotation data generated after preprocessing.

8. The segmentation and key point detection method for lower limb CT full-length films according to claim 7, characterized in that: The classification and cutting process includes: The key point annotation data are divided into three categories, and the key points in each category do not overlap with each other; two of the three categories of key points are all located at the end of the femur and below, and the other category of key points are all located at the end of the femur and above; Let the original image height be h, then the image height h′ of the femur end and the following parts is retained as: Intercept the pixels from 0 to h' on the z-axis of the original image; Let the image height h″ of the femur end and above be: Intercept the pixels from h″ to h on the z-axis of the original image.

9. A segmentation and key point detection system for lower limb CT full-length films, characterized in that: include: A preprocessing module is used to obtain CT image data, segmentation annotation data and key point annotation data of lower limb case images, and preprocess the CT image data and key point annotation data to convert them into a data format suitable for a neural network, thereby generating a corresponding lower limb CT full-length film and its segmentation annotation data; An image segmentation module is used to train the model using the nnU-Net segmentation model based on the lower limb CT full-length film and its segmentation annotation data generated after preprocessing to obtain a medical image segmentation model that can perform bone structure segmentation on lower limb case images; The key point detection module is used to train the model using the nnDetection detection model based on the full-length CT film of the lower limbs, the segmentation annotation data and the key point annotation data generated after preprocessing, so as to obtain a key point detection model that can perform key point detection on the lower limb case images.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for segmentation and key point detection applicable to full-length CT films of lower limbs as described in any one of claims 1 to 9 is implemented.

11. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the terminal executes the segmentation and key point detection method applicable to the full-length CT film of the lower limbs as described in any one of claims 1 to 9.

Images