Rib automatic recognition method and system based on three-dimensional point cloud segmentation

Abstract

The application belongs to the technical field of medical image processing, and provides a rib automatic recognition method and system based on three-dimensional point cloud segmentation. The method comprises the following steps: obtaining a chest CT image and preprocessing the chest CT image to obtain normalized point cloud data; performing multiple data sampling on the normalized point cloud data, and obtaining a corresponding number of prediction probability results of rib labels of each point cloud data based on the sampled point cloud data and a rib automatic recognition model; fusing all prediction probability results of rib labels of each point cloud data, and converting the point cloud data after the prediction probability results are fused into image data; and automatically recognizing rib names in the image data according to the rib labels of the point cloud data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to a rib automatic recognition method and system based on three-dimensional point cloud segmentation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Rib fracture is the most common injury in chest trauma and is one of the main contents of medical identification. Rib fracture is generally caused by external violence. When direct violence acts on the chest, rib fracture often occurs at the impacted site, and the fracture end is folded inward, causing damage to the internal organs. Therefore, rib fracture has a high incidence and mortality risk.

[0004] At present, the conventional chest CT is the main detection means for rib fracture. The physician can locate the fracture and its complication position according to the chest CT imaging, but when determining the rib number of the rib fracture, the physician needs to count from the first rib appearing in the CT to determine the rib number of the fracture. However, chest CT generally has dozens to hundreds of images, and the process of checking the number is relatively cumbersome and labor-intensive. Another method to determine the rib number is to perform three-dimensional reconstruction of the ribs, but this generally increases the patient's cost.

[0005] Deep learning technology has been widely studied in medical imaging applications, such as detection of lung nodules, rib fractures, and brain hemorrhages. It plays a high role in practical applications, but the conventional method for automatic recognition of ribs is to segment the ribs, find the rib area, and then calculate the connected domain of each rib. The connected part is considered as a rib, and the rib label is given from top to bottom. However, the inventors found that this method relies on the accuracy of rib segmentation and is not easy to handle in the case of misaligned fractures, which can lead to counting errors. SUMMARY

[0006] To solve the technical problems in the background art, the present application provides a rib automatic recognition method and system based on three-dimensional point cloud segmentation, which can effectively avoid the problem of insufficient segmentation accuracy of the traditional deep learning method based on images for the fracture area, especially the misaligned fracture area.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a rib automatic recognition method based on three-dimensional point cloud segmentation, which comprises:

[0009] acquire a chest CT image and pre-process it to obtain normalized point cloud data;

[0010] perform multiple data sampling on the normalized point cloud data, and based on the sampled point cloud data and the rib automatic recognition model, obtain a corresponding number of prediction probability results of the rib label of each point cloud data;

[0011] fuse all prediction probability results of the rib label of each point cloud data, and convert the point cloud data after fusion of the prediction probability results into image data, and automatically identify the rib name in the image data according to the rib label of the point cloud data.

[0012] As an implementation form, the rib automatic recognition model is a two-stage model, wherein a first-stage model is used to segment a rib region and remove point cloud data of bone and noise regions; and a second-stage model takes the output of the first-stage model as input, extracts features of the input point cloud data, and performs rib label recognition.

[0013] As an implementation form, the rib automatic recognition model is a dynamic graph edge convolution network, which is composed of a basic part, a rib segmentation part, and a rib recognition part; the basic part and the rib segmentation part constitute a first-stage model, and the rib recognition part is a second-stage model.

[0014] As an implementation form, the rib label includes 25 categories, including 24 rib labels and one non-rib label.

[0015] As an implementation form, in the process of performing multiple data sampling on the normalized point cloud data, all point cloud data are extracted by using a non-replacement random sampling method.

[0016] As an implementation form, the process of fusing all prediction probability results of the rib label of each point cloud data is as follows:

[0017] Taking an average value of all prediction probability results of the rib label of each point cloud data.

[0018] A second aspect of the present application provides a rib automatic recognition system based on three-dimensional point cloud segmentation, which comprises:

[0019] A point cloud data acquisition module is configured to acquire a chest CT image and pre-process it to obtain normalized point cloud data.

[0020] A point cloud label prediction module is configured to perform multiple data sampling on the normalized point cloud data, and based on the sampled point cloud data and the rib automatic recognition model, obtain a corresponding number of prediction probability results of the rib label of each point cloud data.

[0021] The rib automatic identification module is used for fusing all prediction probability results of each point cloud data rib label, converting the point cloud data after the prediction probability results are fused into image data, and automatically identifying rib names in the image data according to the rib label of the point cloud data.

[0022] A third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps in the rib automatic identification method based on three-dimensional point cloud segmentation as described above.

[0023] A fourth aspect of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps in the rib automatic identification method based on three-dimensional point cloud segmentation as described above when executing the program.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] (1) The rib automatic identification method based on point cloud segmentation provided by the present application can effectively avoid the problem of insufficient segmentation accuracy of the traditional deep learning method based on images for the fracture region, especially the dislocation fracture region, by pre-processing the chest CT image to obtain normalized point cloud data, combining the sampled point cloud data with the rib automatic identification model to obtain a corresponding number of prediction probability results of the rib label of each point cloud data, then fusing all prediction probability results of each point cloud data rib label, and converting the point cloud data after the prediction probability results are fused into image data to automatically identify the rib names in the image data according to the rib label of the point cloud data.

[0026] (2) The rib automatic identification method based on point cloud segmentation provided by the present application has the advantages of simple processing, high timeliness and stable effect, and can effectively process various chest CT images. The rib automatic identification model is a two-stage model, wherein the first stage model is used to segment the rib region and remove the point cloud data of the bone region and the noise region. The second stage model takes the output of the first stage model as input, extracts the features of the input point cloud data, and performs rib label recognition, effectively solving the problem of interference of non-rib parts on the rib label, thereby obtaining more accurate rib recognition results.

[0027] Advantages of additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of the application, together with its

[0029] Figure 1 Flow chart of rib automatic recognition method based on three-dimensional point cloud segmentation of an embodiment of the application;

[0030] Figure 2 Flow chart of chest CT data converted into point cloud data marked by medical experts of an embodiment of the application;

[0031] Figure 3 Structural schematic diagram of rib automatic recognition model of an embodiment of the application;

[0032] Figure 4 Three-dimensional effect diagram of chest CT for automatically recognizing ribs of an embodiment of the application;

[0033] Figure 5 Structural schematic diagram of rib automatic recognition system based on three-dimensional point cloud segmentation of an embodiment of the application. DETAILED DESCRIPTION

[0034] The application will be further described below with reference to the drawings and embodiments.

[0035] It should be noted that the following detailed description is illustrative only, and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0036] It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It should be noted that the terms “comprises”, “comprising”, “includes”, “including” and the like are specifically intended to be open-ended. Generally these terms are intended to mean that the statement resultantly placed before the respective term(s) will be present, but compositions including items not specifically recited are also contemplated. Further, it should be noted that the terms “first”, “second”, and the like, are intended to modify, respectively, a first element or step and a second element or step etc.

[0037] Terminology:

[0038] Human ribs 12 pairs, left and right symmetrical, the rear end is related to the thoracic vertebrae, the front end is only the first to seventh rib connected with the sternum by cartilage, called true rib; The eighth to twelfth ribs are called false ribs, among which the eighth to tenth ribs are connected with the cartilage of the upper rib by rib cartilage to form a rib arch, and the eleventh and twelfth ribs are free at the front end, also known as floating ribs.

[0039] Embodiment one

[0040] Reference Figure 1This embodiment provides an automatic rib identification method based on three-dimensional point cloud segmentation, which specifically includes the following steps:

[0041] S101: Acquire chest CT images and preprocess them to obtain normalized point cloud data.

[0042] Specifically, during the preprocessing of chest CT images, voxel values ​​exceeding a set voxel threshold are filtered out.

[0043] For example: Extract the image from the DICOM data, then filter out voxel values ​​that exceed the set voxel threshold according to the set voxel threshold, and record the coordinates (x, y, z) here. The set voxel threshold is 200.

[0044] In the specific implementation process, during the multiple data sampling of normalized point cloud data, all point cloud data are extracted using a random sampling method without replacement.

[0045] For example: each time, 30,000 points are extracted to form a sub-point cloud. The remaining points are then extracted from the 3,000 points extracted in the previous step, ensuring that there are 30,000 points each time. This operation is repeated twice.

[0046] S102: Perform multiple data sampling on the normalized point cloud data, and based on the sampled point cloud data and the rib automatic recognition model, obtain the prediction probability results of the corresponding number of rib labels for each point cloud data.

[0047] In a specific implementation, the automatic rib recognition model is a two-stage model. The first-stage model is used to segment the rib region and remove the point cloud data of the bone region and noise region. The second-stage model takes the output of the first-stage model as input, extracts the features of the input point cloud data, and performs rib label recognition.

[0048] like Figure 3 As shown, the automatic rib recognition model is a dynamic graph-edge convolutional network, consisting of a base part, a rib segmentation part, and a rib recognition part; the base part and the rib segmentation part constitute the first-stage model, and the rib recognition part is the second-stage model.

[0049] The basic part is used to perform spatial transformation and feature extraction on the input point cloud data in sequence; the rib segmentation part is used to extract the features output by the basic part and fuse them, and then classify each point cloud through a multilayer perceptron; the input of the rib recognition part consists of the features of the basic part and the output of the rib segmentation part. After feature extraction and fusion, each point cloud is classified through a multilayer perceptron and the rib name corresponding to the point cloud is output.

[0050] The rib label includes 25 categories, including 24 rib labels and one non-rib label.

[0051] It should be noted that the rib automatic recognition model is pre-trained.

[0052] As shown in Figure 2 The process of constructing a data set for training the model includes:

[0053] S10211 preprocessing step and S10212 three-dimensional point cloud data generation step.

[0054] The S10211 preprocessing step specifically includes:

[0055] S10211. Separate the chest CT sequence from the collected cases;

[0056] S10212. Obtain the original CT image from the chest CT sequence;

[0057] S10213. Thresholding the original CT image to obtain image data above the threshold value. Since the HU value of bone tissue in CT is generally greater than 200, the threshold value is set to 200.

[0058] The S10212 three-dimensional point cloud data generation step specifically includes

[0059] S10221. Label the rib image data after thresholding, delineate the area containing the ribs, and give the rib number; where the rib number is left rib L1, L2…L12, right rib R1, R2…R12, the label of the labeled area corresponds to the rib number one by one, L1, L2…L12 corresponds to 1, 2…12 respectively, R1, R2…R12 corresponds to 13, 14…24 respectively;

[0060] S10222. Obtain all voxel point coordinates (x, y, z) in the thresholded rib image data that exceed the threshold value to form three-dimensional point cloud data;

[0061] S10223. According to the labeled rib label and the coordinates of each point in the three-dimensional point cloud data, set the label of each point to form three-dimensional point cloud data with labels, the data format is (x, y, z, label), and then store the point cloud data in the text for use when training.

[0062] The following training method mainly involves the division of point cloud data set and training and precision testing, and the specific implementation method includes:

[0063] According to the processed point cloud data, the training set and the test set are divided, the proportion is 8:2, that is, 80% of the point cloud data (single chest CT processed point cloud data as a group of point cloud) is randomly selected to form the training set, and the remaining 20% of the point cloud data is used as the test set;

[0064] Due to the influence of CT equipment, layer thickness, shooting body position and other factors, the position of the ribs in the image changes a lot, resulting in irregular coordinate distribution of the original point cloud data, so it is necessary to normalize the point cloud data. Taking a group of point cloud as an example, the specific normalization calculation method is:

[0065] ①First, calculate the mean value of point cloud coordinates, that is, the mean value of x, y and z coordinates (x_mean, y_mean, z_mean);

[0066] ②Move all point cloud coordinates, specifically, subtract the coordinate mean value from all point cloud coordinates, that is, (x-x_mean, y-y_mean, z-z_mean);

[0067] ③Calculate the Euclidean distance d of the point farthest from the mean point;

[0068] ④Divide all points calculated in ② by the d value calculated in ③ to obtain the normalized coordinate point cloud data.

[0069] Next, take the rib automatic recognition model as an example Figure 3 As shown in the TS-DGCNN (Two-Stage Dynamic GraphCNN), which is a two-stage dynamic graph edge convolution network.

[0070] Set the TS-DGCNN classification number to 25 (0 represents non-rib point cloud, 1-24 represents 24 ribs respectively), and set the input point cloud number n to 30000;

[0071] Set the training parameters, specifically set the training optimization method to Adam, set the training cycle number to 300, set the training batch size to 10, and set the training initial learning rate to 3e-3.

[0072] In Figure 3In the specific implementation, the TS-DGCNN is composed of a basic part, a rib segmentation part and a rib recognition part. The basic part is composed of a Spatial transform and an EdgeConv; the rib segmentation part is composed of an EdgeConv and a pooling layer to extract and fuse features, and then a multilayer perceptron is used to classify each point cloud; the input of the rib recognition part is composed of the features of the basic part and the output of the rib segmentation part, and then two EdgeConv and a pooling layer are used to extract and fuse features, and finally a multilayer perceptron is used to classify each point cloud to output the rib name corresponding to the point cloud.

[0073] The input of the rib recognition part is composed of two parts, one part is the feature of the basic part, and the other part is the output of the rib segmentation part, and the two parts are combined in the form of argmax operation on the nx2 matrix of the rib segmentation part to obtain an n*1 matrix with values of 0 and 1, 0 representing non-rib and 1 representing rib. According to the points with a value of 1 in the n x 1 matrix (the total number is m, m<n), the features of the basic part are intercepted, and the n x 64 is the m x 64 feature, which is input into two EdgeConv and a pooling layer to extract and fuse features, and then a multilayer perceptron is used to classify each point into 25 categories, and the rib name of the m points is output.

[0074] The rib automatic recognition model training process is as follows: in the first 50 cycles, the rib naming part loss is controlled to be 0, only the basic part and the rib segmentation part are trained, in the 50-100 cycles, the learning rate of the basic part is controlled to be 0, the rib segmentation part loss is controlled to be 0, only the rib naming part is trained. Then the learning rate is adjusted to 1e-4, and the basic part, the rib segmentation part and the rib naming part are trained. The loss is the sum of the rib segmentation part loss and the rib naming part loss.

[0075] In the training, 30000 points are randomly sampled from each group of point cloud data as input in each cycle.

[0076] After 100 cycles, the training accuracy is tested every 10 cycles, the test accuracy is recorded, the model parameters are saved, and the specific test method is as follows:

[0077] Each group of point cloud data in the test set is extracted by non-replacement random sampling, 30000 points are extracted each time to form a sub-point cloud, the real label of each point is recorded, and each group of point cloud contains k sub-point clouds.

[0078] The extracted sub-point cloud is put into the trained model for prediction, wherein the rib segmentation part outputs points with a value of 0 to determine that the points are non-rib points, and the remaining points are given a predicted label by the rib naming part, and 30,000 predicted labels are obtained in total. The prediction results of the k groups of sub-point clouds are put back into the original point cloud (the repeated sampling is processed by averaging), and the accuracy of each group of point cloud segmentation is calculated according to the true label and the predicted label.

[0079] The average value of the accuracy of all groups of point clouds in the test set is taken as the model segmentation accuracy.

[0080] After 300 cycles of training, the model with the highest model segmentation accuracy is taken as the final model.

[0081] After the training and testing are completed, the obtained model can be deployed into a prediction system, which accepts any chest CT data and outputs rib data with a given name, and the effect is as shown in Figure 4 .

[0082] In this embodiment, the prediction result is a softmax value of 25 categories, and the point cloud with a prediction result >0.5 for a label of 0 is discarded. The remaining points are classified according to the maximum value of the remaining 24 categories. The labels of other point clouds that do not exist in the 24 rib regions are set to 0.

[0083] S103: Fuse all prediction probability results of the rib labels of each point cloud data, and convert the point cloud data after the fusion of the prediction probability results into image data, and automatically identify the rib names in the image data according to the rib labels of the point cloud data.

[0084] In specific implementation, the process of fusing all prediction probability results of the rib labels of each point cloud data is as follows:

[0085] The average value of all prediction probability results of the rib labels of each point cloud data is taken.

[0086] In other embodiments, the method of fusing all prediction probability results of the rib labels of each point cloud data can also use the following method: for each point cloud, the highest value and the lowest value of all prediction probabilities are selected, and the average value is calculated. Those skilled in the art can select the corresponding prediction result fusion method according to the actual accuracy requirement.

[0087] In the specific implementation process, before the point cloud data is converted into image data, in order to ensure the accuracy of the data and avoid data interference, the outliers of the point cloud are also removed.

[0088] In this step, the rib label will also be converted into rib number, and the rib name automatic recognition will be completed, wherein the conversion of rib number is as follows: 1, 2…12 correspond to L1, L2…L12 respectively, and 13, 14…24 correspond to R1, R2…R12 respectively.

[0089] Embodiment Two

[0090] With reference to Figure 5 The embodiment provides a rib automatic recognition system based on three-dimensional point cloud segmentation, which specifically comprises the following modules.

[0091] (1) A point cloud data acquisition module is used to acquire a chest CT image and pre-process the chest CT image to obtain normalized point cloud data.

[0092] (2) A point cloud label prediction module is used to perform multiple data sampling on the normalized point cloud data, and based on the sampled point cloud data and a rib automatic recognition model, a corresponding number of prediction probability results of the rib label of each point cloud data is obtained.

[0093] The rib automatic recognition model is a two-stage model, wherein a first-stage model is used to segment a rib region and remove point cloud data of bone regions and noise regions; and a second-stage model takes the output of the first-stage model as input, extracts features of the input point cloud data, and performs rib label recognition.

[0094] (3) A rib automatic recognition module is used to fuse all prediction probability results of the rib label of each point cloud data, and then convert the point cloud data after the fusion of the prediction probability results into image data, and automatically recognize rib names in the image data according to the rib label of the point cloud data.

[0095] It should be noted that each module in the embodiment corresponds to each step in Embodiment One, and the specific implementation process is the same, which will not be repeated here.

[0096] Embodiment Three

[0097] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the rib automatic recognition method based on three-dimensional point cloud segmentation.

[0098] Embodiment Four

[0099] The embodiment provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps in the rib automatic recognition method based on three-dimensional point cloud segmentation when executing the program.

[0100] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses and computer program products according to various embodiments of the present application. In this regard, each flowchart block and / or block in the Figures can represent a module, segment, or portion of code, which comprises one or more executable Figure 1 The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks.

[0101] The specific embodiments of the present application have been shown and described in detail so that they can be better understood and fully appreciated. While the application has been described and illustrated in connection with certain embodiments, those skilled in the art will understand that the application is capable of further modifications. This application is intended to cover any variations or equivalents of the application included within the spirit and scope of the application.

Claims

1. A rib automatic recognition method based on three-dimensional point cloud segmentation, characterized in that, The application relates to a method for automatically identifying rib bones in chest CT images. The method comprises the following steps: obtaining chest CT images and pre-processing the images to obtain normalized point cloud data; performing multiple data sampling on the normalized point cloud data, and obtaining a corresponding number of prediction probability results of rib bone labels of each point cloud data based on the sampled point cloud data and a rib bone automatic identification model; fusing all prediction probability results of rib bone labels of each point cloud data, and converting the point cloud data after the prediction probability results are fused into image data, and automatically identifying rib bone names in the image data according to the rib bone labels of the point cloud data; the rib bone automatic identification model is a two-stage model, wherein a first-stage model is used to segment rib bone regions and remove point cloud data of bone regions and noise regions; a second-stage model takes the output of the first-stage model as input, extracts features of the input point cloud data, and identifies rib bone labels; in the process of performing multiple data sampling on the normalized point cloud data, all point cloud data are extracted in a non-replacement random sampling mode; the process of fusing all prediction probability results of rib bone labels of each point cloud data is that the average value of all prediction probability results of rib bone labels of each point cloud data is taken; 2.The rib automatic recognition method based on three-dimensional point cloud segmentation of claim 1, wherein, the rib bone automatic identification model is a dynamic graph edge convolution network, which comprises a basic part, a rib bone segmentation part and a rib bone identification part; the basic part and the rib bone segmentation part constitute the first-stage model, and the rib bone identification part is the second-stage model, wherein the basic part comprises a Spatial transform and an EdgeConv; the rib bone segmentation part extracts and fuses features through an EdgeConv and a pooling layer, and then classifies each point cloud through a multilayer perceptron; the input of the rib bone identification part is composed of features of the basic part and output of the rib bone segmentation part, and then features are extracted and fused through two EdgeConv and a pooling layer, and finally each point cloud is classified through a multilayer perceptron, and the rib bone name corresponding to the point cloud is output.

3. A rib automatic recognition system based on three-dimensional point cloud segmentation, adopting the rib automatic recognition method based on three-dimensional point cloud segmentation according to any one of claims 1-2, characterized in that, The rib bone labels comprise 25 categories, including 24 rib bone labels and one non-rib bone label. The application relates to a method for automatically identifying rib bones in chest CT images. The method comprises the following steps: a point cloud data acquisition module is used for obtaining chest CT images and pre-processing the images to obtain normalized point cloud data; 4. The rib automatic identification system based on three-dimensional point cloud segmentation of claim 3, wherein, a point cloud label prediction module is used for performing multiple data sampling on the normalized point cloud data, and obtaining a corresponding number of prediction probability results of rib bone labels of each point cloud data based on the sampled point cloud data and a rib bone automatic identification model; a rib bone automatic identification module is used for fusing all prediction probability results of rib bone labels of each point cloud data, and converting the point cloud data after the prediction probability results are fused into image data, and automatically identifying rib bone names in the image data according to the rib bone labels of the point cloud data. the rib bone automatic identification model is a two-stage model, wherein a first-stage model is used to segment rib bone regions and remove point cloud data of bone regions and noise regions; a second-stage model takes the output of the first-stage model as input, extracts features of the input point cloud data, and identifies rib bone labels.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the steps in the rib automatic recognition method based on three-dimensional point cloud segmentation in any one of claims 1-2.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the rib automatic recognition method based on three-dimensional point cloud segmentation in any one of claims 1-2.

Images