Image processing method and device and image processing equipment

Abstract

The invention discloses an image processing method and device and image processing equipment, and relates to the technical field of image processing. After acquiring an original lung image, the image processing device can acquire an image structure of a lung trachea based on the original lung image, acquire a skeleton line segmentation result of the trachea based on the original lung image and the image structure, and acquire a lung segment segmentation result based on the original lung image and the skeleton line segmentation result of the trachea. Since the graph structure is a topological structure of the tracheas and the graph structure contains the adjacency relationship between the tracheas with the connection relationship, the image processing equipment can accurately distinguish different types of tracheas with adjacent positions based on the graph structure. Therefore, based on the original lung image and the image structure, the accuracy of the obtained skeleton line segmentation result of the trachea is relatively high, so that the accuracy of the determined lung segment segmentation result can be improved, and the accuracy of lung segment segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device, and image processing equipment. Background Art

[0002] In the process of diagnosing and treating lung diseases, doctors can remove the lesions according to the lung segment where they are located. Therefore, the accuracy of lung segmentation results is crucial for accurate diagnosis of the disease. However, the current solution has low accuracy in lung segmentation. Summary of the invention

[0003] The present invention provides an image processing method and apparatus, and an image processing device, which can solve the problem of low accuracy of lung segmentation in related technologies. The technical solution is as follows:

[0004] In one aspect, an image processing method is provided, which is applied to an image processing device, and the method comprises:

[0005] Acquire original lung images;

[0006] Acquire the image structure of lung trachea based on the original lung image;

[0007] Obtaining the skeleton line segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure, the trachea including the segmental trachea and the non-segmental trachea;

[0008] Based on the original lung image and the trachea skeleton segmentation result, the lung segmentation result is obtained.

[0009] Optionally, a map structure of the lung trachea is obtained based on the original lung image, including:

[0010] Obtaining lung trachea segmentation results based on the original lung image;

[0011] Obtaining the skeleton line of the pulmonary trachea based on the segmentation result of the pulmonary trachea;

[0012] The image structure of the pulmonary trachea is obtained based on the skeleton line of the pulmonary trachea.

[0013] Optionally, the image structure of the pulmonary trachea is obtained based on the skeleton line of the pulmonary trachea, including:

[0014] Each skeleton line branch in the skeleton line of the lung trachea is used as a node of the graph structure;

[0015] For any two nodes, if the skeleton line branches corresponding to the two nodes have the same bifurcation point, it is determined that the two nodes are connected in the graph structure.

[0016] Optionally, a segmentation result of the lung and trachea is obtained based on the original lung image, including:

[0017] Normalize the original lung image to obtain a standard lung image;

[0018] In the case where the standard lung image contains target tissue other than the lung, the target tissue in the standard lung image is removed, and a segmentation result of the lung trachea is obtained based on the processed standard lung image.

[0019] Optionally, a skeleton segmentation result of the trachea in the lung trachea is obtained based on the original lung image and the graph structure, including:

[0020] Obtain high-dimensional features of lung trachea based on original lung images;

[0021] Based on the graph structure and high-dimensional features, the skeleton segmentation results of the trachea in the lung trachea are obtained.

[0022] Optionally, high-dimensional features of the lung trachea are obtained based on the original lung image, including:

[0023] Obtaining lung trachea segmentation results based on the original lung image;

[0024] The standard lung image and the segmentation results of the lung and trachea are input into the high-dimensional feature extraction model to obtain the high-dimensional features of the lung and trachea;

[0025] The standard lung image is obtained by normalizing the original lung image.

[0026] Optionally, based on the original lung image and the trachea skeleton segmentation result, a lung segmentation result is obtained, including:

[0027] Based on the tracheal skeleton segmentation result and the lung trachea segmentation result, the trachea segmentation result is obtained;

[0028] Based on the original lung image and the trachea segmentation results, a lung segmentation result is obtained.

[0029] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method of the above aspect is implemented.

[0030] On the other hand, an image processing device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the image processing method of the above aspect is implemented.

[0031] In another aspect, an image processing method and device is provided, the device comprising:

[0032] A first acquisition module is used to acquire an original lung image;

[0033] A second acquisition module is used to acquire a lung trachea image structure based on an original lung image;

[0034] A third acquisition module is used to acquire a skeleton line segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure;

[0035] The fourth acquisition module is used to acquire the lung segmentation result based on the original lung image and the trachea skeleton line segmentation result.

[0036] In summary, the embodiments of the present invention provide an image processing method and apparatus, and an image processing device. After the image processing device obtains the original lung image, it can obtain the graph structure of the lung trachea based on the original lung image, obtain the skeleton line segmentation result of the trachea based on the original lung image and the graph structure, and obtain the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0037] Since the graph structure is the topological structure of the trachea, and the graph structure includes the adjacency relationship between the connected tracheae, the image processing device can accurately distinguish between adjacent tracheae of different categories based on the graph structure. Based on the original lung image and the graph structure, the accuracy of the obtained tracheal skeleton line segmentation result is high, thereby improving the accuracy of the determined lung segmentation result and improving the accuracy of lung segmentation.

[0038] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present invention;

[0040] Figure 2 is a flow chart of another image processing method provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a lung trachea segmentation result provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a skeleton line of a lung trachea provided in an embodiment of the present invention;

[0043] Figure 5 is a partial schematic diagram of a skeleton line of a lung trachea provided by an embodiment of the present invention;

[0044] Figure 6 is a schematic diagram of a graph structure provided by an embodiment of the present invention;

[0045] Figure 7It is a schematic diagram of a first trachea segmentation result obtained by processing a sample standard lung image and a second sample lung trachea segmentation result by a second initial model provided by an embodiment of the present invention;

[0046] Figure 8 is a schematic diagram of a lung lobe segmentation result provided by an embodiment of the present invention;

[0047] Fig. 9 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0048] Fig.10 It is a block diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0050] In the diagnosis and treatment of lung diseases, locating the lesion is very important. Lung segmentation is a key task in the positioning module of the lesion-assisted diagnosis system. Clinicians can resect the lesion according to the lung segment where the lesion is located. This method can maximize the preservation of the remaining normal lung tissue in the lobe where the lesion is located. However, due to the lack of obvious anatomical gaps and unclear boundaries between lung segments, segment resection is difficult and easy to damage adjacent tissues. Therefore, accurate segmentation of different lung segments is very important and meaningful.

[0051] As an organ with a clear topological structure in the lung, the trachea in the trachea can include segmental trachea and non-segmental trachea, which can be divided into 28 categories in total. Among them, there are 18 types of segmental trachea, 8 of which are distributed in the left lung, and 10 in the right lung. Each segmental trachea is distributed in different lobes, and its division and determination depends on the topological structure of the trachea. Non-segmental trachea can include trunk trachea and lobar trachea. The trunk trachea can be divided into 3 categories, and the lobar trachea can be divided into 7 categories.

[0052] The segment-level tracheal classification results determine the distribution results of the lung segments. Therefore, segmenting the segment-level trachea and completing the segmentation and positioning of the lung segments on this basis is a very common and important auxiliary diagnosis and treatment method. However, the current segmentation of trachea faces the following problems: First, deep learning requires a large number of samples, and manual labeling is time-consuming and laborious. Several tracheal binary segmentation methods based on convolutional neural networks developed in related technologies have made significant progress in binary segmentation, but the accuracy of tracheal segmentation results still needs to be improved due to the large number of categories and different morphologies of tracheal segmentation. Second, the trunk trachea is thick and long, followed by the lobar trachea, while the segment-level trachea is sparsely distributed and very thin, which is easy to be misdetected or undetected. Third, there is no obvious interface between adjacent segments of the trachea, that is, the voxels around the interface may have similar local appearance and grayscale distribution, but different labels. In addition, individual differences and pathological changes in trachea pose a threat to network training. Therefore, a more accurate lung and trachea segmentation method is needed to meet the diagnostic and treatment needs of lung segmentation.

[0053] According to the method provided by the embodiment of the present invention, after the image processing device obtains the original lung image, it can obtain the graph structure of the lung trachea based on the original lung image, obtain the skeleton line segmentation result of the trachea based on the original lung image and the graph structure, and obtain the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0054] Since the graph structure is the topological structure of the trachea, and the graph structure includes the adjacency relationship between the connected tracheae, the image processing device can accurately distinguish between adjacent tracheae of different categories based on the graph structure. Based on the original lung image and the graph structure, the accuracy of the obtained tracheal skeleton line segmentation result is high, thereby improving the accuracy of the determined lung segmentation result and improving the accuracy of lung segmentation.

[0055] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present invention, the method is applied to an image processing device, and refers to Figure 1 , the method comprising:

[0056] Step 101: Acquire an original lung image.

[0057] The image processing device can obtain the original lung image.

[0058] Step 102: Acquire the image structure of the lung trachea based on the original lung image.

[0059] After the image processing device acquires the original lung image, the graph structure of the lung trachea can be acquired based on the original lung image, wherein the graph structure may include an adjacency relationship between tracheas having a connection relationship.

[0060] In an embodiment of the present invention, the skeleton line of the trachea in the lung trachea includes multiple skeleton line branches. Each skeleton line branch in the multiple skeleton line branches can be a skeleton line between a bifurcation point and an end point of the skeleton line, or between a bifurcation point and a bifurcation point, or between an end point and an end point. The graph structure is a data structure for representing the multiple skeleton line branches and the connection relationship between the multiple skeleton line branches. It can be seen that according to the graph structure, the trachea with a connection relationship can be obtained, and then the adjacency relationship between the trachea with a connection relationship can be obtained.

[0061] Step 103: Based on the original lung image and the graph structure, a skeleton line segmentation result of the trachea in the lung trachea is obtained.

[0062] The image processing device obtains the skeleton line segmentation result of the trachea in the pulmonary trachea based on the original lung image and the graph structure. The trachea in the pulmonary trachea may include segmental trachea and non-segmental trachea, the segmental trachea may be the third-level bronchus branched after the lobar bronchus enters the hilum, and the segmental trachea may include the left segmental trachea and the right segmental trachea. The non-segmental trachea may include the trunk trachea and the lobar trachea.

[0063] Step 104: Obtain a lung segmentation result based on the original lung image and the trachea skeleton segmentation result.

[0064] After the image processing device obtains the skeleton line segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure, it can obtain the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0065] In summary, an embodiment of the present invention provides an image processing method. After the image processing device obtains the original lung image, it can obtain the graph structure of the lung trachea based on the original lung image, obtain the skeleton line segmentation result of the trachea based on the original lung image and the graph structure, and obtain the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0066] Since the graph structure is the topological structure of the trachea, and the graph structure includes the adjacency relationship between the connected tracheae, the image processing device can accurately distinguish between adjacent tracheae of different categories based on the graph structure. Based on the original lung image and the graph structure, the accuracy of the obtained tracheal skeleton line segmentation result is high, thereby improving the accuracy of the determined lung segmentation result and improving the accuracy of lung segmentation.

[0067] Figure 2 is a flowchart of another image processing method provided by an embodiment of the present invention, the method is applied to an image processing device, and refers to Figure 2 , the method may include:

[0068] Step 201: Acquire an original lung image.

[0069] The image processing device can obtain an original lung image. The image processing device can be a three-dimensional computed tomography (CT) device, and the image processing device can pre-store an original lung image, which can be a three-dimensional image. The size of the original lung image can be D×H×W, where D, H, and W are all positive integers. The original lung image can be a lung image taken by a three-dimensional CT that meets the digital imaging and communications in medicine (DICOM) standard.

[0070] Step 202: normalize the original lung image to obtain a standard lung image.

[0071] After the image processing device acquires the original lung image, it can perform normalization processing on the original lung image to obtain a standard lung image, wherein the standard lung image can be a three-dimensional image, and the size of the standard lung image can be D×H×W.

[0072] Optionally, the image processing device may truncate voxel values ​​in the original lung image that are outside a preset value range, and normalize the truncated original lung image to obtain a standard lung image, in which the voxel values ​​in the standard lung image may be distributed between 0 Hounsfield units (HU) and 1 HU. For example, the preset value range may be -1000HU to 200HU.

[0073] By normalizing the original lung image, the noise data in the original lung image can be removed to ensure that the voxel value in the standard lung image is within a reasonable range, thereby reducing abnormal data.

[0074] Step 203: Acquire the segmentation result of the lung and trachea based on the standard lung image.

[0075] After the image processing device normalizes the original lung image to obtain a standard lung image, a segmentation result of the pulmonary trachea can be obtained based on the standard lung image. The segmentation result of the pulmonary trachea only includes the pulmonary trachea in the standard lung image. The segmentation result of the pulmonary trachea can be a three-dimensional binary image, and the size of the segmentation result of the pulmonary trachea can be D×H×W. Figure 3 , the voxel value of the trachea in the trachea segmentation result can be 1.

[0076] In an optional implementation, the image processing device may input a standard lung image into a lung-trachea segmentation model, thereby obtaining a lung-trachea segmentation result output by the lung-trachea segmentation model.

[0077] The lung and trachea segmentation model can be determined based on multiple sample standard lung images, multiple first sample lung and trachea segmentation results and a first initial model. The multiple sample standard lung images are all three-dimensional images, and the multiple first sample lung and trachea segmentation results are all three-dimensional binary images.

[0078] The image processing device may pre-store a first initial model, a plurality of sample standard lung images, and a plurality of first sample tracheal segmentation results, wherein the plurality of sample standard lung images correspond one to one with the plurality of first sample tracheal segmentation results, and the size of each first sample tracheal segmentation result is the same as the size of the trachea in a sample standard lung image, and the size of the first sample tracheal segmentation result may be D×H×W. Each first sample tracheal segmentation result includes the trachea in a corresponding sample standard lung image, and the first sample tracheal segmentation result is obtained by annotating the trachea in a corresponding sample standard lung image. The first initial model may be a deep learning model, and the deep learning model may be a convolutional neural network.

[0079] The image processing device can input multiple sample standard lung images into the first initial model to obtain multiple first lung airway segmentation results output by the first initial model, and the multiple first lung airway segmentation results correspond one-to-one to the multiple sample standard lung images.

[0080] The image processing device can input the segmentation result of each first pulmonary trachea and the corresponding segmentation result of the first sample pulmonary trachea into the first loss function to obtain a first loss value output by the first loss function, thereby obtaining multiple first loss values, and adjusting the first initial model based on the sum of the multiple first loss values. The sum of the multiple first loss values ​​is the error between the segmentation result of the first pulmonary trachea and the segmentation result of the first sample pulmonary trachea.

[0081] The image processing device can continuously adjust the parameters of the first initial model based on the error between the first pulmonary trachea segmentation result and the first sample pulmonary trachea segmentation result until the accuracy of the first initial model meets the requirements, and finally determine the first initial model with the required accuracy as the pulmonary trachea segmentation model.

[0082] In another optional implementation, the image processing device may use the main airway in the standard lung image as an initial seed point based on a traditional region growing method, and determine the segmentation result of the lung airway from the standard lung image in combination with a threshold region growing algorithm.

[0083] In another optional implementation, the image processing device can divide the standard lung image into multiple image blocks based on the sliding window method, input each image block into the trachea prediction model, and obtain the prediction result of the voxel in the image block belonging to the trachea. The prediction result of the voxel belonging to the trachea can be the probability that the voxel belongs to the trachea. For each voxel, if the prediction result of the voxel belonging to the trachea is greater than or equal to a preset value, it can be determined that the voxel belongs to the trachea. If the prediction result of the voxel belonging to the trachea is less than a preset value, it can be determined that the voxel does not belong to the trachea, thereby obtaining the segmentation result of the trachea.

[0084] The image processing device may pre-store the size of the image blocks and the overlap ratio between the image blocks. For example, the size of the image blocks may be 96×96×96, and the overlap ratio between the image blocks may be 0.5.

[0085] Optionally, there may be overlap between different image blocks. When there are multiple prediction results for the same voxel, the image processing device may perform weighted averaging on the multiple prediction results, and update the prediction result of the voxel belonging to the pulmonary trachea to the prediction result after the weighted averaging.

[0086] In an embodiment of the present invention, when the standard lung image contains target tissue other than the lung, the image processing device can remove the target tissue in the standard lung image and obtain the segmentation result of the lung trachea based on the processed standard lung image. For example, the target tissue can be the heart.

[0087] Step 204: Obtain the skeleton line of the pulmonary trachea based on the segmentation result of the pulmonary trachea.

[0088] After the image processing device obtains the segmentation result of the trachea based on the standard lung image, the skeleton line of the trachea can be obtained based on the segmentation result of the trachea. The skeleton line of the trachea can be a three-dimensional binary image, and the size of the skeleton line of the trachea can be D×H×W. Figure 4 A schematic diagram of a skeleton line of a lung trachea provided in an embodiment of the present invention, referring to Figure 3 and Figure 4 The skeleton of the pulmonary trachea is Figure 3 The segmentation result of the lung trachea shown is obtained by thinning the lung trachea.

[0089] Optionally, the image processing device may extract the skeleton line of the pulmonary trachea from the segmentation result of the pulmonary trachea based on a skeleton line extraction algorithm. The skeleton line extraction algorithm may be a thinning algorithm.

[0090] The image processing device can perform parallel iterative refinement based on a refinement algorithm, gradually delete simple voxels that do not affect the local connection relationship, and obtain the skeleton line of the lung trachea.

[0091] Step 205: Acquire the image structure of the pulmonary trachea based on the skeleton line of the pulmonary trachea.

[0092] After the image processing device obtains the skeleton line of the trachea based on the segmentation result of the trachea, the graph structure of the trachea can be obtained based on the skeleton line of the trachea. The graph structure may include the adjacency relationship between the tracheas with a connection relationship in the trachea. The trachea in the trachea may include segmental trachea and non-segmental trachea. The segmental trachea may be the tertiary bronchus branched after the lobar bronchus enters the hilum. The segmental trachea may include the left segmental trachea and the right segmental trachea. The non-segmental trachea may include the trunk trachea and the lobar trachea.

[0093] Optionally, the image processing device can use each skeleton line branch in the skeleton line of the lung trachea as a node of the graph structure. For any two nodes, if the skeleton line branches corresponding to the two nodes have the same bifurcation point, it can be determined that the two nodes in the graph structure are connected, that is, the two nodes have a connection relationship. If the skeleton line branches corresponding to the two nodes do not have the same bifurcation point, it can be determined that the two nodes in the graph structure are not connected, that is, the two nodes do not have a connection relationship.

[0094] The image processing device can determine bifurcation points and endpoints from the skeleton line of the pulmonary trachea, and determine the skeleton lines between bifurcation points, between bifurcation points and endpoints, or between endpoints as skeleton line branches.

[0095] like Figure 5 As shown, assuming that the image processing device determines the bifurcation point 11, the first endpoint 12, the second endpoint 13 and the third endpoint 14 from the skeleton line of the lung trachea, the image processing device can determine the skeleton line between the bifurcation point 11 and each of the first endpoint 12 to the third endpoint 14 as a skeleton line branch, thereby obtaining three skeleton line branches. In addition, since the first skeleton line branch between the first endpoint 12 and the bifurcation point 11 and the second skeleton line branch between the second endpoint 13 and the bifurcation point 11 have the same bifurcation point 11, the first skeleton line branch is connected to the second skeleton line branch in the graph structure. Figure 6 In the graph structure shown, each node represents a skeleton line branch, and there is a connection relationship between two connected nodes.

[0096] Step 206: Acquire high-dimensional features of the lung trachea based on the original lung image.

[0097] After the image processing device obtains the graph structure of the pulmonary trachea based on the skeleton line of the pulmonary trachea, the high-dimensional features of the pulmonary trachea can be obtained based on the original lung image.

[0098] Optionally, the image processing device may input the standard lung image and the segmentation results of the lung and trachea into a high-dimensional feature extraction model to obtain high-dimensional features of the lung and trachea.

[0099] Among them, the high-dimensional feature of the pulmonary trachea can be an image, and the size of the image is the same as the size of the standard lung image. The high-dimensional feature can characterize the spatial characteristics of the foreground pixels in the segmentation result of the pulmonary trachea (i.e., the pixels included in the pulmonary trachea in the segmentation result), and the high-dimensional feature can provide richer image details and more accurate image representation in multiple dimensions. The multiple dimensions may include but are not limited to color, texture, shape, spatial relationship, semantic information, etc. Compared with low-dimensional features (such as edges, corners, etc.), high-dimensional features are more abstract and complex, and can more comprehensively describe the content and properties of the pulmonary trachea.

[0100] The high-dimensional feature extraction model can be a neural network model, for example, the neural network model can be a deep neural network model, and the deep neural network model can be a convolutional neural network. Since the lungs and trachea are distributed in three-dimensional space, the deep neural network model can process more complex data, can learn more representative high-dimensional features of the data, and has strong representation ability. Therefore, using a deep neural network model to extract high-dimensional features of the lungs and trachea can obtain more representative spatial features.

[0101] In an embodiment of the present invention, a high-dimensional feature extraction model can be determined by an image processing device using a deep learning method based on multiple sample standard lung images, multiple second sample lung and airway segmentation results, multiple first sample airway segmentation results, and a second initial model. Multiple sample standard lung images, multiple second sample lung and airway segmentation results, and multiple first sample airway segmentation results correspond one to one. The size of each second sample lung and airway segmentation result is the same as the size of the lung and airway in a sample standard lung image, and the size can be D×H×W. The segmentation result of the second sample lung and airway is a three-dimensional binary image.

[0102] Each second sample pulmonary trachea segmentation result includes pulmonary trachea in a corresponding sample standard lung image, and the second sample pulmonary trachea segmentation result is obtained by annotating the pulmonary trachea in the corresponding sample standard lung image. The second sample pulmonary trachea may include segmental trachea and non-segmental trachea, and voxels in the same category of trachea are represented by the same numerical value, and voxel values ​​of different categories of trachea are represented by different numerical values.

[0103] The image processing device can pre-store multiple sample standard lung images, multiple second sample lung trachea segmentation results, multiple first sample trachea segmentation results and a second initial model. The second initial model can be a neural network model, and the neural network model can be a deep neural network model. For example, the deep neural network model can be a convolutional neural network.

[0104] refer to Figure 7The image processing device can input multiple sample standard lung images 20 and multiple second sample lung trachea segmentation results 21 into the second initial model 22, and obtain multiple first trachea segmentation results 23 output by the second initial model 22. The second initial model 22 includes: a high-dimensional feature extraction layer 221 and a classification layer 222. The high-dimensional feature extraction layer 221 is used to extract high-dimensional features of each sample standard lung image 20 and high-dimensional features of each segmentation result 21, and input the high-dimensional features into the classification layer 222 to obtain the segmentation result 23 output by the classification layer 222.

[0105] In the image processing device, the segmentation results of the multiple first tracheae and the segmentation results of the multiple first sample tracheae can be input into the second loss function to obtain multiple second loss values, and the parameters of the second initial model can be continuously adjusted based on the sum of the multiple second loss values ​​until the accuracy of the second initial model meets the requirements, and finally the determined second initial model is determined as the first trachea segmentation model. The sum of the multiple second loss values ​​is the error between the segmentation results of the multiple first tracheae and the segmentation results of the multiple first sample tracheae.

[0106] Afterwards, refer to Figure 7 , the image processing device can delete the classification layer 222 in the first trachea segmentation model to obtain a high-dimensional feature extraction model 24.

[0107] The image processing device can use deep learning to train multiple sample standard lung images, multiple second sample lung and trachea segmentation results, and multiple first sample trachea segmentation results to obtain a first trachea segmentation model. Subsequently, the high-dimensional feature extraction model determined by the first trachea segmentation model is used to extract high-dimensional features of the lung and trachea, thereby assisting doctors in identifying the location of lesions in clinical lung CT images.

[0108] refer to Figure 7 After the image processing device inputs a plurality of sample standard lung images 20 and a plurality of second sample lung trachea segmentation results 21 into the second initial model 22, the second initial model 22 may also output a plurality of key point heat maps 25. Each key point heat map may include a plurality of key points, which are used to segment the trachea.

[0109] In the image processing device, multiple key point heat maps and multiple sample key point heat maps can be input into the second loss function to obtain multiple third loss values, and the parameters of the second initial model can be continuously adjusted based on the sum of the multiple second loss values ​​and the sum of the multiple third loss values ​​until the accuracy of the second initial model meets the requirements, and finally the determined second initial model is determined as the first trachea segmentation model. The sum of the multiple third loss values ​​is the error between the multiple key point heat maps and the multiple sample key point heat maps. Figure 1 One to one, and each sample key point heat map may include multiple sample key points, which are used to segment the trachea.

[0110] In an embodiment of the present invention, while adjusting the parameters of the second initial model using the error between multiple first trachea segmentation results and multiple first sample trachea segmentation results, the parameters of the second initial model are also adjusted using the error between multiple key point heat maps and multiple sample key point heat maps, thereby improving the accuracy of the determined first trachea segmentation model.

[0111] Step 207: Based on the graph structure and high-dimensional features, obtain the skeleton line segmentation result of the trachea in the lung trachea.

[0112] After the image processing device obtains the high-dimensional features of the lung trachea based on the original lung image, the image processing device can obtain the skeleton line segmentation result of the trachea in the lung trachea based on the graph structure and the high-dimensional features. The skeleton line segmentation result of the trachea can include skeleton line segmentation results of 28 types of trachea, and the skeleton line segmentation result of the trachea can include skeleton lines of the trachea, and the voxel points of the skeleton lines of different types of trachea are represented by different numerical values.

[0113] Since the graph structure is the topological structure of the trachea, and the graph structure contains the adjacency relationship between the connected tracheas, the image processing device can accurately distinguish between adjacent tracheas and different categories based on the graph structure. In addition, since high-dimensional features can more comprehensively describe the content and attributes of the lung trachea, and have a stronger ability to represent the characteristics of the lung trachea, the skeleton line segmentation results of the trachea determined based on high-dimensional features can include more representative features of the trachea. Then, the image processing device obtains the skeleton line segmentation results of the trachea in the lung trachea based on the graph structure and high-dimensional features, and can achieve fine distinction between different categories of trachea, thereby ensuring the accuracy of the obtained tracheal skeleton line segmentation results, and further ensuring the accuracy of the subsequent lung segmentation results.

[0114] Furthermore, the method provided in the embodiment of the present invention has high applicability. Even for data with pulmonary tracheal heterogeneity or the presence of serious diseases, the method provided in the embodiment of the present invention can achieve fine distinction between different types of trachea.

[0115] Optionally, the image processing device can input the graph structure and high-dimensional features into the tracheal skeleton segmentation model, thereby obtaining the tracheal skeleton segmentation result output by the tracheal skeleton segmentation model. Among them, the skeleton segmentation model can be a graph convolutional neural network. The tracheal skeleton segmentation model can be obtained by training based on multiple sample graph structures, multiple sample high-dimensional features and multiple sample tracheal skeleton segmentation results. The multiple sample graph structures, multiple sample high-dimensional features and multiple sample tracheal skeleton segmentation results correspond to each other one by one. The image processing device can pre-store multiple sample graph structures, multiple sample high-dimensional features and multiple sample tracheal skeleton segmentation results.

[0116] The embodiment of the present invention utilizes the segmentation results of the lung trachea to construct a graph of the trachea in the lung trachea, and uses a graph convolutional neural network as a classification model to process the graph structure and high-dimensional features to obtain the skeleton line segmentation results of the trachea, which effectively utilizes the advantage of the graph convolutional neural network in effectively processing complex data with a graph structure, compared with the use of physical features (such as coordinates, directions, which have weaker representation capabilities) and machine learning. The use of a graph convolutional neural network can obtain more accurate classification results, thereby further improving the refined segmentation of the trachea, improving the accuracy of the obtained trachea skeleton line segmentation results, and thus improving the accuracy of the subsequent determination of the lung segmentation results.

[0117] Step 208: Determine the trachea segmentation result based on the trachea skeleton segmentation result and the lung trachea segmentation result.

[0118] After the image processing device obtains the skeleton segmentation result of the trachea based on the graph structure and high-dimensional features, it can determine the segmentation result of the trachea based on the skeleton segmentation result of the trachea and the segmentation result of the lung trachea. Among them, the segmentation result of the trachea can be represented by an image, and the size of the image is the same as the size of the segmentation result of the lung trachea. And the segmentation result of the trachea may include segmental trachea and non-segmental trachea, and the voxels of the same category of trachea are represented by the same numerical value, and the voxel values ​​of different categories of trachea are represented by different numerical values. Optionally, the image processing device can calculate the distance between each voxel point in the segmentation result and each skeleton segmentation result in the 28 categories of skeleton segmentation results of the trachea according to the distance field principle and in combination with the segmentation result of the lung trachea, and determine the category of the skeleton segmentation result closest to the foreground voxel point of the trachea. The category of the skeleton segmentation result refers to: the type of voxel point of the skeleton segmentation result. Finally, the image processing device maps the skeleton segmentation result of the trachea to the segmentation result of the lung trachea, that is, determines the category of the foreground voxel point of the trachea as the category of the nearest skeleton segmentation result, thereby extracting the trachea segmentation result from the segmentation result of the lung trachea.

[0119] Step 209: Obtain a lung segmentation result based on the original lung image and the trachea segmentation result.

[0120] After determining the segmentation result of the trachea based on the skeleton line segmentation result of the trachea and the segmentation result of the lung trachea, the image processing device can obtain the lung segmentation result based on the original lung image and the segmentation result of the trachea. The lung segmentation result may include the lung segmentation result of the left lung lobe and the lung segmentation result of the right lung lobe. The lung segment may include 18 types of segment-level trachea, 8 types of segment-level trachea are distributed in the left lung, and 10 types of segment-level trachea are distributed in the right lung.

[0121] In an embodiment of the present invention, the image processing device can map the trachea of ​​the trachea segmentation result to the trachea segmentation result according to the distance field principle and the trachea segmentation result, so as to map different trachea segmentations to the corresponding trachea, thereby obtaining the trachea segmentation result. The trachea segmentation result can include the left trachea segmentation result and the right trachea segmentation result.

[0122] Optionally, for each foreground voxel point in the lung lobe segmentation result, the image processing device may determine the distance between the foreground voxel point and each segment-level trachea in the trachea segmentation result that is classified as the segment-level trachea according to the distance field principle, and then the image processing device may determine the category of the segment-level trachea closest to the foreground voxel point as the category of the foreground voxel point. Thus, the image processing device can obtain the lung segmentation result from the lung lobe segmentation result.

[0123] The image processing device can input the standard lung image into the lung lobe segmentation model to obtain the lung lobe segmentation result output by the lung lobe segmentation model, such as Figure 8 The lung lobe segmentation results are shown.

[0124] The pulmonary lobe segmentation model can be determined based on multiple sample standard lung images, multiple sample pulmonary lobe segmentation results and a third initial model. The image processing device can pre-store the third initial model, multiple sample standard lung images and multiple sample pulmonary lobe segmentation results. Multiple sample standard lung images correspond to multiple sample pulmonary lobe segmentation results one by one, and each sample pulmonary lobe segmentation result is obtained by annotating the pulmonary lobe area in a corresponding sample standard lung image. The third initial model can be a neural network model, and the neural network model can be a convolutional neural network.

[0125] The image processing device can input multiple sample standard lung images into the third initial model to obtain multiple first lung lobe segmentation results output by the third initial model, wherein the multiple first lung lobe segmentation results correspond one-to-one to the multiple sample standard lung images.

[0126] The image processing device can input each first lung lobe segmentation result and the corresponding sample lung lobe segmentation result into the third loss function to obtain a fourth loss value output by the third loss function. The image processing device can thereby obtain multiple fourth loss values, and adjust the third initial model based on the sum of the multiple fourth loss values ​​to obtain a lung lobe segmentation model. The sum of the multiple fourth loss values ​​is the error between the first lung lobe segmentation result and the sample lung lobe segmentation result. The image processing device can continuously adjust the parameters of the third initial model based on the error between the first lung lobe segmentation result and the sample lung lobe segmentation result until the accuracy of the third initial model meets the requirements, and finally determine the third initial model whose accuracy meets the requirements as the lung lobe segmentation model.

[0127] The method provided by the embodiment of the present invention can assist radiologists in locating lung lesions and help radiologists in surgical navigation of lung diseases. The present invention proposes a coarse-to-fine bronchial classification method, which combines three-dimensional image information, does not use physical features, does not use machine learning, and obtains the skeleton line segmentation results of the trachea in the lung trachea based on graph structure and high-dimensional features, which can achieve fine distinction between different categories of trachea, thereby ensuring that the accuracy of the obtained tracheal skeleton line segmentation results is high, and further ensuring the accuracy of the determination of the lung segmentation results.

[0128] In summary, an embodiment of the present invention provides an image processing method. After the image processing device obtains the original lung image, it can obtain the graph structure of the lung trachea based on the original lung image, obtain the skeleton line segmentation result of the trachea based on the original lung image and the graph structure, and obtain the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0129] Since the graph structure is the topological structure of the trachea, and the graph structure includes the adjacency relationship between the connected tracheae, the image processing device can accurately distinguish between adjacent tracheae of different categories based on the graph structure. Based on the original lung image and the graph structure, the accuracy of the obtained tracheal skeleton line segmentation result is high, thereby improving the accuracy of the determined lung segmentation result and improving the accuracy of lung segmentation.

[0130] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented. For example, Figure 1 or Figure 2 The image processing method shown.

[0131] Fig. 9 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention, such as Fig. 9As shown, the image processing device 90 may include a memory 901, a processor 902, and a computer program stored in the memory 901 and executable on the processor 902. When the processor 902 executes the computer program, the image processing method shown in the above embodiment is implemented. Figure 1 or Figure 2 The image processing method shown.

[0132] Fig.10 is a module diagram of an image processing device provided by an embodiment of the present invention, such as Fig. 9 As shown, the device comprises:

[0133] A first acquisition module 1001 is used to acquire an original lung image;

[0134] A second acquisition module 1002 is used to acquire a lung trachea image structure based on the original lung image;

[0135] The third acquisition module 1003 is used to acquire the skeleton line segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure, and the trachea includes the segmental trachea and the non-segmental trachea;

[0136] The fourth acquisition module 1004 is used to acquire a lung segmentation result based on the original lung image and the trachea skeleton line segmentation result.

[0137] Optionally, the first acquisition module 1001 is used to:

[0138] Obtaining lung trachea segmentation results based on the original lung image;

[0139] Obtaining the skeleton line of the pulmonary trachea based on the segmentation result of the pulmonary trachea;

[0140] The image structure of the pulmonary trachea is obtained based on the skeleton line of the pulmonary trachea.

[0141] Optionally, the first acquisition module 1001 is used to:

[0142] Each skeleton line branch in the skeleton line of the lung trachea is used as a node of the graph structure;

[0143] For any two nodes, if the skeleton line branches corresponding to the two nodes have the same bifurcation point, it is determined that the two nodes are connected in the graph structure.

[0144] Optionally, the first acquisition module 1001 is used to:

[0145] Normalize the original lung image to obtain a standard lung image;

[0146] In the case where the standard lung image contains target tissue other than the lung, the target tissue in the standard lung image is removed, and a segmentation result of the lung trachea is obtained based on the processed standard lung image.

[0147] Optionally, the second acquisition module 1002 is used to:

[0148] Obtain high-dimensional features of lung trachea based on original lung images;

[0149] Based on the graph structure and high-dimensional features, the skeleton segmentation results of the trachea in the lung trachea are obtained.

[0150] Optionally, the second acquisition module 1002 is used to:

[0151] Obtaining lung trachea segmentation results based on the original lung image;

[0152] The standard lung image and the segmentation results of the lung and trachea are input into the high-dimensional feature extraction model to obtain the high-dimensional features of the lung and trachea;

[0153] The standard lung image is obtained by normalizing the original lung image.

[0154] Optionally, a fourth acquisition module 1004 is used to: acquire a segmentation result of the trachea based on the skeleton segmentation result of the trachea and the segmentation result of the lung and trachea;

[0155] Based on the original lung image and the trachea segmentation results, a lung segmentation result is obtained.

[0156] In summary, an embodiment of the present invention provides an image processing device. After acquiring the original lung image, the image processing device can acquire the graph structure of the lung trachea based on the original lung image, acquire the skeleton line segmentation result of the trachea based on the original lung image and the graph structure, and acquire the lung segmentation result based on the original lung image and the skeleton line segmentation result of the trachea.

[0157] Since the graph structure is the topological structure of the trachea, and the graph structure includes the adjacency relationship between the connected tracheae, the image processing device can accurately distinguish between adjacent tracheae of different categories based on the graph structure. Based on the original lung image and the graph structure, the accuracy of the obtained tracheal skeleton line segmentation result is high, thereby improving the accuracy of the determined lung segmentation result and improving the accuracy of lung segmentation.

[0158] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0159] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0161] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0162] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0163] In the present invention, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific implementation situation.

[0164] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0165] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain original lung images; Acquiring a lung trachea structure graph based on the original lung image; Based on the original lung image and the graph structure, obtaining a skeleton line segmentation result of the trachea in the lung trachea, wherein the trachea includes a segmental trachea and a non-segmental trachea; A lung segmentation result is obtained based on the original lung image and the trachea skeleton segmentation result.

2. The method according to claim 1, characterized in that Acquiring a lung trachea structure graph based on the original lung image, including: Acquire a segmentation result of the lung and trachea based on the original lung image; Acquire a skeleton line of the pulmonary trachea based on the segmentation result of the pulmonary trachea; The image structure of the pulmonary trachea is acquired based on the skeleton line of the pulmonary trachea.

3. The method according to claim 2, characterized in that Acquiring a graphical structure of the pulmonary trachea based on a skeleton line of the pulmonary trachea includes: Using each skeleton line branch in the skeleton line of the lung trachea as a node of the graph structure; For any two of the nodes, if the skeleton line branches corresponding to the two nodes have the same bifurcation point, it is determined that the two nodes in the graph structure are connected.

4. The method according to claim 2, characterized in that Acquiring a segmentation result of the lung and trachea based on the original lung image, comprising: performing normalization processing on the original lung image to obtain a standard lung image; In a case where the standard lung image includes target tissue other than the lung, the target tissue in the standard lung image is removed, and the lung and trachea segmentation result is acquired based on the processed standard lung image.

5. The method according to any one of claims 1 to 4, characterized in that: Obtaining a skeleton line segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure, including: Acquiring high-dimensional features of the lung trachea based on the original lung image; Based on the graph structure and the high-dimensional features, a skeleton line segmentation result of the trachea in the lung trachea is obtained.

6. The method according to claim 5, characterized in that Acquiring high-dimensional features of the lung trachea based on the original lung image includes: Acquire a segmentation result of the lung and trachea based on the original lung image; Inputting the standard lung image and the segmentation result of the lung and trachea into a high-dimensional feature extraction model to obtain high-dimensional features of the lung and trachea; The standard lung image is obtained by normalizing the original lung image.

7. The method according to any one of claims 1 to 4, characterized in that: Obtaining a lung segmentation result based on the original lung image and the trachea skeleton segmentation result, including: Acquire a segmentation result of the trachea based on the skeleton segmentation result of the trachea and the segmentation result of the lung and trachea; The lung segmentation result is obtained based on the original lung image and the trachea segmentation result.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

9. An image processing device, characterized in that The image processing method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image processing method according to any one of claims 1 to 7 is implemented.

10. An image processing method and device, characterized in that: The device comprises: A first acquisition module is used to acquire an original lung image; A second acquisition module is used to acquire a lung trachea structure image based on the original lung image; a third acquisition module, configured to acquire a skeleton segmentation result of the trachea in the lung trachea based on the original lung image and the graph structure, wherein the trachea includes a segmental trachea and a non-segmental trachea; The fourth acquisition module is used to acquire a lung segmentation result based on the original lung image and the trachea skeleton line segmentation result.

Images