Lung image processing method, device, equipment and storage medium

Abstract

This invention relates to a lung image processing method, apparatus, device, and storage medium. The method includes: performing lung lobe segmentation processing on a lung image to obtain a segmented lung lobe image; performing lesion prediction processing on the segmented lung lobe image to obtain mask information of the predicted lesion region, the mask information including mask values ​​of pixels within the predicted lesion region; forming a cone structure based on the mask values ​​in the mask information; and optimizing the cone structure to obtain an optimized lesion region, the optimized lesion region matching the shape of a lesion. This invention can improve the detection accuracy of lesion regions.

Description

Technical Field

[0001] The present invention relates to the fields of financial technology and artificial intelligence, and in particular to a lung image processing method, device, equipment and storage medium. Background Art

[0002] In recent years, the combination of artificial intelligence and medical imaging can provide great convenience for medical diagnosis. Through artificial intelligence technology, the location of lesions can be determined from medical images. However, in the prior art, when detecting the lesion area, the lesion area is usually determined by a rectangular detection frame. However, the detection frame will include features other than lesion features, such as normal tissue features or other pathological features, which will affect the analysis of lesion features. Therefore, if the above method is used to detect the lesion area, the accuracy of the lesion area detection will be low. Summary of the invention

[0003] The main purpose of the present invention is to provide a lung image processing method, device, equipment and storage medium, aiming to solve the problem of low accuracy of existing lesion area detection.

[0004] To achieve the above objectives, in a first aspect, the present invention provides a lung image processing method, comprising:

[0005] Performing lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image;

[0006] Performing lesion prediction processing on the lung lobe segmentation image to obtain mask information of a predicted lesion area, wherein the mask information includes mask values ​​of pixel points in the predicted lesion area;

[0007] forming a vertebral structure based on the mask value in the mask information;

[0008] The vertebral structure is optimized to obtain an optimized lesion area, and the optimized lesion area matches the lesion shape.

[0009] Optionally, the mask value represents the confidence that the pixel point is located at the lesion position, and the forming of the vertebral structure based on the mask value in the mask information includes:

[0010] Obtaining the three-dimensional coordinates of the pixel point based on the position of the pixel point in the predicted lesion area in the lung image and the mask value corresponding to the pixel point in the mask information;

[0011] The vertebral structure is formed based on the three-dimensional coordinates of the pixel points in the predicted lesion area.

[0012] Optionally, the forming a vertebral structure based on the mask value in the mask information includes:

[0013] Determine a pixel point in the predicted lesion area whose mask value exceeds a first threshold as a first pixel point;

[0014] Determining the vertex of the vertebral structure based on the position of the first pixel point in the predicted lesion area and a preset height value;

[0015] The vertebral structure is formed based on the predicted lesion area and the apex.

[0016] Optionally, the optimizing the vertebral structure to obtain an optimized lesion area includes:

[0017] Performing a planar clustering optimization process on the vertebral structure to obtain an optimized vertebral structure;

[0018] Based on the bottom surface of the optimized vertebral structure, the optimized lesion area of ​​the lung image is obtained.

[0019] Optionally, performing planar clustering optimization processing on the vertebral structure to obtain an optimized vertebral structure includes:

[0020] Performing planar clustering on the first pixel point included in the mask information to obtain the side surface to which the first pixel point belongs;

[0021] Iteratively updating the side surface based on a first pixel point included in the side surface to obtain an optimized side surface;

[0022] The optimized vertebral body structure is formed based on the optimized side faces.

[0023] Furthermore, the method further comprises:

[0024] For each of the lung lobe segmentation images, determining a first region corresponding to a first disease and a second region corresponding to a second disease in the optimized lesion region, wherein the second disease can be unidirectionally converted into the first disease;

[0025] A level of the first disease is determined, and a predicted level of progression of the first disease is determined based on the level of the first disease, the first region, and the second region.

[0026] Optionally, the lung lobe segmentation image is a multi-layer image of the lung lobe, and determining the predicted development level of the first disease based on the level of the first disease, the first area, and the second area includes:

[0027] Acquire a first volume of the first disease on the lung lobe according to the first region in each layer of the lung lobe segmentation image, and acquire a second volume of the second disease on the lung lobe according to the second region in each layer of the lung lobe segmentation image;

[0028] calculating the ratio of the first volume to the second volume for the lung lobe respectively;

[0029] The predicted development level of the lung lobe corresponding to the first disease is determined based on the level of the first disease, the ratio and a preset threshold.

[0030] Optionally, determining the predicted development level of the lung lobe corresponding to the first disease based on the level of the first disease, the ratio and a preset threshold value includes:

[0031] In response to the ratio being greater than the preset threshold, determining the predicted progression level to be a next level lighter than the level of the first disease;

[0032] In response to the ratio being less than or equal to the preset threshold, the predicted progression level is determined to be a level higher than the level of the first disease.

[0033] In a second aspect, the present invention further provides a lung image processing device, the lung image processing device comprising:

[0034] A segmentation module, used for performing lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image;

[0035] A processing module, used for performing lesion prediction processing on the lung lobe segmentation image to obtain mask information of the predicted lesion area, wherein the mask information includes mask values ​​of pixel points in the predicted lesion area;

[0036] A forming module, configured to form a vertebral structure based on the mask value in the mask information;

[0037] The processing module is further used to optimize the vertebral structure to obtain an optimized lesion area, and the optimized lesion area matches the lesion shape.

[0038] Optionally, the mask value represents the confidence that the pixel point is located at the lesion position, and the forming module is specifically used to obtain the three-dimensional coordinates of the pixel point based on the position of the pixel point in the predicted lesion area in the lung image and the mask value corresponding to the pixel point in the mask information; and form the vertebral structure based on the three-dimensional coordinates of the pixel point in the predicted lesion area.

[0039] Optionally, the forming module comprises:

[0040] A determining unit, configured to determine a pixel point in the predicted lesion area whose mask value exceeds a first threshold as a first pixel point;

[0041] The determining unit is further configured to determine the vertex of the vertebral structure based on the position of the first pixel point in the predicted lesion area and a preset height value;

[0042] A forming unit is used to form the vertebral structure based on the predicted lesion area and the vertex.

[0043] Optionally, the processing unit 20 includes:

[0044] A processing unit, configured to perform a planar clustering optimization process on the vertebral structure to obtain an optimized vertebral structure;

[0045] The acquisition unit is used to obtain the optimized lesion area of ​​the lung image based on the bottom surface of the optimized vertebral structure.

[0046] Optionally, the acquisition unit is specifically used to perform planar clustering on the first pixel point contained in the mask information to obtain the side surface to which the first pixel point belongs; based on the first pixel point included in the side surface, iteratively update the side surface to obtain an optimized side surface; and form the optimized vertebral structure based on the optimized side surface.

[0047] Optionally, the device further includes: a determination module, the determination module including:

[0048] A first determination unit is used to determine, for each of the lung lobe segmentation images, a first region corresponding to a first disease and a second region corresponding to a second disease in the optimized lesion region, where the second disease can be unidirectionally converted into the first disease;

[0049] The second determining unit is configured to determine a level of the first disease, and determine a predicted development level of the first disease based on the level of the first disease, the first region, and the second region.

[0050] Optionally, the lung lobe segmentation image is a multi-layer image of the lung lobe, and the first determining unit includes:

[0051] an acquisition subunit, configured to acquire a first volume of the first disease on the lung lobe according to the first region in each layer of the lung lobe segmentation image, and to acquire a second volume of the second disease on the lung lobe according to the second region in each layer of the lung lobe segmentation image;

[0052] a calculation subunit, configured to calculate a ratio of the first volume to the second volume for each of the pulmonary lobes;

[0053] A determination subunit is used to determine the predicted development level of the lung lobe corresponding to the first disease based on the level of the first disease, the ratio and a preset threshold.

[0054] Optionally, the determination subunit is specifically used to determine that the predicted development level is the next level lighter than the level of the first disease in response to the ratio being greater than the preset threshold; or to determine that the predicted development level is the previous level more serious than the level of the first disease in response to the ratio being less than or equal to the preset threshold.

[0055] In a third aspect, the present invention further proposes a lesion area detection device, which comprises: a memory, a processor, and a lesion area detection program stored in the memory and executable on the processor, wherein the lesion area detection program is executed by the processor to perform the steps of the lung image processing method.

[0056] In a fourth aspect, the present invention further proposes a computer-readable storage medium, on which a lesion area detection program is stored, and when the lesion area detection program is executed by a processor, the steps of the lung image processing method are implemented.

[0057] The present invention provides a lung image processing method, device, equipment and storage medium. After segmenting the lung image to obtain a lung lobe segmentation image, the predicted lesion area in the lung lobe segmentation image can be predicted, and mask information corresponding to the predicted lesion area can be obtained. Using the mask information of each pixel point in the predicted lesion area, a vertebral structure can be formed, and the optimized lesion area can be obtained by optimizing the vertebral structure. Since the optimized lesion area matches the shape of the lesion, the feature information other than the lesion feature in the lesion area is reduced. Therefore, the lesion area detection accuracy can be improved, and the subsequent lesion analysis accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a lung image processing method according to an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a network structure of a lung lobe segmentation method according to an embodiment of the present invention;

[0061] Figure 4 Flow chart showing step S20 in the lung image processing method according to an embodiment of the present invention

[0062] Figure 5 is a flow chart of another lung image processing method according to an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of functional modules of a lung image processing device according to an embodiment of the present invention.

[0064] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0066] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0067] It should be noted that the lung image processing method provided in the embodiment of the present invention can be used to improve the detection accuracy of the lesion area in the lung image. The executor of the method can be any lung image processing device. For example, the lung image processing method can be executed by a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., without specific limitation here.

[0068] like Figure 1 As shown, the lung image processing device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0069] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the lung image processing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0070] like Figure 1As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a lung image processing program. The operating system is a program that manages and controls the hardware and software resources of the device, and supports the operation of the lung image processing program and other software or programs.

[0071] exist Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the lung image processing program stored in the memory 1005 and execute the lung image processing method of the embodiment of the present invention.

[0072] Based on the above structure, various embodiments of the lung image processing method are proposed.

[0073] Reference Figure 2 , Figure 2 The figure is a schematic diagram of a lung image processing process according to an embodiment of the present invention.

[0074] S10: performing lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image;

[0075] S20: performing lesion prediction processing on the lung lobe segmentation image to obtain mask information of the lesion area, wherein the mask information includes mask values ​​of pixel points in the lesion area;

[0076] S30: forming a vertebral structure based on the mask value in the mask information;

[0077] S40: Optimizing the vertebral structure to obtain an optimized lesion region, wherein the optimized lesion region matches the lesion shape.

[0078] In an embodiment of the present invention, a lung image may be first acquired. The lung image may be acquired in real time by shooting CT (Computed Tomography), or may communicate with other electronic devices to receive the transmitted lung image. In an embodiment of the present invention, the acquired lung image may be a lung image under three viewing angles, and the three viewing angles include a sagittal plane viewing angle, a coronal plane viewing angle, and a cross-sectional viewing angle. Correspondingly, a plurality of tomographic images, i.e., lung images, may be acquired under each viewing angle, and the lung image under the same viewing angle may be constructed to form a three-dimensional lung image. For example, multiple layers of images under the same viewing angle may be stacked to obtain a three-dimensional lung image, or linear fitting or surface fitting may be performed to obtain a three-dimensional lung image.

[0079] In the following embodiments, the lung image may be a lung image of any layer at any viewing angle, which is not specifically limited in the embodiments of the present invention.

[0080] In addition, when performing pulmonary lobe segmentation processing on the lung image to obtain the pulmonary lobe segmentation image, the obtained pulmonary lobe segmentation image may include at least one of: a left lung image and a right lung image. The left lung image includes: a left upper lobe and a left lower lobe; the right lung image includes: a right upper lobe, a right middle lobe, and a right lower lobe, that is, the pulmonary lobe segmentation image is a pulmonary lobe image corresponding to any one or more pulmonary lobes of the five pulmonary lobes. The embodiment of the present invention can obtain segmentation images of the five pulmonary lobes respectively through pulmonary lobe segmentation processing.

[0081] In some possible implementations, a lung image is subjected to lung lobe segmentation processing to obtain a lung lobe segmentation image, comprising: step 1: acquiring the lung lobe fissure features in the sagittal plane, the lung lobe fissure features in the coronal plane, and the lung lobe fissure features in the transverse plane of the lung image; step 2: correcting the third lung lobe fissure feature using any two of the lung lobe fissure features in the sagittal plane, the coronal plane, and the transverse plane; and step 3: segmenting the lung image using the corrected lung lobe fissure features.

[0082] Step 1: Obtain the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the transverse plane.

[0083] In some possible implementations, the lobe fissure features of the lung images at different viewing angles may be extracted by feature extraction processing. The lobe fissure features are features used to perform segmentation processing of each lobe region in the lung image. For example, the lobe fissure features may be used to determine the position of the fissure surface between the lobes to achieve lobe segmentation.

[0084] The embodiment of the present invention can perform feature extraction processing on the lung images in the sagittal plane, coronal plane and cross-sectional perspectives respectively, and obtain the fissure features of the lung images in the corresponding perspectives, that is, the lobe fissure features of the lung images in the sagittal plane, the lobe fissure features in the coronal plane and the lobe fissure features in the cross-section can be obtained respectively. In the embodiment of the present invention, the lobe fissure features in each perspective can be expressed in the form of a matrix or a vector, and the lobe fissure features can represent the characteristic values ​​of the lung images at each pixel point in the corresponding perspective.

[0085] In some possible implementations, feature extraction processing of fissure features of lung images can be performed by a feature extraction neural network. For example, a neural network can be trained to achieve accurate extraction of lobe fissure features of lung images by the neural network, and lobe segmentation can be performed using the obtained features. When the accuracy of lobe segmentation exceeds the accuracy threshold, it means that the accuracy of the lobe fissure features obtained by the neural network meets the requirements. At this time, the network layer that performs segmentation in the neural network can be removed, and the retained network portion can be used as a feature extraction neural network of an embodiment of the present invention. Among them, the feature extraction neural network can be a convolutional neural network, such as a residual network, a pyramid feature network, or a U network. The above is only an exemplary description and is not a specific limitation of the present invention.

[0086] Step 2: Correct the third lobe fissure feature using any two lobe fissure features in the sagittal plane, coronal plane and transverse plane.

[0087] In some possible implementations, when the lobe fissure features at three perspectives are obtained, the lobe fissure features at two perspectives can be used to correct the lobe fissure features at the third perspective to improve the accuracy of the lobe fissure features at the third perspective.

[0088] In one example, an embodiment of the present invention can use the lobe fissure features in the coronal and cross-sectional perspectives to correct the lobe fissure features in the sagittal perspective. In other embodiments, another lobe fissure feature can also be corrected by any two of the lobe fissure features in the three perspectives. For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is described in the following embodiment. The first lobe fissure feature, the second lobe fissure feature and the third lobe fissure feature correspond to the lobe fissure features in the three perspectives of the embodiment of the present invention, respectively.

[0089] In some possible implementations, the first lobe fissure feature and the second lobe fissure feature can be converted to the perspective of the third lobe fissure feature by mapping, and the two lobe fissure features obtained by mapping can be used to perform feature fusion with the third lobe fissure feature to obtain a corrected lobe fissure feature.

[0090] Step 3: Segment the lung image using the corrected lobe fissure features.

[0091] In some possible implementations, lobe segmentation can be performed directly using the corrected lobe fissure feature to obtain a segmentation result of the lobe fissure. Alternatively, in other implementations, the corrected lobe fissure feature can be subjected to feature fusion processing with the third lobe fissure feature, and lobe segmentation is performed based on the fusion result to obtain a segmentation result of the lobe fissure. The segmentation result may include the position information corresponding to each partition in the identified lung image. For example, the lung image may include five lobe regions, namely, the right upper lobe, the right middle lobe, the right lower lobe, the left upper lobe, and the left lower lobe, and the segmentation result may include the position information of the above five lobes in the lung image. The embodiment of the present invention may represent the segmentation result by means of a mask feature, that is, the segmentation result obtained by the embodiment of the present invention may be a feature represented in the form of a mask. For example, the embodiment of the present invention may assign unique corresponding mask values, such as 1, 2, 3, 4, and 5, to the above five lobe regions, and the area formed by each mask value is the location area where the corresponding lobe is located. The above mask values ​​are only exemplary, and other mask values ​​may also be configured in other embodiments.

[0092] Based on the above embodiments, the features of the lung lobe fissures under three perspectives can be fully integrated to improve the information content and accuracy of the corrected fissure features, thereby improving the accuracy of the lung lobe segmentation results.

[0093] In order to explain the embodiment of the present invention in detail, each process of the embodiment of the present invention is described below respectively.

[0094] In the embodiment of the present invention, the method for obtaining the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the cross section is:

[0095] A multi-sequence lung image in the sagittal plane, the coronal plane and the cross-section is obtained; and lobe fissure features are extracted from the multi-sequence lung images in the sagittal plane, the coronal plane and the cross-section, respectively, to obtain lobe fissure features in the sagittal plane, lobe fissure features in the coronal plane and lobe fissure features in the cross-section.

[0096] The embodiment of the present invention can first obtain multi-sequence lung images under three viewing angles. As described in the above embodiment, multi-layer lung images (multi-sequence images) of the lung images under different viewing angles can be collected by CT imaging, and a three-dimensional lung image can be obtained through the multi-layer lung images under each viewing angle.

[0097] When a multi-sequence lung image under three viewing angles is obtained, feature extraction processing can be performed on each lung image, for example, the lung image under each viewing angle is subjected to feature extraction processing by the feature extraction neural network, and the lobe fissure features of each image under the three viewing angles are obtained, such as the lobe fissure features under the sagittal plane, the lobe fissure features under the coronal plane, and the lobe fissure features under the cross section. Among them, since each viewing angle may include multiple lung images, the embodiment of the present invention can perform feature extraction processing of the multiple lung images in parallel through multiple feature extraction neural networks, thereby improving feature extraction efficiency.

[0098] Figure 3 FIG. 4 is a schematic diagram showing a network structure of a lung lobe segmentation method according to an embodiment of the present invention. Figure 3 As shown, the network that performs feature extraction processing in the embodiment of the present invention can be a U network (U-net) or other convolutional neural networks that can perform feature extraction.

[0099] When the pulmonary lobe fissure features of the lung images under various viewing angles are obtained, the third pulmonary lobe fissure feature can be corrected using any two pulmonary lobe fissure features under the sagittal plane, the coronal plane, and the transverse plane. The process may include:

[0100] The arbitrary two pulmonary lobe fissure features are mapped to the viewing angle of the third pulmonary lobe fissure feature; and the third pulmonary lobe fissure feature is corrected using the mapped arbitrary two pulmonary lobe fissure features.

[0101] For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is taken as an example to illustrate below.

[0102] Since the extracted lobe fissure features are different under different viewing angles, an embodiment of the present invention can convert the lobe fissure feature maps under three viewing angles to one viewing angle. Among them, the method of mapping any two lobe fissure features to the viewing angle where the third lobe fissure feature is located is: mapping the lobe fissure features of any two multi-sequence lung images of the sagittal plane, coronal plane and cross-section to the viewing angle where the third lobe fissure feature is located. In other words, the first lobe fissure feature and the second lobe fissure feature can be converted to the viewing angle where the third lobe fissure feature is located. Among them, through the mapping conversion of the viewing angle, the feature information of the viewing angle before mapping can be integrated into the lobe fissure feature obtained after mapping.

[0103] As described in the above embodiment, the embodiment of the present invention can obtain multiple lung images at each viewing angle, and the multiple lung images correspond to multiple pulmonary lobe fissure features. Each feature value in the pulmonary lobe fissure feature corresponds one-to-one to each pixel point of the corresponding lung image.

[0104] The embodiment of the present invention can determine the position mapping relationship between each pixel point in the lung image when the perspective is converted to another perspective based on the three-dimensional lung image formed by multiple lung images at one perspective, such as when a certain pixel point switches from the first position of the first perspective to the second position of the second perspective, at this time, the feature value corresponding to the first position at the first perspective is mapped to the second position. Through the above embodiment, the mapping conversion between the lung lobe fissure features of each lung image at different perspectives can be achieved.

[0105] In some possible implementations, when the lobe fissure features of three perspectives are mapped to the same perspective, the two mapped lobe fissure features can be used to perform correction processing on the third lobe fissure feature to improve the information content and accuracy of the third lobe fissure feature.

[0106] In the embodiment of the present invention, the method for correcting the third lobe fissure feature by using any two of the mapped lobe fissure features is:

[0107] The spatial attention feature fusion is performed using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure feature to obtain a first fusion feature and a second fusion feature; and the corrected third pulmonary lobe fissure feature is obtained based on the first fusion feature and the second fusion feature.

[0108] In the embodiment of the present invention, the feature after the first lobe fissure feature is mapped may be referred to as the first mapping feature, and the feature after the second lobe fissure feature is mapped may be referred to as the second mapping feature. When the first mapping feature and the second mapping feature are obtained, a spatial attention feature fusion between the first mapping feature and the third lobe fissure feature may be performed to obtain a first fused feature, and a spatial attention feature fusion between the second mapping feature and the third lobe fissure feature may be performed to obtain a second fused feature.

[0109] The method of fusing the spatial attention features by respectively using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure to obtain the first fused feature and the second fused feature is:

[0110] Respectively connect any two of the lobe fissure features with the third lobe fissure feature to obtain a first connection feature and a second connection feature; perform a first convolution operation on the first connection feature to obtain a first convolution feature, and perform a first convolution operation on the second connection feature to obtain a second convolution feature; perform a second convolution operation on the first convolution feature to obtain a first attention coefficient, and perform a second convolution operation on the second convolution feature to obtain a second attention coefficient; use the first convolution feature and the first attention coefficient to obtain the first fusion feature, and use the second convolution feature and the second attention coefficient to obtain the second fusion feature.

[0111] In some possible implementations, such as Figure 3 As shown, the above-mentioned spatial attention feature fusion processing can be performed by the network module of the spatial attention mechanism. The embodiment of the present invention takes into account the different importance of lobe fissure features at different positions and adopts a spatial attention mechanism. Among them, the convolution processing based on the attention mechanism can be implemented through the spatial attention neural network (attention), and the important features are further highlighted in the obtained fusion features. In the training process of the spatial attention neural network, the importance of each position of the spatial feature can be adaptively learned to form an attention coefficient with the feature object at each position. For example, the coefficient can represent a coefficient value in the interval [0,1]. The larger the coefficient, the more important the feature at the corresponding position.

[0112] In the process of performing spatial attention fusion processing, the first mapping feature and the third lobe fissure feature can first be concatenated to obtain a first connection feature, and the second mapping feature and the third lobe fissure feature can be concatenated to obtain a second connection feature. The above connection processing can be concatenated in the channel direction. In an embodiment of the present invention, the scales of the first mapping feature, the second mapping feature, and the third lobe fissure feature can all be identified as (C / 2, H, W), where C represents the number of channels of each feature, H represents the height of the feature, and W represents the width of the feature. Correspondingly, the scales of the first connection feature and the second connection feature obtained by the connection processing can be expressed as (C, H, W).

[0113] When the first connection feature and the second connection feature are obtained, the first convolution operation can be performed on each of the first connection feature and the second connection feature, such as using the convolution layer A to perform the first convolution operation through a 3*3 convolution kernel, and then batch normalization (bn) and activation function (relu) processing can be performed to obtain the first convolution feature corresponding to the first connection feature and the second convolution feature corresponding to the second connection feature. The scale of the first convolution feature and the second convolution feature can be expressed as (C / 2, H, W). The first convolution operation can reduce the parameters in the feature map and reduce the subsequent calculation cost.

[0114] In some possible implementations, when the first convolution feature and the second convolution feature are obtained, the second convolution operation and the sigmoid function processing can be performed on the first convolution feature and the second convolution feature respectively to obtain the corresponding first attention coefficient and second attention coefficient respectively. The first attention coefficient can represent the importance of the features of each element of the first convolution feature, and the second attention coefficient can represent the importance of the features of the elements in the second convolution feature.

[0115] like Figure 3As shown, for the first convolution feature or the second convolution feature, two convolution layers B and C can be used to perform the second convolution operation, wherein the convolution layer B is processed by a 1*1 convolution kernel, and then batch normalization (bn) and activation function (relu) are performed to obtain the first intermediate feature. The scale of the first intermediate feature map can be expressed as (C / 8, H, W), and then the second convolution layer C is used to perform a convolution operation of a 1*1 convolution kernel on the first intermediate feature map to obtain a second intermediate feature map of (1, H, W). Further, the second intermediate feature map can be processed by an activation function using a sigmoid function to obtain the attention coefficient corresponding to the first convolution feature or the second performance feature, and the coefficient value of the attention coefficient can be a value in the range of [0,1].

[0116] Through the above-mentioned second convolution operation, the first connection feature and the second connection feature can be subjected to dimensionality reduction processing to obtain the attention coefficient of a single channel.

[0117] In some possible implementations, when a first attention coefficient corresponding to a first convolution feature and a second attention coefficient corresponding to a second convolution feature are obtained, a product process may be performed on the first convolution feature and the first attention coefficient, and the product result may be added to the first convolution feature to obtain a first fused feature. And a product process may be performed on the second convolution feature and the second attention coefficient matrix, and the product result may be added to the second convolution feature to obtain a second fused feature. Among them, the product process (mul) may be corresponding element multiplication, and the feature addition (add) may be corresponding element addition. In the above manner, effective fusion of features from three perspectives may be achieved.

[0118] Alternatively, in some other implementations, the feature obtained by multiplying the first convolution feature by the first attention coefficient may be added to the first convolution feature, and the added feature may be subjected to several convolution operations to obtain the first fused feature; and the feature obtained by multiplying the second convolution feature by the second attention coefficient may be added to the second convolution feature, and the added feature may be subjected to several convolution operations to obtain the second fused feature. In this way, the accuracy of the fused feature may be further improved, and the information content of the fusion may be improved.

[0119] When the first fusion feature and the second fusion feature are obtained, the first fusion feature and the second fusion feature may be used to obtain a corrected third lobe fissure feature.

[0120] In some possible implementations, since the first fused feature and the second fused feature respectively include feature information from three perspectives, the first fused feature and the second fused feature can be directly connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature. Alternatively, the first fused feature, the second fused feature, and the third lobe fissure feature can be connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature.

[0121] The third convolution operation may include group convolution processing. The third convolution operation may further achieve further fusion of feature information in each feature. Figure 3 As shown, the third convolution operation of the embodiment of the present invention may include grouped convolution D (depth wise conv), wherein the grouped convolution can speed up the convolution speed while improving the accuracy of the convolution features.

[0122] When the corrected third lobe fissure feature is obtained through the third convolution operation, the lung image can be segmented using the corrected lobe fissure feature. The embodiment of the present invention can obtain the segmentation result corresponding to the corrected lobe fissure feature by convolution. Figure 3 As shown, in the embodiment of the present invention, the corrected lobe fissure features can be input into the convolution layer E, and the standard convolution is performed through the 1*1 convolution kernel to obtain the segmentation result of the lung image. As described in the above embodiment, the segmentation result can represent the location areas where the five lobes in the lung image are located. Figure 3 As shown, each lung lobe area in the lung image is distinguished by light and dark filling colors.

[0123] Based on the above configuration, the lung lobe segmentation method based on multiple perspectives provided by the embodiment of the present invention can solve the technical problem of not fully utilizing the information from other perspectives to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0124] As described in the above embodiments, the embodiments of the present invention can be implemented by a neural network, such as Figure 3 As shown, the neural network for executing the lung lobe segmentation method under multi-viewpoints in an embodiment of the present invention may include a feature extraction neural network, a spatial attention neural network, and a segmentation network (including convolutional layers D and E).

[0125] The embodiment of the present invention may include three feature extraction neural networks, which are respectively used to extract the features of the lung lobe fissures under different viewing angles. Among them, the three feature extraction networks can be referred to as the first branch network, the second branch network and the third branch network. Among them, the structures of the three branch networks of the embodiment of the present invention are exactly the same, and the input images of each branch network are different. For example, the sagittal lung image sample is input to the first branch network, the coronal lung image sample is input to the second branch network, and the transverse lung image sample is input to the third branch network, so as to respectively perform feature extraction processing of the lung image samples under each viewing angle.

[0126] Specifically, in the embodiment of the present invention, the process of training the feature extraction neural network includes:

[0127] Acquire training samples in the sagittal plane, the coronal plane and the transverse plane, wherein the training samples are lung image samples with marked lobe fissure features; use the first branch network to perform feature extraction on the lung image samples in the sagittal plane to obtain a first predicted lobe fissure feature; use the second branch network to perform feature extraction on the lung image samples in the coronal plane to obtain a second predicted lobe fissure feature; use the third branch network to perform feature extraction on the lung image samples in the transverse plane to obtain a third predicted lobe fissure feature; use the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature and the corresponding marked lobe fissure feature to obtain the network losses of the first branch network, the second branch network and the third branch network, and use the network losses to adjust the parameters of the first branch network, the second branch network and the third branch network.

[0128] As described in the above embodiment, the first branch network, the second branch network and the third branch network are respectively used to perform feature extraction processing of lung image samples in the sagittal plane, the coronal plane and the cross-sectional perspective, and then the corresponding predicted features, namely the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature, can be obtained.

[0129] When each predicted lobe fissure feature is obtained, the first predicted lobe fissure feature, the second predicted lobe fissure feature, and the third predicted lobe fissure feature can be used together with the corresponding labeled lobe fissure feature to obtain the network losses of the first branch network, the second branch network, and the third branch network. For example, the loss function of the embodiment of the present invention can be a logarithmic loss function, and the network loss of the first branch network can be obtained by the first predicted lobe fissure feature and the labeled real lobe fissure feature, the network loss of the second branch network can be obtained by the second predicted lobe fissure feature and the labeled real lobe fissure feature, and the network loss of the third branch network can be obtained by the third predicted lobe fissure feature and the labeled real lobe fissure feature.

[0130] When the network loss of each branch network is obtained, the parameters of the first branch network, the second branch network and the third branch network can be adjusted according to the network loss of each network until the termination condition is met. Among them, the embodiment of the present invention can use the network loss of any branch of the first branch network, the second branch network and the third branch network to adjust the network parameters of the first branch network, the second branch network and the third branch network, such as convolution parameters, etc., respectively. Thereby, the network parameters under any perspective can be correlated with the features under the other two perspectives, the correlation between the extracted lobe fissure features and the lobe fissure features under the other two perspectives can be improved, and the preliminary fusion of the lobe fissure features under each perspective can be achieved.

[0131] In addition, the training termination condition of the feature extraction neural network is that the network loss of each branch network is less than the first loss threshold, which indicates that each branch network of the feature extraction neural network can accurately extract the lobe fissure features of the lung image under the corresponding viewing angle.

[0132] When the feature extraction neural network is trained, the feature extraction neural network, the spatial attention neural network and the segmentation network can be used for training at the same time, and the segmentation result output by the segmentation network and the corresponding marking result in the marked lobe fissure feature can be used to determine the network loss of the entire neural network. The network loss of the entire neural network is further used to feedback and adjust the network parameters of the feature extraction neural network, the spatial attention neural network and the segmentation network until the network loss of the entire neural network is less than the second loss threshold. In the embodiment of the present invention, the first loss threshold is greater than or equal to the second loss threshold, so that the network accuracy of the network can be improved.

[0133] When applying the neural network of the embodiment of the present invention to perform lung lobe segmentation based on multiple perspectives, the lung images of the same lung under different perspectives can be input into the three branch networks respectively, and finally the final lung image segmentation result can be obtained through this neural network.

[0134] In summary, the lung lobe segmentation method provided in the embodiment of the present invention can fuse multi-view feature information and perform lung lobe segmentation of lung images, thereby solving the problem of not fully utilizing information from other viewpoints to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0135] When a segmented lung lobe image corresponding to a lung image is obtained, a lesion prediction can be performed on the segmented lung lobe image, that is, whether a preset lesion exists in the segmented lung lobe image. The type of lesion in the embodiment of the present invention can be set according to demand, for example, it can be at least one of a lung nodule and lung cancer, or it can be other types, and the present invention does not specifically limit this. It should be noted that the segmented lung lobe image can be an image corresponding to any lung lobe of the total lung image, and the embodiment of the present invention can detect the lesion area of ​​any lung lobe.

[0136] Figure 4 The flowchart of step S20 in the lung image processing method of the embodiment of the present invention is shown, wherein the lesion prediction processing is performed on the lung lobe segmentation image to obtain mask information of the predicted lesion area. The process may include:

[0137] S201: performing lesion detection processing on the lung lobe segmentation image to determine a predicted lesion area of ​​the lung lobe segmentation image;

[0138] S202: Perform feature processing on the first feature map corresponding to the predicted lesion area to obtain mask information of the lesion area.

[0139] The mask information includes mask values ​​of multiple pixels in the lesion area, and the mask value can represent the confidence that the position of the pixel corresponds to the lesion position, and the mask value is a numerical value greater than or equal to zero and less than or equal to 1.

[0140] In some possible implementations, the obtained lobe segmentation image can be subjected to lesion detection processing to detect the location of the lesion area in the image. The lesion area can be the area where the lung nodules are located, or the area where the lung cancer information is located. The lobe segmentation image can be processed by a convolutional neural network, such as by a region of interest (ROI) and other means to obtain the lesion area. In some examples, the area defined by the output limiting box is used as the lesion area, and the limiting box can be a rectangular box, but the present invention does not specifically limit this.

[0141] In an embodiment of the present invention, performing lesion detection processing on the lobe segmentation image to determine the predicted lesion area of ​​the lobe segmentation image may include: inputting the lobe segmentation image into a region candidate network (RPN), using the region candidate network to perform lesion area detection, and obtaining the lesion area in the lobe segmentation image. That is, in an embodiment of the present invention, a region candidate network (RPN) may be trained to perform lesion area detection to obtain a detection frame corresponding to the lesion, i.e., the lesion area. In other embodiments, the determination of the lesion area may also be performed by other neural networks capable of performing target detection, such as a residual network, etc. The present invention does not make specific limitations on this. Among them, the predicted lesion area in the embodiment of the present invention may be a lesion area of ​​a lung nodule or lung cancer, but this is not a specific limitation of the present invention. In other embodiments, it may also be a lesion area of ​​other types.

[0142] Among them, the region candidate network of the embodiment of the present invention may include a first convolution group A, a second convolution group B, and a classifier C. Among them, the first convolution group includes at least one convolution layer, which is used to perform feature extraction processing of the input lobe segmentation image to obtain the feature map F1 of the lobe segmentation image. The first convolution group can be a residual network structure or a pyramid feature extraction network structure, and the present invention does not specifically limit this. The second convolution group B may also include at least one convolution layer, which is used to perform convolution processing on the input feature map F1 to obtain a detection frame of the predicted lesion area in the lobe segmentation image. The detection frame may be a rectangular frame, which is used to indicate the location of the lesion, that is, corresponding to the predicted lesion area. The classifier C can perform classification of the lesion type in the detection frame, such as whether the lesion is lung cancer or a lung nodule. The above configuration can realize the determination of the predicted lesion area in the lobe segmentation image. Among them, the convolution kernel size of the first convolution group and the second convolution group can be set according to demand, such as 1*1 or 3*3, and the present invention does not specifically limit this.

[0143] In some possible implementations, since the predicted lesion region includes more background information, and the shape of the lesion region is not related to the actual lesion shape, the detection frame of the lesion region is a rectangular frame regardless of the shape of the lesion, which may contain other information of the lesion information, and the detected lesion region may be inaccurate. In order to further improve the detection accuracy of the lesion region, the embodiment of the present invention can optimize the predicted lesion region based on the predicted lesion region to form a final lesion region that matches the lesion shape.

[0144] Among them, the first feature map corresponding to the predicted lesion area can be processed to further obtain mask information corresponding to the predicted lesion area. The mask information can include mask values ​​corresponding to each pixel in the lesion area, which can be specifically expressed as a mask matrix or a mask map or other forms, but the embodiment of the present invention is not limited to this. The mask value in the mask information can be a value in the range of [0,1]. In the related art, the corresponding mask value obtained in the mask information is usually 0 or 1. The embodiment of the present invention can represent the confidence that the corresponding pixel in the predicted lesion area is the lesion information by generating a mask value in the range of [0,1]. It can also represent the distance of the pixel from the center of the lesion area. For example, the higher the mask value, the higher the probability that the corresponding pixel is the lesion information, and the closer the distance to the center of the lesion information, but the embodiment of the present invention is not limited to this.

[0145] In some possible implementations, the first feature map of the predicted lesion area can be subjected to convolution processing and / or other processing through at least one convolution layer to obtain the above-mentioned mask information. Among them, first, the first feature map corresponding to the predicted lesion area can be obtained. For example, feature extraction can be performed on the image area of ​​the predicted lesion area in the lobe segmentation image to obtain the first feature map corresponding to the lesion area. Alternatively, feature extraction can be performed on the lobe segmentation image to obtain the feature map of the lobe segmentation image, and the feature corresponding to the position of the predicted lesion area in the feature map is determined as the first feature map. The above is only an exemplary explanation of the method of obtaining the first feature map, and is not a specific limitation of the present invention. Feature extraction can be achieved through a feature extraction network, for example, it can be achieved through a residual network, a pyramid network, etc.

[0146] In some possible implementations, when a first feature map of a predicted lesion area is obtained, feature processing can be performed on the first feature map to obtain corresponding mask information. The acquisition of the mask information can be achieved through a mask prediction neural network, wherein the mask prediction neural network may include multiple convolutional layers. At least one layer of convolution processing can be performed on the first feature map to obtain a corresponding mask feature map, that is, predict the mask information, and the size and parameters of the convolution kernel of each convolution layer can be parameters obtained by training. Or in some embodiments, feature processing can also be performed on the first feature map in combination with hole convolution and standard convolution (using a convolution kernel without holes to perform convolution processing) to obtain mask information. For example, n layers of hole convolution can be performed on the first feature map, and then standard convolution can be performed, where n is a value greater than or equal to 1, such as n can be 4, but it is not a specific limitation of the present invention. By performing convolution processing in combination with hole convolution, the perception field of view can be expanded and the accuracy of the mask value can be improved. In addition, in some possible implementations, in the process of performing feature processing on the first feature map, including upsampling processing of feature information, an embodiment of the present invention can convert the transposed convolution layer used in the prior art into a bilinear interpolation and convolution processing method, wherein the convolution processing can be performed by a 3*3 convolution kernel, and the present invention does not make specific limitations on this.

[0147] In an embodiment of the present invention, different existing mask prediction neural networks Mask RCNN obtain a mask map composed of 0 or 1, where 1 represents the area where the detection object is located, and 0 represents the area outside the detection object. The embodiment of the present invention trains the MaskRCNN network so that the elements (mask values) in the mask feature map output by the Mask RCNN can be the confidence that the pixel point in the lesion area is the area where the lesion is located. At the same time, the mask value also represents the distance between the corresponding pixel point and the boundary of the lesion. The mask value is composed of a value in the range of [0,1]. That is to say, the mask value in the mask information obtained in the embodiment of the present invention is a value in the range of [0,1].

[0148] In some possible implementations, when training the Mask RCNN network or training a neural network for performing lesion prediction according to an embodiment of the present invention, the true mask value corresponding to the pixel point can be determined according to the distance between the pixel point and the boundary of the true lesion. The Mask RCNN is supervised and trained using the true mask value as supervision.

[0149] After obtaining the mask information corresponding to the lesion area predicted by the lung lobe segmentation image, the vertebral structure can be formed using the mask information, that is, executing step S30.

[0150] In some possible implementations, since the predicted pixel points in the lesion area correspond one-to-one to the mask values ​​in the mask information, the embodiments of the present invention can construct a pyramidal structure using the positions of the pixel points and the mask values. The pyramidal structure can be a pyramid, and the pyramidal structure can include a bottom surface and a plurality of inclined side surfaces connected to the bottom surface. When the plurality of inclined side surfaces intersect at one point, a pyramid is formed. When the inclined side surfaces do not intersect, a prism can be formed. In some possible embodiments, the bottom surface can be constructed as an n-gon, where n is a value greater than or equal to 3. The following embodiments are described with the bottom surface being a quadrilateral, but the present invention does not impose specific limitations on this. The principles of the bottom surfaces with other numbers of sides are the same and will not be repeated.

[0151] In some possible implementations, the mask value corresponding to the pixel point in the lesion area can be used as the height corresponding to the pixel point, and the embodiment of the present invention can construct a vertebral structure according to each height value. In one example, the maximum mask value among the mask values ​​corresponding to each pixel point can be determined. In the case where there is only one pixel point with the maximum mask value, the height corresponding to the mask value of the pixel point can be determined as the height of the vertex of the vertebral structure, that is, the point at the height corresponding to the maximum mask value of the pixel point can be used as the vertex of the vertebral structure, and the lesion area is used as the bottom surface of the main structure, so that the vertebral structure can be determined, and the vertebral structure is a polygonal pyramid at this time. Alternatively, the mask value (height) corresponding to the pixel point at the center of the lesion area can be used to determine the vertex of the vertebral structure, thereby obtaining a vertebral structure. At this time, the vertebral structure is also a polygonal pyramid.

[0152] In another example, the maximum mask value among the mask values ​​corresponding to each pixel point can be determined. When there are multiple pixel points with the maximum mask value, the height corresponding to the mask values ​​of the multiple pixel points can be used to determine the height of the top surface of the vertebral structure, that is, the top surface of the vertebral structure can be constructed using the points at the height corresponding to the maximum mask value of the multiple pixel points, and the lesion area is used as the bottom surface of the vertebral structure, so that the vertebral structure can be determined. At this time, the vertebral structure is a polygonal pyramid.

[0153] In some possible implementations, the mask value of each pixel in the predicted lesion area included in the mask information can also be used to construct the three-dimensional coordinates for each pixel, and the vertebral structure can be constructed using each three-dimensional coordinate. The specific principle is the same as the principle of determining the vertebral structure according to the height of each pixel. The embodiment of the present invention can use the position of each pixel in the predicted lesion area and the corresponding mask value to establish the three-dimensional coordinate value corresponding to each pixel, so that each pixel in the predicted lesion area can be converted into a point set of (x, y, z), where (x, y, z) can be a coordinate form under a three-dimensional coordinate system, (x, y) represents the position of the pixel in the lesion area, and z represents the mask value corresponding to the pixel. The vertebral structure can be established by the three-dimensional coordinates corresponding to each pixel, wherein the three-dimensional coordinates corresponding to the pixel with the largest z value can be used as the vertex of the vertebral structure, or the plane constructed by the three-dimensional coordinates corresponding to multiple pixels with the largest z value can be used as the top surface of the vertebral structure, thereby constructing a vertebral structure in the form of a multi-sided pyramid or a multi-sided pyramid.

[0154] The above is only an exemplary embodiment of determining the pyramidal structure based on the mask value of the pixel point. In other embodiments, the pyramidal structure can also be constructed in other ways, such as using the pixel points that meet the preset conditions to construct the pyramidal structure, that is, selecting the first pixel point that meets the preset conditions from the pixel points, and constructing the pyramidal structure according to the mask value corresponding to the first pixel point or the preset configuration information.

[0155] The process of forming the vertebral structure is described in detail below. For example, in the above embodiment, the vertebral structure can be constructed based on the mask value corresponding to each pixel point in the lesion area as the height of the pixel point, or a three-dimensional coordinate can be formed based on the position of each pixel point in the lesion area and the corresponding mask value, thereby constructing the vertebral structure. In addition, the embodiment of the present invention can also determine a first pixel point that meets the conditions from the pixel points, and determine the fixed point or top surface of the vertebral structure through the first pixel point, thereby constructing the vertebral structure.

[0156] In an embodiment of the present invention, forming a vertebral structure based on the mask value in the mask information may include: determining a pixel point in the predicted lesion area whose mask value exceeds a first threshold as a first pixel point; determining the vertex of the vertebral structure based on the position of the first pixel point in the predicted lesion area and a preset height value; and forming the vertebral structure based on the predicted lesion area and the vertex.

[0157] In some possible implementations, as described above, the mask value corresponding to each pixel in the predicted lesion area can be a value in the range of 0 to 1, and the embodiment of the present invention can select a pixel whose mask value is greater than a first threshold, and determine the pixel whose mask value is greater than the first threshold as the first pixel. The first threshold can be a value less than 0.5, such as 0.01, but the present invention does not specifically limit this. By selecting a first pixel with a larger mask value, the number of pixels in the subsequent clustering process can be reduced without affecting the construction of the vertebral structure, thereby accelerating the optimization speed of the vertebral structure.

[0158] In some possible implementations, the positions of the obtained multiple first pixel points are used to determine the center position corresponding to the multiple first pixel points, wherein the position of the first pixel point can be expressed as (x, y), and the coordinates of the center position can be obtained by the position mean of each first pixel point, that is, the mean of the x-coordinate value and the mean of the y-coordinate value of the first pixel point are respectively obtained to obtain the coordinates of the center position. At the same time, the embodiment of the present invention can set the height corresponding to the center position to a preset height, for example, the preset height can be 1. The spatial point corresponding to the preset height of the center position can be used as the side vertex of the vertebral structure. At this time, since the center position is a point, the corresponding vertex is one.

[0159] In some possible embodiments, when a plurality of first pixel points are obtained, a second pixel point whose mask value is greater than a third threshold value may also be obtained among the plurality of first pixel points, wherein the third threshold value is greater than the first threshold value, such as a value greater than 0.5. In the embodiment of the present invention, the third threshold value may be 0.8, but is not a specific limitation of the present invention. When a second pixel point whose mask value is greater than the third threshold value is obtained, the height of the plurality of second pixel points may be set to a preset height, such as 1, and at this time, the plane formed by the second pixel point at the preset height may be used as the top surface formed by the side vertices.

[0160] In some possible implementations, when the side vertices are obtained, the lesion area can be used as the bottom surface of the vertebral structure, and each vertex of the bottom surface can be connected with the corresponding side vertices to obtain the vertebral structure. Wherein, when the side vertices are determined based on the center position of the first pixel point, each vertex (vertex point) of the bottom surface can be connected with the side vertices to obtain a vertebral structure in the form of a pyramid. For example, when the lesion area is a rectangle or a quadrilateral, the obtained vertebral structure is a quadrangular pyramid.

[0161] In addition, when the side vertex is determined based on the first pixel point whose mask value is greater than the third threshold, the vertices of the bottom surface can be connected to the plane formed by the side vertices to obtain a pyramidal structure in the form of a polygonal pyramid. For example, when the lesion area is a rectangle or a quadrilateral, the obtained pyramidal structure is a quadrangular pyramid.

[0162] The above is only an exemplary description, in other embodiments, the vertebral structure can also be obtained by other methods. In addition, the embodiment of the present invention can obtain an optimized lesion area with a regional shape adapted to the lesion information by constructing a vertebral structure to perform lesion area detection.

[0163] In the embodiment of the present invention, the optimizing processing of the vertebral structure to obtain an optimized lesion area (step S40) may include: performing planar clustering optimization processing on the vertebral structure to obtain an optimized vertebral structure; and obtaining an optimized lesion area of ​​the lung image based on the optimized vertebral structure.

[0164] In some possible implementations, after obtaining the vertebral structure corresponding to the predicted lesion area, the plane of the vertebral structure can be optimized. Among them, plane clustering processing can be performed on each side of the vertebral structure, and each side is continuously optimized to finally obtain the optimized vertebral structure. Among them, the plane clustering processing can be a plane clustering processing based on the least squares method, but it is not specifically limited to the embodiment of the present invention. In other embodiments, the optimization of the vertebral structure can also be performed by other plane clustering algorithms. After performing plane clustering optimization on each side of the vertebral structure, based on the continuous optimization of the side, the boundary shape of the bottom surface of the vertebral structure also changes accordingly. Through the optimization processing, the boundary of the bottom surface is continuously close to the boundary of the lesion. After performing the optimization processing of the vertebral structure, the final lesion area can be determined based on the bottom surface area of ​​the optimized vertebral structure, that is, the final lesion area of ​​the image can be obtained based on the intersection of the multiple side surfaces contained in the optimized vertebral structure and the bottom surface of the vertebral structure.

[0165] The process of optimizing the vertebral structure is described in detail below. The optimization process of performing plane clustering on the vertebral structure to obtain the optimized vertebral structure includes: performing plane clustering on the first pixel point included in the mask information to obtain the side surface to which the first pixel point belongs; iteratively updating the side surface based on the first pixel point included in the side surface to obtain multiple optimized side surfaces; and forming the optimized vertebral structure based on the optimized side surfaces.

[0166] As described above, each pixel point in the lesion area can be constructed as a point set of (x, y, z), and the embodiment of the present invention can represent the reference plane as plane z = 0, and each surface in the vertebral structure can be uniquely determined by the equation Ax + By + Cz + D = 0, C = 1. Therefore, the task of optimizing the vertebral structure in the present invention can be reduced to: finding the optimal parameters {A, B, D} of each plane.

[0167] The embodiment of the present invention can perform optimization processing of the vertebral structure through the first pixel point in the lesion area, and the first pixel point is a pixel point whose mask value is greater than the first threshold. The corresponding step of performing planar clustering optimization on the vertebral structure based on the first pixel point may include grouping and updating, and the grouping includes assigning the three-dimensional coordinate point (x, y, z) corresponding to each first pixel point in the lesion area to the side surface with the closest distance, that is, the side surface to which each first pixel point belongs can be obtained.

[0168] Wherein, the embodiment of the present invention can assign the three-dimensional coordinates corresponding to the first pixel points to the corresponding side of the vertebral structure based on the center position of each first pixel point, wherein the embodiment of the present invention can form a top-view area corresponding to the side of the vertebral structure on the bottom surface based on the center position of the first pixel point. The center position of the first pixel point can be obtained according to the average value of the position coordinates (x, y) of the first pixel point, and then a plurality of bottom surface partitions are obtained based on the connection line between the vertebral vertebrae and the center position, and each bottom surface partition is constructed as a top-view area of ​​the corresponding side. Wherein, the predicted lesion area is a quadrilateral as an example for explanation, wherein the bottom surface (predicted lesion area) can be divided into a plurality of bottom surface areas a, b, c and d by the center position and the four vertices of the predicted lesion area. The plurality of bottom surface areas can be the top-view areas of the corresponding side surfaces, respectively. Correspondingly, the bottom surface area can be grouped into corresponding side surfaces, so as to determine the side surface to which each first pixel point belongs. For example, the bottom surface area a can be determined as the top-view area of ​​the side surface S1, that is, the side surface to which the three-dimensional coordinate point corresponding to the first pixel point in the bottom surface area a belongs is determined to be the side surface corresponding to S1, the bottom surface area b can be determined as the top-view area of ​​the side surface S2, that is, the side surface to which the three-dimensional coordinate point corresponding to the first pixel point in the bottom surface area b belongs is determined to be the side surface corresponding to S2, the bottom surface area c can be determined as the top-view area of ​​the side surface S3, that is, the side surface to which the three-dimensional coordinate point corresponding to the first pixel point in the bottom surface area c belongs is determined to be the side surface corresponding to S3, and the bottom surface area d can be determined as the top-view area of ​​the side surface S4, that is, the side surface to which the three-dimensional coordinate point corresponding to the first pixel point in the bottom surface area d belongs is determined to be the side surface corresponding to S4. That is, in the embodiment of the present invention, the grouped side surfaces corresponding to the three-dimensional coordinate points of each first pixel point in the bottom surface can be determined according to the center position of the first pixel point and the bottom surface boundary of the vertebral structure.

[0169] The updating may include regressing each side surface separately through the robust least squares algorithm (RLS) so that the sum of the distances between the three-dimensional coordinates corresponding to the first pixel point corresponding to each side surface and the corresponding side surface is minimized. That is, through the first pixel point corresponding to each side surface, the corresponding plane fitting can be performed to obtain a new side surface. After obtaining the new side surface, a new bottom surface is obtained at the same time, and then the center position of each first pixel point in the new bottom surface can be re-determined, and the first pixel points are grouped (determining the side surface to which they belong), and then the side surface is updated, so that through repeated iterative grouping and updating processes, an optimized vertebral structure is finally obtained. Among them, the cutoff conditions for iterative grouping and updating can include at least one of the following methods: the number of iterative updates reaches a second threshold, and the distances between the multiple first pixel points and the multiple side surfaces after the update meet the preset conditions.

[0170] That is to say, in the embodiment of the present invention, whether the optimization of the vertebral structure meets the cutoff condition can be determined by the number of iterations, such as the number of iterations can be 10 times, or in other embodiments, the number of iterations can also be other values, which are not specifically limited by the present invention. Alternatively, in other embodiments, when the regression residual returned by RLS is small enough, the final optimized vertebral structure is obtained. Wherein, when the optimization is performed on the vertebral structure obtained by the nth optimization, the sum of the distances between the three-dimensional coordinates of the positive image point in the bottom surface of the vertebral structure obtained by the n-1th optimization and the corresponding side surface of the vertebral structure obtained by the nth optimization is less than the distance threshold. At this time, it can be determined that the distances between the positive image point in the bottom surface of the vertebral structure obtained by the n-1th optimization and the multiple side surfaces of the vertebral structure obtained by the nth optimization after the update meet the preset conditions, and the optimization can be terminated at this time. Wherein n represents the number of optimizations, and n is an integer value greater than 1. In the embodiment of the present invention, the maximum number of iterations (second threshold) and the residual threshold (distance threshold) are set to 10 and 1e-4 respectively, but are not specifically limited to the present invention.

[0171] After the vertebral structure is optimized, the final lesion area can be determined according to the intersection between the bottom and the side of the optimized vertebral structure. In the clustering process of the side, the vertices of the side remain unchanged, and the position of the bottom edge is continuously optimized, thereby completing the clustering optimization of the side, and then realizing the optimization of the lesion area of ​​the bottom. The embodiment of the present invention regresses the most suitable vertebral structure with the help of the information of the entire predicted lesion area and the corresponding label value (mask value). Since the final lesion area is obtained from the support plane (side) instead of the boundary pixel, it is more robust to the inaccurate bounding box and can regress a more accurate lesion boundary.

[0172] To summarize, in an embodiment of the present invention, by performing lesion detection processing on the lobe segmentation image, a predicted lesion area in the lobe segmentation image is obtained, and the characteristic information of the lesion area can be used to obtain mask information about the lesion area. The mask information obtained in the embodiment of the present invention is a value between 0 and 1. The vertebral structure can be constructed through the mask information, and the final lesion area can be predicted by processing the main structure. Among them, there is a background area in the predicted lesion. The predicted lesion area is optimized by constructing the vertebral structure to obtain the final lesion area, which can reduce the influence of the background area and improve the detection accuracy of the lesion area.

[0173] Furthermore, in an embodiment of the present invention, when the optimized lesion area in the lung lobe segmentation image is determined, the lesion can be analyzed, for example, the disease type corresponding to the lesion, the current status (disease level), and the future development of the disease can be predicted (prediction of development level).

[0174] Figure 5 A flowchart of another lung image processing method according to an embodiment of the present invention is shown. The lung image processing method further includes:

[0175] S50: for each of the lung lobe segmentation images, determining a first region corresponding to a first disease and a second region corresponding to a second disease in the optimized lesion region, where the second disease can be unidirectionally transformed into the first disease;

[0176] S60: Determine the level of the first disease;

[0177] S70: Determine a predicted development level of the first disease based on the level of the first disease, the first region, and the second region.

[0178] As described in the above embodiment, the lesion area in the lung lobe segmentation image may include the lesion location of at least one type of disease, such as a lung nodule lesion area and a lung cancer lesion area. For example, in an embodiment of the present invention, the first disease of the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung cancer, and the second disease of the lesion area of ​​each lung lobe in the lung lobe segmentation image is a lung nodule.

[0179] For each of the lung lobe segmentation images, the first area corresponding to the first disease and the second area corresponding to the second disease in the optimized lesion area are determined, and it can be applied to: determine the corresponding first disease and the associated second disease according to each lung lobe segmentation image. For example, each lung lobe image corresponding to the lung segmentation image can be input into a neural network that has been trained to identify lung nodules and lung cancer, and the first disease (lung cancer) and the associated second disease (lung nodules) of each lung lobe are detected by the neural network, and the lesion areas corresponding to the first disease and the second disease can also be obtained. The neural network may include feature extraction of the lung lobe image, a region candidate network, and a classification network. The feature extraction network performs feature extraction processing on the lung lobe image to obtain the lung lobe feature, and the region candidate network is used to perform convolution processing on the lung lobe feature to obtain the lesion area of ​​the first disease and the second disease in the lung lobe image, and the classification network is used to classify and identify the disease type of the lesion area to obtain the disease type (first disease or second disease) corresponding to the lung lobe segmentation image.

[0180] It should be noted that, in the process of detecting the first disease and the associated second disease of each lung lobe segmentation image, not every lung lobe necessarily has the first disease and the second disease, and there may be a situation where only the second disease exists, or there may be a situation where neither the first disease nor the second disease exists. The embodiment of the present invention can use different disease identifiers to represent different diseases, and whether a disease exists.

[0181] When a first region of a first disease and a second region of a second disease are detected in the lung lobe segmentation image, a level corresponding to the first disease may be determined.

[0182] That is to say, the embodiment of the present invention can achieve the level of the first disease in the lesion area of ​​the left upper lobe and the left lower lobe of the left lung image; and respectively determine the level of the first disease in the lesion area of ​​the right upper lobe, the right middle lobe, and the right lower lobe of the right lung image. That is to say, the level of the first disease of each lobe can be determined. Among them, in the embodiment of the present invention, a neural network can be used to detect the level of the first disease of each lobe. For example, each lobe segmentation image obtained by segmentation can be input into the neural network, and the level of the current first disease of each lobe segmentation image can be classified and detected through the processing of the neural network. Among them, the neural network can include a convolutional neural network, for example, it can perform feature extraction of the lobe image (for example, through a residual network), and after obtaining the lobe features, the classification network (multiple classifiers) is used to perform the detection of the first disease level corresponding to the lobe features. Different classification levels can be set for different first diseases.

[0183] If the first disease is lung cancer, the grades of lung cancer include: stage I, stage II, stage III, stage IV and stage III and IV.

[0184] Stage I: The tumor is relatively small and has no lymph node metastasis, so it can be completely removed by surgery. According to the size of the tumor, it is divided into stage IA and stage IB. Smaller tumors are stage IA, and larger tumors are stage IB.

[0185] Stage II: It can also be divided into two subtypes: stage IIA and stage IIB. Stage IIA includes two situations: one is a slightly larger tumor without adjacent lymph node metastasis, and the other is a smaller tumor with peripheral lymph node metastasis. Stage IIB refers to a larger tumor with lymph node metastasis, or a large tumor with or without involvement of peripheral lung structures, but no lymph node metastasis.

[0186] Stage III: It is divided into stages IIIA and IIIB. Many stage IIIA and almost all stage IIIB tumors are difficult or impossible to remove surgically. For example, the tumor may involve the mediastinal lymph nodes or invade adjacent structures in the lungs. In some cases, due to various factors, the tumor cannot be completely removed in one go and can only be removed in multiple times. In this case, it is difficult to completely remove the tumor.

[0187] Stage IV: includes: cancer cells have spread to multiple parts of the opposite lung, or to fluid around the lung or heart, or to other parts of the body through the bloodstream. Once cancer cells enter the bloodstream, they can spread to any part of the body, but the most common sites of metastasis are the brain, bones, liver, and adrenal glands.

[0188] Stage III and IV: Lung cancer is in the middle and late stages and cannot be cured by surgery. Radiotherapy and chemotherapy are generally the main treatment methods. In the late stages, the main focus is on reducing the patient's pain and palliative care. When lung cancer has the following conditions, surgical resection is not possible, including: tumor metastasis to the supraclavicular lymph nodes, or tumor involving important organs in the chest cavity, such as the heart, large blood vessels or trachea.

[0189] Through the above configuration, the level of the first disease corresponding to each lung lobe in the lung lobe segmentation image can be obtained.

[0190] In case of determining the level of the first disease, determining a predicted development level of the first disease based on the level of the first disease, the first region, and the second region (step S40 ) may be performed.

[0191] In an embodiment of the present invention, the determination of the predicted development level of the first disease based on the level, first region and second region of the first disease includes: obtaining the first volume of the first disease on the lobe according to the first region in each layer of the lobe segmentation image, and obtaining the second volume of the second disease on the lobe according to the second region in each layer of the lobe segmentation image; respectively calculating the ratio of the first volume to the second volume for the lobe; and determining the predicted development level of the lobe corresponding to the first disease based on the level of the first disease, the ratio and a preset threshold. As described in the above embodiment, the lobe segmentation image can be an image of any lobe segmented from the lung image, wherein, since the lung image can be a multi-layer image acquired by CT, the corresponding lobe segmentation image can also be a multi-layer image for any lobe, and the lesion area in the embodiment of the present invention can include the lesion position of each layer of the multi-layer image. In addition, the level of the first disease is the disease level represented by the current lobe segmentation image, and the predicted development level of the first disease is used to represent the future development of the disease, that is, the predicted development level of the first disease can be used to determine whether the first disease is getting better or worse, and the degree of improvement or worsening can be represented by the level number of the predicted level.

[0192] Among them, the first volume and the second volume are the first volume of the lesion area of ​​the first disease and the second volume of the lesion area of ​​the second disease of each lung lobe, respectively. That is to say, the first volume includes: the first volume of the lesion area of ​​the first disease of the left upper lobe, the first volume of the lesion area of ​​the first disease of the left lower lobe, the first volume of the lesion area of ​​the first disease of the right upper lobe, the first volume of the lesion area of ​​the first disease of the right middle lobe, and the first volume of the lesion area of ​​the first disease of the right lower lobe. The second volume includes: the second volume of the lesion area of ​​the second disease of the left upper lobe, the second volume of the lesion area of ​​the second disease of the left lower lobe, the second volume of the lesion area of ​​the second disease of the right upper lobe, the second volume of the lesion area of ​​the second disease of the right middle lobe, and the second volume of the lesion area of ​​the second disease of the right lower lobe.

[0193] The method of determining the predicted development level of the lung lobe corresponding to the first disease based on the level of the first disease, the ratio and the preset threshold value includes: in response to the ratio being greater than the preset threshold value, determining the predicted development level to be the next level lighter than the level of the first disease; or in response to the ratio being less than or equal to the preset threshold value, determining the predicted development level to be the previous level more serious than the level of the first disease. For example, in the left upper lobe, the level of lung cancer is stage IB, and the first volume of the left upper lobe lung cancer area is divided by the second volume of the left upper lobe lung nodule area to obtain a ratio. The ratio is less than or equal to the preset threshold value, indicating that although the left upper lobe lung cancer area is small, the lung nodule area is large, and the possibility of the lung nodule area being transformed into a lung cancer area is high. The level of lung cancer in the left upper lobe can be predicted to stage IIA or stage IIB in the previous level II stage. If the ratio is greater than the preset threshold value, the level of lung cancer in the left upper lobe can be predicted to the next level IA stage. Among them, in the case where the first disease is lung cancer, the preset threshold value can be 0.2.

[0194] The embodiment of the present invention can grid the lung lobe segmentation image, and calculate the grid volumes of the first area and the second area after gridding respectively, the sum of the grid volumes of the first area is the first volume, and the sum of the grid volumes of the second area is the second volume.

[0195] In an embodiment of the present invention, the method for gridding the lobe segmentation image is: determining the number of layers of the lobe segmentation image; and determining the first volume and the second volume for the grid areas of the first area and the second area in each layer of the image, respectively, according to the grid areas of the first area and the second area of ​​each lobe of each layer, the number of layers, the CT scan layer thickness, and the scanning layer spacing.

[0196] In an embodiment of the present invention, a method for determining a first volume corresponding to a first region is as follows: using the grid area corresponding to the first region in the multi-layer image in each pulmonary lobe segmentation image, as well as the inter-layer spacing and the layer thickness, a first sub-volume formed by the first region of two adjacent layers of the image is obtained, and using the sum of the first sub-volumes formed by all adjacent two layers of the first region in the pulmonary lobe segmentation image to obtain the first volume corresponding to the first region of the pulmonary lobe. Among them, the structure formed by the adjacent two layers of the first region can be regarded as a prism, and the area of ​​the upper and lower bottom surfaces of the prism is the grid area corresponding to the first region. The height of the prism can be determined by the inter-layer spacing and the layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of the first region can be the sum of the layer thickness*2 and the inter-layer spacing, and the height of the prism formed by the remaining two adjacent layers of the first region can be the sum of the layer thickness and the inter-layer spacing. Based on the upper and lower bottom areas and the height, the first sub-volume of the first region formed by the adjacent layers can be determined. Then, the sum of the first sub-volumes can be used to obtain the first volume.

[0197] The method for determining the second volume corresponding to the second region is: using the grid area corresponding to the two adjacent layers of the second region in each lobe segmentation image, as well as the inter-layer spacing and the layer thickness, to obtain the second sub-volume formed by the two adjacent layers of the small airway region, and using the sum of the second sub-volumes formed by the second regions in all two adjacent layers of the lobe segmentation image to obtain the second volume of the second region of the lobe. Among them, the structure formed by the two adjacent layers of the second region can be regarded as a prism, and the area of ​​the upper and lower bottom surfaces of the prism is the grid area corresponding to the second region. The height of the prism can be determined by the inter-layer spacing and the layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of the second region can be the sum of the layer thickness*2 and the inter-layer spacing, and the height of the prism formed by the remaining two adjacent layers of the second region can be the sum of the layer thickness and the inter-layer spacing. The second sub-volume of the second region formed by the adjacent layers can be determined based on the upper and lower bottom areas and the height. Then, the second volume can be obtained by the sum of the second sub-volumes.

[0198] In an embodiment of the present invention, the specific method of gridding the first area of ​​each layer of the image in the lung lobe segmentation image to obtain the grid area of ​​the first area is: Step 1: respectively determine the edge of the first area of ​​each layer of the image in the lung lobe segmentation image (corresponding to the edge of the optimized lesion area of ​​the first disease); respectively determine a first standard shape in each layer of the first area; Step 2: The first standard shape extends toward the edge of the first area; Step 3: When a number of set points of the first standard shape touch the edge of the first area, the first standard shape stops extending and generates a second standard shape outside the first standard shape; generate in accordance with steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the areas of all standard shapes, and obtain the grid area of ​​the first area of ​​each layer of each lung lobe; wherein the areas of the first standard shape, the second standard shape, and the standard shapes corresponding to the set number decrease in sequence.

[0199] In an embodiment of the present invention, the specific method of gridding the second area of ​​each layer of the image in the lung lobe segmentation image to obtain the grid area of ​​the second area is: Step 1: respectively determine the edge of the second area in each layer of the image in the lung lobe segmentation image (corresponding to the edge of the optimized lesion area of ​​the second disease); respectively determine a first standard shape in the second area of ​​each layer of the image; Step 2: The first standard shape extends toward the edge of the second area of ​​each layer of the image; Step 3: When a number of set points of the first standard shape touch the edge of the second area, the first standard shape stops extending and generates a second standard shape outside the standard shape; generate in accordance with steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the areas of all standard shapes, and obtain the grid area of ​​the second area of ​​each layer and each lung lobe; wherein the areas of the first standard shape, the second standard shape, and the standard shapes corresponding to the set number decrease in sequence.

[0200] Among them, the first specification shape is preferably at the geometric center of the first area and the second area. The specification shape can be selected from any one of a circle, an ellipse, a rectangle, and a square. For example, if the specification shape is a circle, a number of set points can be evenly set on the circumference of the circle, such as 20 set points. When a number of set points of the specification shape touch the edge of the first area of ​​the right lung or the edge of the second area of ​​the right lung, the first specification shape stops extending and generates a second specification shape outside the specification shape; the second specification shape can also be selected from any one of a circle, an ellipse, a rectangle, and a square, but the second specification shape is smaller than the first specification shape to achieve the purpose of fine division. For the setting of the value of the set quantity, those skilled in the art can set it by themselves according to the accuracy requirements.

[0201] When the level of the first disease and the ratio between the first volume and the second volume are obtained, the development level of the first disease can be predicted. For example, in some other embodiments of the present invention, the first disease can also be chronic obstructive pulmonary disease, and the corresponding second disease can be small airway disease.

[0202] In the present invention, the grades of COPD include: GOLD grade 0 (absent), GOLD grade 1 (mild), GOLD grade 2 (moderate), GOLD grade 3 (severe) and GOLD grade 4 (extremely severe).

[0203] In an embodiment of the present invention, a method for determining each lobe on each lung lobe according to each lung lobe, comprises: determining the first volume of the COPD area on each lung lobe and the total volume of each lung lobe, and determining the grade of COPD on each lung lobe according to the ratio of the first volume to the total volume.

[0204] For example, four thresholds are set, namely, a first threshold, a second threshold, a third threshold and a fourth threshold. If the ratio of the first volume to the total volume is less than the first threshold, the grade of COPD is GOLD0 (non-existent); if the ratio of the first volume to the total volume is between the first threshold and the second threshold, the grade of COPD is GOLD1 (mild); if the ratio of the first volume to the total volume is between the second threshold and the third threshold, the grade of COPD is GOLD2 (moderate); if the ratio of the first volume to the total volume is between the third threshold and the fourth threshold, the grade of COPD is GOLD3 (severe); if the ratio of the first volume to the total volume is greater than the fourth threshold, the grade of COPD is GOLD4 (extremely severe). The first threshold, the second threshold, the third threshold and the fourth threshold may be 20%, 30%, 50% and 80% respectively.

[0205] For example, if the ratio of the first volume of the left upper lobe COPD area to the total volume of the left upper lobe is between the second threshold and the third threshold, the grade of COPD is GOLD 2 (moderate).

[0206] For example, on the left upper lobe, based on the grade of COPD, the predicted grade of COPD in the left upper lobe can be determined according to the ratio of the first volume of the left upper lobe to the second volume of the left upper lobe and a preset threshold. The method of determining the predicted grade of the first disease in the corresponding lobe according to the ratio of the first area and the second area of ​​each lobe and a preset threshold based on the grade of the first disease is: if the ratio is greater than the preset threshold, the predicted grade is the next grade of the grade of the first disease; otherwise, the predicted grade is the previous grade of the grade of the first disease; wherein the previous grade is more serious than the grade of the first disease.

[0207] For example, in the left upper lobe, the grade of COPD is GOLD 2 (moderate), and the ratio obtained by dividing the first volume of the COPD area in the left upper lobe by the second volume of the small airway lesion area in the left upper lobe is less than or equal to the preset threshold, indicating that although the COPD area in the left upper lobe is small, the small airway lesion area is large, and the possibility of the small airway lesion area being transformed into the COPD area is high, and the grade of COPD in the left upper lobe can be predicted to the previous grade GOLD 3 (severe). If the ratio is greater than the preset threshold, it is considered that drug treatment can be performed, and the grade of COPD in the left upper lobe can be predicted to the next grade GOLD 1 (mild). Among them, when the first disease is lung cancer, the preset threshold can be 0.4.

[0208] In summary, the embodiments of the present invention can not only optimize the detection accuracy of the lung lobe lesion area, but also predict the future development level of the disease based on the optimized lesion area and the disease level of the current lung lobe.

[0209] In addition, the embodiment of the present invention also provides a lung image processing device, referring to Figure 6 , the lung image processing device comprises:

[0210] The segmentation module 10 is used to perform lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image;

[0211] A processing module 20 is used to perform lesion prediction processing on the lung lobe segmentation image to obtain mask information of the predicted lesion area, wherein the mask information includes mask values ​​of pixel points in the predicted lesion area;

[0212] A forming module 30, configured to form a vertebral structure based on the mask value in the mask information;

[0213] The processing module 20 is further used to optimize the vertebral structure to obtain an optimized lesion area, and the optimized lesion area matches the lesion shape.

[0214] Furthermore, the mask value represents the confidence that the pixel point is located at the lesion position, and the forming module 30 is specifically used to obtain the three-dimensional coordinates of the pixel point based on the position of the pixel point in the predicted lesion area in the lung image and the mask value corresponding to the pixel point in the mask information; and form the vertebral structure based on the three-dimensional coordinates of the pixel point in the predicted lesion area.

[0215] Furthermore, the forming module 30 includes:

[0216] A determining unit, configured to determine a pixel point in the predicted lesion area whose mask value exceeds a first threshold as a first pixel point;

[0217] The determining unit is further configured to determine the vertex of the vertebral structure based on the position of the first pixel point in the predicted lesion area and a preset height value;

[0218] A forming unit is used to form the vertebral structure based on the predicted lesion area and the vertex.

[0219] Furthermore, the processing unit 20 includes:

[0220] A processing unit, configured to perform a planar clustering optimization process on the vertebral structure to obtain an optimized vertebral structure;

[0221] The acquisition unit is used to obtain the optimized lesion area of ​​the lung image based on the bottom surface of the optimized vertebral structure.

[0222] Furthermore, the acquisition unit is specifically used to perform planar clustering on the first pixel point contained in the mask information to obtain the side surface to which the first pixel point belongs; based on the first pixel point included in the side surface, iteratively update the side surface to obtain an optimized side surface; and form the optimized vertebral structure based on the optimized side surface.

[0223] Furthermore, the device further comprises: a determination module. The determination module comprises:

[0224] A first determination unit is used to determine, for each of the lung lobe segmentation images, a first region corresponding to a first disease and a second region corresponding to a second disease in the optimized lesion region, where the second disease can be unidirectionally converted into the first disease;

[0225] The second determining unit is configured to determine a level of the first disease, and determine a predicted development level of the first disease based on the level of the first disease, the first region, and the second region.

[0226] Furthermore, the lung lobe segmentation image is a multi-layer image of the lung lobe, and the first determining unit includes:

[0227] an acquisition subunit, configured to acquire a first volume of the first disease on the lung lobe according to the first region in each layer of the lung lobe segmentation image, and to acquire a second volume of the second disease on the lung lobe according to the second region in each layer of the lung lobe segmentation image;

[0228] a calculation subunit, configured to calculate a ratio of the first volume to the second volume for each of the pulmonary lobes;

[0229] A determination subunit is used to determine the predicted development level of the lung lobe corresponding to the first disease based on the level of the first disease, the ratio and a preset threshold.

[0230] Furthermore, the determination subunit is specifically used to determine that the predicted development level is the next level lighter than the level of the first disease in response to the ratio being greater than the preset threshold; or to determine that the predicted development level is the previous level more serious than the level of the first disease in response to the ratio being less than or equal to the preset threshold.

[0231] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a lung image processing program is stored. When the lung image processing program is executed by a processor, the steps of the lung image processing method described below are implemented.

[0232] The various embodiments of the lung image processing device and the computer-readable storage medium of the present invention may refer to the various embodiments of the lung image processing method of the present invention, and will not be described in detail here.

[0233] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0234] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0236] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lung image processing method, characterized in that: include: Performing lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image; Performing lesion prediction processing on the lung lobe segmentation image to obtain mask information of the predicted lesion area, the mask information including mask values ​​of pixels in the predicted lesion area, the mask values ​​indicating the confidence that the pixels are located at the lesion position, and the mask values ​​being values ​​greater than or equal to zero and less than or equal to 1; forming a vertebral structure based on the mask value in the mask information; The forming of the vertebral structure based on the mask value in the mask information includes: Obtaining the three-dimensional coordinates of the pixel point based on the position of the pixel point in the predicted lesion area in the lung image and the mask value corresponding to the pixel point in the mask information; forming the vertebral structure based on the three-dimensional coordinates of the pixel points in the predicted lesion area; The vertebral structure is optimized to obtain an optimized lesion area, and the optimized lesion area matches the lesion shape.

2. The method according to claim 1, characterized in that The forming of a vertebral structure based on the mask value in the mask information includes: Determine a pixel point in the predicted lesion area whose mask value exceeds a first threshold as a first pixel point; Determining the vertex of the vertebral structure based on the position of the first pixel point in the predicted lesion area and a preset height value; The vertebral structure is formed based on the predicted lesion area and the vertex.

3. The method according to claim 1, characterized in that The step of optimizing the vertebral structure to obtain an optimized lesion area includes: Performing a planar clustering optimization process on the vertebral structure to obtain an optimized vertebral structure; Based on the bottom surface of the optimized vertebral structure, the optimized lesion area of ​​the lung image is obtained.

4. The method according to claim 3, characterized in that The performing planar clustering optimization processing on the vertebral structure to obtain an optimized vertebral structure includes: Performing planar clustering on the first pixel point included in the mask information to obtain the side surface to which the first pixel point belongs; Iteratively updating the side surface based on a first pixel point included in the side surface to obtain an optimized side surface; The optimized vertebral body structure is formed based on the optimized side faces.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: For each of the lung lobe segmentation images, determining a first region corresponding to a first disease and a second region corresponding to a second disease in the optimized lesion region, wherein the second disease can be unidirectionally converted into the first disease; A level of the first disease is determined, and a predicted level of progression of the first disease is determined based on the level of the first disease, the first region, and the second region.

6. The method according to claim 5, characterized in that The lung lobe segmentation image is a multi-layer image of the lung lobe, and determining the predicted development level of the first disease based on the level of the first disease, the first region, and the second region includes: Acquire a first volume of the first disease on the lung lobe according to the first region in each layer of the lung lobe segmentation image, and acquire a second volume of the second disease on the lung lobe according to the second region in each layer of the lung lobe segmentation image; calculating the ratio of the first volume to the second volume for the lung lobe respectively; The predicted development level of the lung lobe corresponding to the first disease is determined based on the level of the first disease, the ratio and a preset threshold.

7. The method according to claim 6, characterized in that The determining the predicted development level of the lung lobe corresponding to the first disease based on the level of the first disease, the ratio and a preset threshold value includes: In response to the ratio being greater than the preset threshold, determining the predicted progression level to be a next level lighter than the level of the first disease; or In response to the ratio being less than or equal to the preset threshold, the predicted progression level is determined to be a level higher than the level of the first disease.

8. A lung image processing device, characterized in that: include: A segmentation module, used for performing lung lobe segmentation processing on the lung image to obtain a lung lobe segmentation image; a processing module, configured to perform lesion prediction processing on the lung lobe segmentation image to obtain mask information of the predicted lesion area, wherein the mask information includes mask values ​​of pixels in the predicted lesion area, wherein the mask values ​​represent the confidence that the pixels are located at the lesion position, and wherein the mask values ​​are numerical values ​​greater than or equal to zero and less than or equal to 1; A forming module, used to form a vertebral structure based on the mask value in the mask information, wherein the forming module is specifically used to obtain the three-dimensional coordinates of the pixel point based on the position of the pixel point in the predicted lesion area in the lung image and the mask value corresponding to the pixel point in the mask information; and form the vertebral structure based on the three-dimensional coordinates of the pixel point in the predicted lesion area; The processing module is further used to optimize the vertebral structure to obtain an optimized lesion area, and the optimized lesion area matches the lesion shape.

9. A lung image processing device, characterized in that: The lung image processing device includes: a memory, a processor, and a lung image processing program stored in the memory and executable on the processor. When the lung image processing program is executed by the processor, the steps of the lung image processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a lung image processing program, which, when executed by a processor, implements the steps of the lung image processing method according to any one of claims 1 to 7.

Images