Disease trend level determination method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for determining a disease trend level, the method comprising: obtaining a segmented image of a lung lobe; determining a first disease and a second disease in the lesion area of ​​each lung lobe in the segmented image of the lung lobe, wherein the second disease can be unidirectionally converted into the first disease; and determining the trend level of the first disease of each lung lobe based on the known level of the first disease and the second disease. The present invention can predict the development trend of a disease on a lung lobe and can provide assistance to doctors in determining a treatment plan.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a disease trend level determination method, device, equipment and storage medium. Background Art

[0002] Both lung cancer and COPD are chronic diseases with a relatively slow onset. Chronic obstructive pulmonary disease (COPD) is a common disease that can be prevented and treated, characterized by persistent respiratory symptoms and airflow limitation, caused by airway and / or alveolar abnormalities due to significant exposure to harmful particles or gases. Based on existing clinical and technical knowledge, we know that the early stage of lung cancer is pulmonary nodule disease, and the early stage of chronic diseases is small airway disease. The early onset of pulmonary nodule disease or small airway disease is not obvious, and most patients do not pay attention to it. However, once it turns into lung cancer or COPD, its cure rate is low and the cost of cure is relatively high.

[0003] If the development trend of lung cancer or COPD in a specific lobe can be predicted, it will be of great significance for targeted treatment and surgery of the lobe, and doctors can give reasonable treatment plans based on the predicted level. However, it is currently impossible to predict the development trend of diseases in the lobe. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for determining a disease trend level, aiming to solve the problem of being unable to predict the development trend of a disease on a lung lobe.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for determining a disease trend level, comprising:

[0006] Acquire lung lobe segmentation images;

[0007] Determine a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, where the second disease can be unidirectionally converted into the first disease;

[0008] Based on the known level of the first disease and the second disease, the trend level of the first disease for each lung lobe is determined respectively.

[0009] Optionally, the determining the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease respectively includes:

[0010] Locating first areas of the first disease and second areas of the second disease in each lung lobe respectively;

[0011] Based on the known level of the first disease, the trend level of the first disease in each lung lobe is determined according to the ratio of the first area to the second area in each lung lobe.

[0012] Optionally, the determining the trend level of the first disease in each lung lobe according to the ratio of the first area to the second area of ​​each lung lobe based on the known level of the first disease includes:

[0013] obtaining a first volume of the first disease on each lung lobe according to the first region, and obtaining a second volume of the second disease on each lung lobe according to the second region;

[0014] calculating a ratio of the first volume to the second volume for each lung lobe respectively;

[0015] Based on the level of the first disease, the trend level of the first disease in the corresponding lung lobe is determined according to the ratio of the first volume to the second volume of each lung lobe.

[0016] Optionally, the determining the trend level of the first disease in each lung lobe according to the ratio of the first area to the second area of ​​each lung lobe based on the known level of the first disease includes:

[0017] In response to the ratio of the first area to the second area being greater than a preset threshold, determining the next level lighter than the known level of the first disease as the trend level of the first disease;

[0018] In response to the ratio of the first area to the second area being less than or equal to a preset threshold, a previous level that is more serious than the known level of the first disease is determined as the trend level of the first disease.

[0019] Optionally, the first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is chronic obstructive pulmonary disease, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is small airway lesions; or

[0020] The first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung cancer, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung nodule.

[0021] Furthermore, before determining the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease, the method further includes:

[0022] The segmented images of each lung lobe are input into a preset neural network to extract image features through a residual network of the preset neural network, and the known level of the first disease is determined based on a classification network of the preset neural network.

[0023] Optionally, acquiring a lung lobe segmentation image includes:

[0024] Acquiring a lung image, wherein the lung image includes a full inspiratory phase lung image and a full expiratory phase lung image;

[0025] Performing lung lobe segmentation on the full inhalation phase lung image and the full exhalation phase lung image respectively to obtain a first lung lobe segmentation image and a second lung lobe segmentation image;

[0026] The determining of the first disease and the second disease of the lesion area of ​​each lung lobe in the lung lobe segmentation image comprises:

[0027] determining the first disease according to the first lung lobe segmentation image or the second lung lobe segmentation image;

[0028] The second disease is determined according to the first lung lobe segmentation image and the second lung lobe segmentation image.

[0029] In a second aspect, the present invention provides a device for determining a disease trend level, the device comprising:

[0030] An acquisition module, used for acquiring a lung lobe segmentation image;

[0031] A first determination module is used to determine a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, where the second disease can be unidirectionally converted into the first disease;

[0032] The second determination module is used to determine the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease.

[0033] Optionally, the second determining module includes:

[0034] a positioning unit, used for positioning a first area of ​​the first disease and a second area of ​​the second disease in each lung lobe respectively;

[0035] A determination unit is used to determine the trend level of the first disease in each lung lobe based on the known level of the first disease and according to the ratio of the first area to the second area of ​​each lung lobe.

[0036] Optionally, the determining unit is specifically configured to obtain a first volume of the first disease on each lung lobe according to the first region, and to obtain a second volume of the second disease on each lung lobe according to the second region;

[0037] calculating a ratio of the first volume to the second volume for each lung lobe respectively;

[0038] Based on the level of the first disease, the trend level of the first disease in the corresponding lung lobe is determined according to the ratio of the first volume to the second volume of each lung lobe.

[0039] Optionally, the determining unit is further configured to determine, in response to a ratio of the first area to the second area being greater than a preset threshold, a next level lighter than a known level of the first disease as a trend level of the first disease;

[0040] In response to the ratio of the first area to the second area being less than or equal to a preset threshold, a previous level that is more serious than the known level of the first disease is determined as the trend level of the first disease.

[0041] Optionally, the first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is chronic obstructive pulmonary disease, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is small airway lesions; or

[0042] The first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung cancer, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung nodule.

[0043] Furthermore, the first determination module is also used to input the segmented images of each lung lobe into a preset neural network to extract image features through the residual network of the preset neural network, and determine the known level of the first disease based on the classification network of the preset neural network.

[0044] Optionally, the acquisition module is specifically used to acquire lung images, which include full inhalation phase lung images and full expiratory phase lung images; and perform lung lobe segmentation on the full inhalation phase lung images and the full expiratory phase lung images, respectively, to obtain a first lung lobe segmentation image and a second lung lobe segmentation image.

[0045] Optionally, the first determination module is specifically used to determine the first disease based on the first lobe segmentation image or the second lobe segmentation image; and to determine the second disease based on the first lobe segmentation image and the second lobe segmentation image.

[0046] In a third aspect, the present invention further proposes a disease trend level determination device, which includes: a memory, a processor, and a disease trend level determination program stored in the memory and executable on the processor, wherein the disease trend level determination program is executed by the processor to perform the steps of the disease trend level determination method.

[0047] In a fourth aspect, the present invention further proposes a computer-readable storage medium, on which a disease trend level determination program is stored, and when the disease trend level determination program is executed by a processor, the steps of the disease trend level determination method are implemented.

[0048] The present invention provides a method, device, equipment and storage medium for determining a disease trend level. After acquiring a lung lobe segmentation image, the present invention can determine a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, and the second disease can be unidirectionally converted into the first disease; thereby, based on the known level of the first disease and the second disease, the trend level of the first disease in each lung lobe can be determined respectively, that is, the disease development trend on the lung lobe can be predicted, and an auxiliary role can be provided for doctors to determine a treatment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0050] Figure 2 is a flow chart of a method for determining a disease trend level according to an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of the network structure of a lung lobe segmentation method according to an embodiment of the present invention.

[0052] Figure 4 It is a functional module diagram of a preferred embodiment of the disease trend level determination device of the present invention.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0055] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0056] It should be noted that the emphysema level detection method provided in an embodiment of the present invention can be used to improve the detection efficiency and accuracy of emphysema level. The executor of the method can be any emphysema level detection device. For example, the emphysema level detection method can be executed by a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., and no specific restrictions are made here.

[0057] like Figure 1 As shown, the emphysema grade detection device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the disease trend grading device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0059] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an emphysema grade detection program. The operating system is a program for managing and controlling the hardware and software resources of the device, and supports the operation of the emphysema grade detection program and other software or programs.

[0060] exist Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the emphysema level detection program stored in the memory 1005 and execute the emphysema level detection method of the embodiment of the present invention.

[0061] In order to achieve the purpose of predicting the development trend of the disease on the lung lobe and providing assistance to doctors in determining the treatment plan, the embodiment of the present invention provides a method for determining the disease trend level, such as Figure 2 Shown, incl.

[0062] S10, obtaining a lung lobe segmentation image;

[0063] S20, determining a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, where the second disease can be unidirectionally converted into the first disease;

[0064] S30. Based on the known level of the first disease and the second disease, determine the trend level of the first disease in each lung lobe respectively.

[0065] In an embodiment of the present invention, for step S10, the lung lobe segmentation image includes: a left lung image and a right lung image. The left lung image includes: a left upper lobe and a left lower lobe; the right lung image includes: a right upper lobe, a right middle lobe, and a right lower lobe.

[0066] Before step S10, that is, before obtaining the lung lobe segmentation image, the lung image needs to be segmented. In some possible implementations, the embodiments of the present invention can obtain lung images at different viewing angles by shooting CT (Computed Tomography). Correspondingly, multiple tomographic images, i.e., lung images, can be obtained at each viewing angle, and the multiple lung images at the same viewing angle can be constructed to form a three-dimensional lung image. For example, the multiple lung images at the same viewing angle can be stacked to obtain a three-dimensional lung image, or linear fitting or surface fitting can be performed to obtain a three-dimensional lung image.

[0067] In an embodiment of the present invention, a method for segmenting a lung image includes:

[0068] Step 1: Obtain the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the transverse plane.

[0069] In some possible implementations, the lobe fissure features of the lung images at different viewing angles may be extracted by feature extraction processing. The lobe fissure features are features used to perform segmentation processing of each lobe region in the lung image. For example, the lobe fissure features may be used to determine the position of the fissure surface between the lobes to achieve lobe segmentation.

[0070] The embodiment of the present invention can perform feature extraction processing on the lung images in the sagittal plane, coronal plane and cross-sectional perspectives respectively, and obtain the fissure features of the lung images in the corresponding perspectives, that is, the lobe fissure features of the lung images in the sagittal plane, the lobe fissure features in the coronal plane and the lobe fissure features in the cross-section can be obtained respectively. In the embodiment of the present invention, the lobe fissure features in each perspective can be expressed in the form of a matrix or a vector, and the lobe fissure features can represent the characteristic values ​​of the lung images at each pixel point in the corresponding perspective.

[0071] In some possible implementations, the feature extraction process can be performed by a feature extraction neural network. For example, the neural network can be trained to achieve accurate extraction of the lobe fissure features of the lung image by the neural network, and lobe segmentation can be performed using the obtained features. When the accuracy of lobe segmentation exceeds the accuracy threshold, it means that the accuracy of the lobe fissure features obtained by the neural network meets the requirements. At this time, the network layer that performs segmentation in the neural network can be removed, and the retained network portion can be used as the feature extraction neural network of an embodiment of the present invention. Among them, the feature extraction neural network can be a convolutional neural network, such as a residual network, a pyramid feature network, or a U network. The above is only an exemplary description and is not a specific limitation of the present invention.

[0072] Step 2: Correct the third lobe fissure feature using any two lobe fissure features in the sagittal plane, coronal plane and transverse plane.

[0073] In some possible implementations, when the lobe fissure features at three perspectives are obtained, the lobe fissure features at two perspectives can be used to correct the lobe fissure features at the third perspective to improve the accuracy of the lobe fissure features at the third perspective.

[0074] In one example, an embodiment of the present invention can use the lobe fissure features in the coronal and cross-sectional perspectives to correct the lobe fissure features in the sagittal perspective. In other embodiments, another lobe fissure feature can also be corrected by any two of the lobe fissure features in the three perspectives. For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is described in the following embodiment. The first lobe fissure feature, the second lobe fissure feature and the third lobe fissure feature correspond to the lobe fissure features in the three perspectives of the embodiment of the present invention, respectively.

[0075] In some possible implementations, the first lobe fissure feature and the second lobe fissure feature can be converted to the perspective of the third lobe fissure feature by mapping, and the two lobe fissure features obtained by mapping can be used to perform feature fusion with the third lobe fissure feature to obtain a corrected lobe fissure feature.

[0076] Step 3: Segment the lung image using the corrected lobe fissure features.

[0077] In some possible implementations, lobe segmentation can be performed directly using the corrected lobe fissure feature to obtain a segmentation result of the lobe fissure. Alternatively, in other implementations, the corrected lobe fissure feature can be subjected to feature fusion processing with the third lobe fissure feature, and lobe segmentation is performed based on the fusion result to obtain a segmentation result of the lobe fissure. The segmentation result may include the position information corresponding to each partition in the identified lung image. For example, the lung image may include five lobe regions, namely, the right upper lobe, the right middle lobe, the right lower lobe, the left upper lobe, and the left lower lobe, and the segmentation result may include the position information of the above five lobes in the lung image. The embodiment of the present invention may represent the segmentation result by means of a mask feature, that is, the segmentation result obtained by the embodiment of the present invention may be a feature represented in the form of a mask. For example, the embodiment of the present invention may assign unique corresponding mask values, such as 1, 2, 3, 4, and 5, to the above five lobe regions, and the area formed by each mask value is the location area where the corresponding lobe is located. The above mask values ​​are only exemplary, and other mask values ​​may also be configured in other embodiments.

[0078] Based on the above embodiments, the features of the lung lobe fissures under three perspectives can be fully integrated to improve the information content and accuracy of the corrected fissure features, thereby improving the accuracy of the lung lobe segmentation results.

[0079] In order to explain the embodiment of the present invention in detail, each process of the embodiment of the present invention is described below respectively.

[0080] In the embodiment of the present invention, the method for obtaining the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the cross section is:

[0081] A multi-sequence lung image in the sagittal plane, the coronal plane and the cross-section is obtained; and lobe fissure features are extracted from the multi-sequence lung images in the sagittal plane, the coronal plane and the cross-section, respectively, to obtain lobe fissure features in the sagittal plane, lobe fissure features in the coronal plane and lobe fissure features in the cross-section.

[0082] The embodiment of the present invention can first obtain multi-sequence lung images under three viewing angles. As described in the above embodiment, multi-layer lung images (multi-sequence images) of the lung images under different viewing angles can be collected by CT imaging, and a three-dimensional lung image can be obtained through the multi-layer lung images under each viewing angle.

[0083] When a multi-sequence lung image under three viewing angles is obtained, feature extraction processing can be performed on each lung image, for example, the lung image under each viewing angle is subjected to feature extraction processing by the feature extraction neural network, and the lobe fissure features of each image under the three viewing angles are obtained, such as the lobe fissure features under the sagittal plane, the lobe fissure features under the coronal plane, and the lobe fissure features under the cross section. Among them, since each viewing angle may include multiple lung images, the embodiment of the present invention can perform feature extraction processing of the multiple lung images in parallel through multiple feature extraction neural networks, thereby improving feature extraction efficiency.

[0084] Figure 3 FIG. 1 is a schematic diagram of a network structure of a lung lobe segmentation method according to an embodiment of the present invention. Figure 3 As shown, the network that performs feature extraction processing in the embodiment of the present invention can be a U network (U-net) or other convolutional neural networks that can perform feature extraction.

[0085] When the pulmonary lobe fissure features of the lung images under various viewing angles are obtained, the third pulmonary lobe fissure feature can be corrected using any two pulmonary lobe fissure features under the sagittal plane, the coronal plane, and the transverse plane. The process may include:

[0086] The any two pulmonary lobe fissure features are mapped to the viewing angle of the third pulmonary lobe fissure feature; and the third pulmonary lobe fissure feature is corrected using the mapped any two pulmonary lobe fissure features.

[0087] For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is taken as an example to illustrate below.

[0088] Since the extracted lobe fissure features are different under different viewing angles, an embodiment of the present invention can convert the lobe fissure feature maps under three viewing angles to one viewing angle. Among them, the method of mapping any two lobe fissure features to the viewing angle where the third lobe fissure feature is located is: mapping the lobe fissure features of any two multi-sequence lung images of the sagittal plane, coronal plane and cross-section to the viewing angle where the third lobe fissure feature is located. In other words, the first lobe fissure feature and the second lobe fissure feature can be converted to the viewing angle where the third lobe fissure feature is located. Among them, through the mapping conversion of the viewing angle, the feature information of the viewing angle before mapping can be integrated into the lobe fissure feature obtained after mapping.

[0089] As described in the above embodiment, the embodiment of the present invention can obtain multiple lung images at each viewing angle, and the multiple lung images correspond to multiple pulmonary lobe fissure features. Each feature value in the pulmonary lobe fissure feature corresponds one-to-one to each pixel point of the corresponding lung image.

[0090] The embodiment of the present invention can determine the position mapping relationship between each pixel point in the lung image when the perspective is converted to another perspective based on the three-dimensional lung image formed by multiple lung images at one perspective, such as when a certain pixel point switches from the first position of the first perspective to the second position of the second perspective, at this time, the feature value corresponding to the first position at the first perspective is mapped to the second position. Through the above embodiment, the mapping conversion between the lung lobe fissure features of each lung image at different perspectives can be achieved.

[0091] In some possible implementations, when the lobe fissure features of three perspectives are mapped to the same perspective, the two mapped lobe fissure features can be used to perform correction processing on the third lobe fissure feature to improve the information content and accuracy of the third lobe fissure feature.

[0092] In the embodiment of the present invention, the method for correcting the third lobe fissure feature by using any two of the mapped lobe fissure features is:

[0093] The spatial attention feature fusion is performed using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure feature to obtain a first fusion feature and a second fusion feature; and the corrected third pulmonary lobe fissure feature is obtained based on the first fusion feature and the second fusion feature.

[0094] In the embodiment of the present invention, the feature after the first lobe fissure feature is mapped may be referred to as the first mapping feature, and the feature after the second lobe fissure feature is mapped may be referred to as the second mapping feature. When the first mapping feature and the second mapping feature are obtained, a spatial attention feature fusion between the first mapping feature and the third lobe fissure feature may be performed to obtain a first fused feature, and a spatial attention feature fusion between the second mapping feature and the third lobe fissure feature may be performed to obtain a second fused feature.

[0095] The method of fusing the spatial attention features by respectively using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure to obtain the first fused feature and the second fused feature is:

[0096] Respectively connect any two of the lobe fissure features with the third lobe fissure feature to obtain a first connection feature and a second connection feature; perform a first convolution operation on the first connection feature to obtain a first convolution feature, and perform a first convolution operation on the second connection feature to obtain a second convolution feature; perform a second convolution operation on the first convolution feature to obtain a first attention coefficient, and perform a second convolution operation on the second convolution feature to obtain a second attention coefficient; use the first convolution feature and the first attention coefficient to obtain the first fusion feature, and use the second convolution feature and the second attention coefficient to obtain the second fusion feature.

[0097] In some possible implementations, such as Figure 3 As shown, the above-mentioned spatial attention feature fusion processing can be performed by the network module of the spatial attention mechanism. The embodiment of the present invention takes into account the different importance of lobe fissure features at different positions and adopts a spatial attention mechanism. Among them, the convolution processing based on the attention mechanism can be implemented through the spatial attention neural network (attention), and the important features are further highlighted in the obtained fusion features. In the training process of the spatial attention neural network, the importance of each position of the spatial feature can be adaptively learned to form an attention coefficient with the feature object at each position. For example, the coefficient can represent a coefficient value in the interval [0, 1]. The larger the coefficient, the more important the feature at the corresponding position.

[0098] In the process of performing spatial attention fusion processing, the first mapping feature and the third lobe fissure feature can first be concatenated to obtain a first connection feature, and the second mapping feature and the third lobe fissure feature can be concatenated to obtain a second connection feature. The above connection processing can be concatenated in the channel direction. In an embodiment of the present invention, the scales of the first mapping feature, the second mapping feature, and the third lobe fissure feature can all be identified as (C / 2, H, W), where C represents the number of channels of each feature, H represents the height of the feature, and W represents the width of the feature. Correspondingly, the scales of the first connection feature and the second connection feature obtained by the connection processing can be expressed as (C, H, W).

[0099] When the first connection feature and the second connection feature are obtained, the first convolution operation can be performed on each of the first connection feature and the second connection feature, such as using the convolution layer A to perform the first convolution operation through a 3*3 convolution kernel, and then batch normalization (bn) and activation function (relu) processing can be performed to obtain the first convolution feature corresponding to the first connection feature and the second convolution feature corresponding to the second connection feature. The scale of the first convolution feature and the second convolution feature can be expressed as (C / 2, H, W). The first convolution operation can reduce the parameters in the feature map and reduce the subsequent calculation cost.

[0100] In some possible implementations, when the first convolution feature and the second convolution feature are obtained, the second convolution operation and the sigmoid function processing can be performed on the first convolution feature and the second convolution feature respectively to obtain the corresponding first attention coefficient and second attention coefficient respectively. The first attention coefficient can represent the importance of the features of each element of the first convolution feature, and the second attention coefficient can represent the importance of the features of the elements in the second convolution feature.

[0101] like Figure 3As shown, for the first convolution feature or the second convolution feature, two convolution layers B and C can be used to perform the second convolution operation, wherein the convolution layer B is processed by a 1*1 convolution kernel, and then batch normalization (bn) and activation function (relu) are performed to obtain the first intermediate feature. The scale of the first intermediate feature map can be expressed as (C / 8, H, W), and then the second convolution layer C is used to perform a convolution operation of a 1*1 convolution kernel on the first intermediate feature map to obtain a second intermediate feature map of (1, H, W). Further, the second intermediate feature map can be processed by an activation function using a sigmoid function to obtain the attention coefficient corresponding to the first convolution feature or the second performance feature, and the coefficient value of the attention coefficient can be a value in the range of [0,1].

[0102] Through the above-mentioned second convolution operation, the first connection feature and the second connection feature can be subjected to dimensionality reduction processing to obtain the attention coefficient of a single channel.

[0103] In some possible implementations, when a first attention coefficient corresponding to a first convolution feature and a second attention coefficient corresponding to a second convolution feature are obtained, a product process may be performed on the first convolution feature and the first attention coefficient, and the product result may be added to the first convolution feature to obtain a first fused feature. And a product process may be performed on the second convolution feature and the second attention coefficient matrix, and the product result may be added to the second convolution feature to obtain a second fused feature. Among them, the product process (mul) may be corresponding element multiplication, and the feature addition (add) may be corresponding element addition. In the above manner, effective fusion of features from three perspectives may be achieved.

[0104] Alternatively, in some other implementations, the feature obtained by multiplying the first convolution feature by the first attention coefficient may be added to the first convolution feature, and the added feature may be subjected to several convolution operations to obtain the first fused feature; and the feature obtained by multiplying the second convolution feature by the second attention coefficient may be added to the second convolution feature, and the added feature may be subjected to several convolution operations to obtain the second fused feature. In this way, the accuracy of the fused feature may be further improved, and the information content of the fusion may be improved.

[0105] When the first fusion feature and the second fusion feature are obtained, the first fusion feature and the second fusion feature may be used to obtain a corrected third lobe fissure feature.

[0106] In some possible implementations, since the first fused feature and the second fused feature respectively include feature information from three perspectives, the first fused feature and the second fused feature can be directly connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature. Alternatively, the first fused feature, the second fused feature, and the third lobe fissure feature can be connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature.

[0107] The third convolution operation may include group convolution processing. The third convolution operation may further achieve further fusion of feature information in each feature. Figure 3 As shown, the third convolution operation of the embodiment of the present invention may include grouped convolution D (depth wise conv), wherein the grouped convolution can speed up the convolution speed while improving the accuracy of the convolution features.

[0108] When the corrected third lobe fissure feature is obtained through the third convolution operation, the lung image can be segmented using the corrected lobe fissure feature. The embodiment of the present invention can obtain the segmentation result corresponding to the corrected lobe fissure feature by convolution. Figure 3 As shown, in the embodiment of the present invention, the corrected lobe fissure features can be input into the convolution layer E, and the standard convolution is performed through the 1*1 convolution kernel to obtain the segmentation result of the lung image. As described in the above embodiment, the segmentation result can represent the location areas where the five lobes in the lung image are located. Figure 3 As shown, each lung lobe area in the lung image is distinguished by light and dark filling colors.

[0109] Based on the above configuration, the lung lobe segmentation method based on multiple perspectives provided by the embodiment of the present invention can solve the technical problem of not fully utilizing the information from other perspectives to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0110] As described in the above embodiments, the embodiments of the present invention can be implemented by a neural network, such as Figure 4 As shown, the neural network for executing the lung lobe segmentation method under multi-viewpoints in an embodiment of the present invention may include a feature extraction neural network, a spatial attention neural network, and a segmentation network (including convolutional layers D and E).

[0111] The embodiment of the present invention may include three feature extraction neural networks, which are respectively used to extract the features of the lung lobe fissures under different viewing angles. Among them, the three feature extraction networks can be referred to as the first branch network, the second branch network and the third branch network. Among them, the structures of the three branch networks of the embodiment of the present invention are exactly the same, and the input images of each branch network are different. For example, the sagittal lung image sample is input to the first branch network, the coronal lung image sample is input to the second branch network, and the transverse lung image sample is input to the third branch network, so as to respectively perform feature extraction processing of the lung image samples under each viewing angle.

[0112] Specifically, in the embodiment of the present invention, the process of training the feature extraction neural network includes:

[0113] Acquire training samples in the sagittal plane, the coronal plane and the transverse plane, wherein the training samples are lung image samples with marked lobe fissure features; use the first branch network to perform feature extraction on the lung image samples in the sagittal plane to obtain a first predicted lobe fissure feature; use the second branch network to perform feature extraction on the lung image samples in the coronal plane to obtain a second predicted lobe fissure feature; use the third branch network to perform feature extraction on the lung image samples in the transverse plane to obtain a third predicted lobe fissure feature; use the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature and the corresponding marked lobe fissure feature to obtain the network losses of the first branch network, the second branch network and the third branch network, and use the network losses to adjust the parameters of the first branch network, the second branch network and the third branch network.

[0114] As described in the above embodiment, the first branch network, the second branch network and the third branch network are respectively used to perform feature extraction processing of lung image samples in the sagittal plane, the coronal plane and the cross-sectional perspective, and then the corresponding predicted features, namely the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature, can be obtained.

[0115] When each predicted lobe fissure feature is obtained, the first predicted lobe fissure feature, the second predicted lobe fissure feature, and the third predicted lobe fissure feature can be used together with the corresponding labeled lobe fissure feature to obtain the network losses of the first branch network, the second branch network, and the third branch network. For example, the loss function of the embodiment of the present invention can be a logarithmic loss function, and the network loss of the first branch network can be obtained by the first predicted lobe fissure feature and the labeled real lobe fissure feature, the network loss of the second branch network can be obtained by the second predicted lobe fissure feature and the labeled real lobe fissure feature, and the network loss of the third branch network can be obtained by the third predicted lobe fissure feature and the labeled real lobe fissure feature.

[0116] When the network loss of each branch network is obtained, the parameters of the first branch network, the second branch network and the third branch network can be adjusted according to the network loss of each network until the termination condition is met. Among them, the embodiment of the present invention can use the network loss of any branch of the first branch network, the second branch network and the third branch network to adjust the network parameters of the first branch network, the second branch network and the third branch network, such as convolution parameters, etc., respectively. Thereby, the network parameters under any perspective can be correlated with the features under the other two perspectives, the correlation between the extracted lobe fissure features and the lobe fissure features under the other two perspectives can be improved, and the preliminary fusion of the lobe fissure features under each perspective can be achieved.

[0117] In addition, the training termination condition of the feature extraction neural network is that the network loss of each branch network is less than the first loss threshold, which indicates that each branch network of the feature extraction neural network can accurately extract the lobe fissure features of the lung image under the corresponding viewing angle.

[0118] When the feature extraction neural network is trained, the feature extraction neural network, the spatial attention neural network and the segmentation network can be used for training at the same time, and the segmentation result output by the segmentation network and the corresponding marking result in the marked lobe fissure feature can be used to determine the network loss of the entire neural network. The network loss of the entire neural network is further used to feedback and adjust the network parameters of the feature extraction neural network, the spatial attention neural network and the segmentation network until the network loss of the entire neural network is less than the second loss threshold. In the embodiment of the present invention, the first loss threshold is greater than or equal to the second loss threshold, so that the network accuracy of the network can be improved.

[0119] When applying the neural network of the embodiment of the present invention to perform lung lobe segmentation based on multiple perspectives, the lung images of the same lung under different perspectives can be input into the three branch networks respectively, and finally the final lung image segmentation result can be obtained through this neural network.

[0120] In summary, the lung lobe segmentation method and device provided in the embodiments of the present invention can fuse multi-view feature information and perform lung lobe segmentation of lung images, thereby solving the problem of not fully utilizing information from other viewpoints to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0121] In addition, an embodiment of the present invention can also perform pulmonary lobe segmentation through a pulmonary lobe segmentation device, which includes: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the pulmonary lobe segmentation method in the above embodiment.

[0122] In some embodiments, the functions or modules included in the device provided by the embodiments of the present invention can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0123] In an embodiment of the present invention, in step 20, the first disease and the second disease of the lesion area of ​​each lobe in the lobe segmentation image are determined: the first disease of the lesion area of ​​each lobe in the lobe segmentation image is COPD, and the second disease of the lesion area of ​​each lobe in the lobe segmentation image is small airway lesions. Or the first disease of the lesion area of ​​each lobe in the lobe segmentation image is lung cancer, and the second disease of the lesion area of ​​each lobe in the lobe segmentation image is pulmonary nodules. Among them, the detection of COPD and pulmonary nodules can be done by existing detection methods, which are not described in the embodiments of the present invention.

[0124] Specifically, in step S20, it is necessary to respectively determine the first disease of the lesion area of ​​the left upper lobe and the left lower lobe of the left lung image and the second disease associated with the first disease; and to respectively determine the first disease of the lesion area of ​​the right upper lobe, the right middle lobe, and the right lower lobe of the right lung image and the second disease associated with the first disease.

[0125] In an embodiment of the present invention, if the first disease is chronic obstructive pulmonary disease and the second disease is small airway lesions. Then before the lobe segmentation image is obtained, a lung CT image is obtained, and the CT image includes: a full inspiratory phase lung image and a full expiratory phase lung image. Correspondingly, the obtained lobe segmentation image may include segmenting the full inspiratory phase lung image and the full expiratory phase lung image respectively to obtain a first lobe segmentation image and a second lobe segmentation image; wherein the first lobe segmentation image is the lobe segmentation result of the full inspiratory phase lung image, and the second lobe segmentation image is the lobe segmentation result of the full expiratory phase lung image. The method for determining the first disease of the lesion area of ​​each lobe in the lobe segmentation image and the second disease associated with the first disease is: determining the first disease according to the first lobe segmentation image or the second lobe segmentation image; determining the second disease according to the first lobe segmentation image and the second lobe segmentation image; wherein the first disease is chronic obstructive pulmonary disease and the second disease is small airway lesions.

[0126] In addition, if the first disease is lung cancer and the second disease is lung nodules, a lung CT image can be obtained before obtaining the lung lobe segmentation image, and the CT image includes: a lung image obtained by holding the breath after a full lung scan after one inhalation. Correspondingly, a lung segmentation image of the lung image can be obtained by lung image segmentation processing, that is, each lung lobe image can be obtained. The method for determining the first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image and the second disease associated with the first disease is: determining the corresponding first disease and the associated second disease according to each lung lobe image. For example, each lung lobe image corresponding to the lung segmentation image can be input into a neural network that has been trained to identify lung nodules and lung cancer. The first disease (lung cancer) and the associated second disease (pulmonary nodule) of each lung lobe are detected by the neural network. Preferably, the lesion areas corresponding to the first disease and the second disease can also be obtained. The neural network may include feature extraction of a lung lobe segmentation image, a region candidate network, and a classification network. The feature extraction network performs feature extraction processing on the lung lobe image to obtain lung lobe features. The region candidate network performs convolution processing on the lung lobe features to obtain lesion regions of the first disease and the second disease in the lung lobe image. The classification network performs classification and identification of the disease type of the lesion region to obtain the disease type (the first disease or the second disease) corresponding to the lung lobe segmentation image.

[0127] It should be noted that, in the process of detecting the first disease and the associated second disease of each lobe, not every lobe necessarily has the first disease and the second disease, and there may be a situation where only the second disease exists, or there may be a situation where neither the first disease nor the second disease exists. The embodiment of the present invention can use different disease identifiers to indicate different diseases, and whether a disease exists.

[0128] In an embodiment of the present invention, before step S30, the method further includes: determining a known level of the first disease. The step of determining a known level of the first disease includes: inputting the segmented images of each lung lobe into a preset neural network to extract image features through a residual network of the preset neural network, and determining the known level of the first disease according to a classification network of the preset neural network.

[0129] In the step of determining the known level of the first disease, it is necessary to determine the known levels of the first disease in the lesion areas of the left upper lobe and the left lower lobe of the left lung image respectively; and to determine the known levels of the first disease in the lesion areas of the right upper lobe, the right middle lobe, and the right lower lobe of the right lung image respectively. That is, the known level of the first disease determined in step 103 is also the known level of the first disease in each lobe of the lung.

[0130] In an embodiment of the present invention, a neural network can be used to detect the known level of the first disease of each lobe. For example, each lobe segmentation image obtained by segmentation can be input into the neural network, and the known level of the first disease of each lobe image can be classified and detected through the processing of the neural network. The neural network can include a convolutional neural network, for example, it can perform feature extraction of the lobe segmentation image (for example, through a residual network), and after obtaining the lobe features, the first disease level corresponding to the lobe features can be detected through a classification network (multiple classifiers). Different classification levels can be set for different first diseases.

[0131] If the first disease is COPD, according to the Global Initiative for Chronic Obstructive Lung Disease (GOLD), the levels of COPD include: GOLD level 0 (non-existent), GOLD level 1 (mild), GOLD level 2 (moderate), GOLD level 3 (severe) and GOLD level 4 (extremely severe).

[0132] In an embodiment of the present invention, a method for determining each lobe on each lung lobe according to each lung lobe, comprises: determining the first volume of the COPD area on each lung lobe and the total volume of each lung lobe, and determining the grade of COPD on each lung lobe according to the ratio of the first volume to the total volume.

[0133] For example, four thresholds are set, namely, a first threshold, a second threshold, a third threshold and a fourth threshold. If the ratio of the first volume to the total volume is less than the first threshold, the grade of COPD is GOLD0 (non-existent); if the ratio of the first volume to the total volume is between the first threshold and the second threshold, the grade of COPD is GOLD1 (mild); if the ratio of the first volume to the total volume is between the second threshold and the third threshold, the grade of COPD is GOLD2 (moderate); if the ratio of the first volume to the total volume is between the third threshold and the fourth threshold, the grade of COPD is GOLD3 (severe); if the ratio of the first volume to the total volume is greater than the fourth threshold, the grade of COPD is GOLD4 (extremely severe). The first threshold, the second threshold, the third threshold and the fourth threshold may be 20%, 30%, 50% and 80% respectively.

[0134] For example, if the ratio of the first volume of the COPD area of ​​the left upper lobe to the total volume of the left upper lobe is between the second threshold and the third threshold, the grade of COPD is GOLD grade 2 (moderate). If the first disease is lung cancer, the grades of lung cancer include: stage I, stage II, stage III, stage IV, and stage III and IV.

[0135] Stage I: The tumor is relatively small and has no lymph node metastasis, so it can be completely removed by surgery. According to the size of the tumor, it is divided into stage IA and stage IB. Smaller tumors are stage IA, and larger tumors are stage IB.

[0136] Stage II: It can also be divided into two subtypes: stage IIA and stage IIB. Stage IIA includes two situations: one is a slightly larger tumor without adjacent lymph node metastasis, and the other is a smaller tumor with peripheral lymph node metastasis. Stage IIB refers to a larger tumor with lymph node metastasis, or a large tumor with or without involvement of peripheral lung structures, but no lymph node metastasis.

[0137] Stage III: It is divided into stages IIIA and IIIB. Many stage IIIA and almost all stage IIIB tumors are difficult or impossible to remove surgically. For example, the tumor may involve the mediastinal lymph nodes or invade adjacent structures in the lungs. In some cases, due to various factors, the tumor cannot be completely removed in one go and can only be removed in multiple times. In this case, it is difficult to completely remove the tumor.

[0138] Stage IV: includes: cancer cells have spread to multiple parts of the opposite lung, or to fluid around the lung or heart, or to other parts of the body through the bloodstream. Once cancer cells enter the bloodstream, they can spread to any part of the body, but the most common sites of metastasis are the brain, bones, liver, and adrenal glands.

[0139] Stage III and IV: Lung cancer is in the middle and late stages and cannot be cured by surgery. Radiotherapy and chemotherapy are generally the main treatment methods. In the late stages, the main focus is on reducing the patient's pain and palliative care. When lung cancer has the following conditions, surgical resection is not possible, including: tumor metastasis to the supraclavicular lymph nodes, or tumor involving important organs in the chest cavity, such as the heart, large blood vessels or trachea.

[0140] Through the above configuration, the level of the first disease corresponding to each lung lobe in the lung lobe segmentation image can be obtained.

[0141] For the embodiment of the present invention, the step S30, the step of determining the trend level of the first disease of each lung lobe based on the known level of the first disease and the second disease, includes: locating the first area of ​​the first disease and the second area of ​​the second disease of each lung lobe respectively; based on the known level of the first disease, determining the trend level of the first disease of each lung lobe according to the ratio of the first area to the second area of ​​each lung lobe. The known level of the first disease is the disease level represented by the current lung image, and the trend level of the first disease is used to represent the future development of the disease, that is, the trend level of the first disease can be used to determine whether the first disease is getting better or worse, and the degree of improvement or worsening can be represented by the level number of the trend level.

[0142] In an embodiment of the present invention, based on the known level of the first disease, the trend level of the first disease in each lobe is determined according to the ratio of the first area to the second area of ​​each lobe, including: obtaining the first volume of the first disease in each lobe according to the first area, and obtaining the second volume of the second disease in each lobe according to the second area; calculating the ratio of the first volume to the second volume for each lobe; based on the level of the first disease, determining the trend level of the first disease in the corresponding lobe according to the ratio of the first volume to the second volume of each lobe.

[0143] The first volume is the total volume of the first disease in the first area of ​​each lung lobe; the second volume is the total volume of the second disease in the second area of ​​each lung lobe.

[0144] The first volume and the second volume are the first volume of the lesion area of ​​the first disease and the second volume of the lesion area of ​​the second disease of each lung lobe, respectively. That is, the first volume includes: the first volume of the lesion area of ​​the first disease of the left upper lobe, the first volume of the lesion area of ​​the first disease of the left lower lobe, the first volume of the lesion area of ​​the first disease of the right upper lobe, the first volume of the lesion area of ​​the first disease of the right middle lobe, and the first volume of the lesion area of ​​the first disease of the right lower lobe. The second volume includes: the second volume of the lesion area of ​​the second disease of the left upper lobe, the second volume of the lesion area of ​​the second disease of the left lower lobe, the second volume of the lesion area of ​​the second disease of the right upper lobe, the second volume of the lesion area of ​​the second disease of the right middle lobe, and the second volume of the lesion area of ​​the second disease of the right lower lobe.

[0145] Alternatively, in an embodiment of the present invention, based on the known level of the first disease, the trend level of the first disease in each lobe is determined according to the ratio of the first area to the second area of ​​each lobe, including: in response to the ratio of the first volume to the second volume being greater than a preset threshold, the next level lighter than the known level of the first disease is determined as the trend level of the first disease; in response to the ratio of the first volume to the second volume being less than or equal to the preset threshold, the previous level more serious than the known level of the first disease is determined as the trend level of the first disease.

[0146] In addition, in an embodiment of the present invention, based on the known level of the first disease, the trend level of the first disease in each lobe is determined according to the ratio of the first area to the second area of ​​each lobe, including: in response to the ratio of the first area to the second area being greater than a preset threshold, the next level lighter than the known level of the first disease is determined as the trend level of the first disease; in response to the ratio of the first area to the second area being less than or equal to the preset threshold, the previous level more serious than the known level of the first disease is determined as the trend level of the first disease.

[0147] In an embodiment of the present invention, the first area of ​​the first disease and the second area of ​​the second disease of each lobe can be located respectively. When the first disease of the lesion area of ​​each lobe in the lobe segmentation image is lung cancer, the second disease of the lesion area of ​​each lobe in the lobe segmentation image is lung nodule, and when the first disease of the lesion area of ​​each lobe in the lobe segmentation image is chronic obstructive pulmonary disease, the second disease of the lesion area of ​​each lobe in the lobe segmentation image is small airway lesions.

[0148] Whether it is lung cancer or lung nodules, the features on the image are very obvious. If the first disease is lung cancer, the lung cancer area (first area) can be located according to the morphological characteristics of lung cancer; if the second disease is lung nodules, the lung cancer area (second area) can be located according to the morphological characteristics of lung nodules. As described in the above embodiment, the prediction of the first disease and the second disease, as well as the determination of the lesion areas of the first disease and the second disease can be achieved by a neural network that can identify the first disease and the second disease corresponding to the segmented image of the lung lobe.

[0149] For example, if the first disease is COPD and the second disease is small airway lesions, the area of ​​COPD (first area) or the area of ​​small airway lesions (second area) cannot be determined on the image, especially the area of ​​small airway lesions, which is not obvious on the image.

[0150] The method for locating the first area of ​​the COPD of each lobe is: judging whether the CT value on the lobe is a COPD area according to the lobe with CT value and the set threshold. Wherein, the set threshold is the set CT value. Since the CT value of the COPD area does not change substantially during deep inhalation, and other normal areas (non-COPD areas) enter the air, the CT value of the air is 1024HU. Since the COPD area is full of air, the CT of the COPD area is close to 1024HU. In the medical field, the set threshold is generally selected as -950HU. If the CT value of the lobe is less than -950HU, it is judged as a COPD area. If the CT value of the lobe is greater than or equal to -950HU, it is judged not to be a COPD area. In other published papers or in some possible implementations, the set threshold may fluctuate. The present invention does not specifically limit the set threshold, and those skilled in the art can appropriately adjust the set threshold.

[0151] As described above, locating the second area of ​​the small airway lesion in each lung lobe requires a full inspiratory phase lung image and a full expiratory phase lung image. The method for locating the second area of ​​the small airway lesions in each lung lobe is as follows: obtaining a first lung lobe segmentation image of a full inspiratory phase lung image; obtaining a second lung lobe segmentation image of a full expiratory phase lung image; respectively extracting multiple full inspiratory phase single lung lobes with CT values ​​in the first lung lobe segmentation image; respectively extracting multiple full expiratory phase single lung lobes with CT values ​​in the second lung lobe segmentation image; respectively registering the full inspiratory phase single lung lobe and the full expiratory phase single lung lobe at corresponding positions to obtain the registered full inspiratory phase single lung lobe and the registered full expiratory phase single lung lobe; comparing the CT values ​​of the registered full inspiratory phase single lung lobe and the registered full expiratory phase single lung lobe with the inspiratory phase set threshold and the expiratory phase set threshold, respectively; if the CT value of the registered full inspiratory phase single lung lobe is less than the inspiratory phase set threshold and the CT value of the registered full expiratory phase single lung lobe is less than the expiratory phase set threshold, it is considered that there is small airway lesions in this area; otherwise, it is considered that there is no small airway lesions in this area.

[0152] Among them, the segmentation method for segmenting the full inhalation phase lung image to obtain the first lung lobe segmentation image and the segmentation method for segmenting the full expiratory phase lung image to obtain the second lung lobe segmentation image adopts the above-mentioned method for segmenting the lung image.

[0153] In the embodiment of the present invention, the full inspiratory phase lung image and the full expiratory phase lung image are lung images of a patient, and the full inspiratory phase lung image is a lung image taken by an imaging device during deep inhalation while maintaining the maximum lung air volume. Similarly, the full expiratory phase lung image is a lung image taken by an imaging device during deep exhalation while maintaining the minimum lung air volume. The full inspiratory phase lung image and the full expiratory phase lung image can be achieved by radiologists in hospitals with the help of imaging equipment (such as CT).

[0154] The registration algorithm for the single lung lobe in the full inspiratory phase and the single lung lobe in the full expiratory phase can use an elastic registration algorithm or utilize a VGG network (VGG-net) in deep learning for registration. The embodiment of the present invention does not limit the specific registration algorithm.

[0155] In the embodiment of the present invention, the inspiratory phase setting threshold and the expiratory phase setting threshold can be set by those skilled in the art as needed. For example, the inspiratory phase setting threshold can be set to -950HU, and the expiratory phase setting threshold can be set to -856HU. The registered single lobe of the whole inspiratory phase is compared with the inspiratory phase setting threshold of -950HU, and the CT value of the single lobe of the whole expiratory phase after the registration is compared with the expiratory phase setting threshold of -856HU.

[0156] In an embodiment of the present invention, for example, the inspiratory phase setting threshold can be set to -950HU, the expiratory phase setting threshold can be set to -856HU, and the CT value of a single lung lobe in the full inspiratory phase after registration is less than the inspiratory phase setting threshold -950HU, and the CT value of a single lung lobe in the full expiratory phase after registration is less than the expiratory phase setting threshold -856HU, then it is considered that small airway lesions exist in this area; otherwise, it is considered that small airway lesions do not exist in this area.

[0157] In the present invention, the first volume of the lung cancer in each lobe is obtained according to the first area of ​​the lung cancer, and the second volume of the lung nodule in each lobe is obtained according to the second area of ​​the lung nodule. In the image, the characteristics of lung cancer and lung nodules are very obvious, and their volumes are well determined. The lung cancer and the lung nodules can be approximated into a circle or an ellipse for volume calculation to obtain the first volume of the lung cancer in each lobe and the second volume of the lung nodule in each lobe. The first volume is the total volume of all lung cancers in one lobe, and the second volume is the total volume of all lung nodules in this lobe. For example, in the above embodiment, the first area of ​​lung cancer and the second area of ​​lung nodules in each lobe can be obtained, and the first area and the second area can be fitted into a circle (which can be regarded as a sphere from the three-dimensional structure) by surface fitting, so that the first volume of the first area corresponding to the lung cancer and the second volume of the second area corresponding to the lung nodule can be conveniently calculated according to the fitting results.

[0158] In lung cancer and lung nodules, based on the known level of lung cancer, the trend level of the first disease in the corresponding lobe is determined according to the ratio of the first volume to the second volume of each lobe and a preset threshold.

[0159] For example, in the left upper lobe, based on the known grade of the lung cancer, the trend grade of the lung cancer in the left upper lobe may be determined according to the ratio of the first volume of the left upper lobe to the second volume of the left upper lobe and a preset threshold.

[0160] The method for determining the trend level of the first disease in the corresponding lobe based on the known level of the first disease and according to the ratio of the first area and the second area of ​​each lobe and a preset threshold is: if the ratio is greater than the preset threshold, the trend level is the next level of the known level of the first disease; otherwise, the trend level is the previous level of the known level of the first disease; wherein the previous level is more serious than the level of the first disease.

[0161] For example, in the left upper lobe, the grade of lung cancer is stage IB, and the first volume of the left upper lobe lung cancer area is divided by the second volume of the left upper lobe lung nodule area to obtain a ratio. The ratio is less than or equal to the preset threshold, indicating that although the left upper lobe lung cancer area is small, the lung nodule area is large, and the possibility of the lung nodule area being transformed into a lung cancer area is high. The grade of lung cancer in the left upper lobe can be predicted to stage IIA or stage IIB in the previous stage II. If the ratio is greater than the preset threshold, the grade of lung cancer in the left upper lobe can be predicted to the next stage IA. Among them, in the case where the first disease is lung cancer, the preset threshold can be 0.2.

[0162] In an embodiment of the present invention, the first volume of COPD on each lung lobe is obtained according to the first area of ​​COPD, and the second volume of small airway lesions on each lung lobe is obtained according to the second area of ​​small airway lesions. In the image, the characteristics of small airway lesions are not obvious, and it is not very easy to calculate the first volume and the second volume on each lung lobe. Therefore, the lung lobe segmentation image can be gridded, and the grid volumes of the first area and the second area after gridding are calculated respectively. The sum of the grid volumes of the first area is the first volume, and the sum of the grid volumes of the second area is the second volume.

[0163] In an embodiment of the present invention, the specific method for gridding the lung lobe segmentation image is: determine the number of layers of the full inspiratory phase lung image or the full expiratory phase lung image; grid the COPD region of each lung lobe in each layer and the small airway lesion region of each lung lobe to obtain the grid area of ​​the COPD region of each lung lobe and the grid area of ​​the small airway lesion region of each lung lobe, and determine the volume of the COPD region of each lung lobe (first volume) and the volume of the small airway lesion region of each lung lobe (second volume) according to the grid area of ​​the COPD region of each lung lobe and the grid area of ​​the small airway lesion region, the number of layers, the scanning layer thickness and the inter-layer spacing. Wherein, the number of layers, the scanning layer thickness and the inter-layer spacing of the full inspiratory phase lung image are the same as those of the full expiratory phase lung image.

[0164] In an embodiment of the present invention, the method for determining the volume (first volume) of the COPD region of each lung lobe is as follows: using the grid area corresponding to the two adjacent layers of COPD regions of each lung lobe, as well as the inter-layer spacing and layer thickness, obtain the first sub-volume formed by the two adjacent layers of COPD regions, and using the sum of the first sub-volumes formed by the two adjacent layers of COPD regions of all lung lobes to obtain the first volume of the COPD region of the lung lobe. Among them, the structure formed by the two adjacent layers of COPD regions can be regarded as a prism, and the area of ​​the upper and lower bottom surfaces of the prism is the grid area corresponding to the COPD region. The height of the prism can be determined by the inter-layer spacing and layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of COPD regions can be the sum of the layer thickness*2 and the inter-layer spacing, and the height of the prism formed by the remaining two adjacent layers of COPD regions can be the sum of the layer thickness and the inter-layer spacing. Based on the upper and lower bottom areas and the height, the first sub-volume of the COPD region formed by the adjacent layers can be determined. Then, the sum of the first sub-volumes can be used to obtain the first volume.

[0165] The method for determining the volume (second volume) of the small airway lesion area of ​​each lung lobe is as follows: using the grid area corresponding to the two adjacent layers of small airway lesion areas of each lung lobe, as well as the interlayer spacing and layer thickness, obtain the second subvolume formed by the two adjacent layers of small airway lesion areas, and using the sum of the second subvolumes formed by the two adjacent layers of small airway lesion areas of all lung lobes to obtain the second volume of the small airway lesion areas of the lung lobe. Among them, the structure formed by the two adjacent layers of small airway lesion areas can be regarded as a prism, and the area of ​​the upper and lower bottom surfaces of the prism is the grid area corresponding to the small airway lesion area. The height of the prism can be determined by the interlayer spacing and layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of small airway lesion areas can be the sum of the layer thickness*2 and the interlayer spacing, and the height of the prism formed by the remaining two adjacent layers of small airway lesion areas can be the sum of the layer thickness and the interlayer spacing. Based on the upper and lower bottom areas and the height, the second subvolume of the small airway lesion area formed by the adjacent layers can be determined. Then, the second volume can be obtained by the sum of the second subvolumes.

[0166] In an embodiment of the present invention, the specific method of gridding the COPD area of ​​each layer and each lung lobe to obtain the grid area of ​​the COPD area of ​​each layer and each lung lobe is as follows: Step 1: Determine the COPD area edge of each layer and each lung lobe respectively; determine a first standard shape in the COPD area of ​​each layer and each lung lobe and in the small airway lesion area of ​​each lung lobe respectively; Step 2: The first standard shape extends to the COPD area edge of each layer and each lung lobe; Step 3: When several set points of the standard shape touch the COPD area edge of each layer and each lung lobe, the first standard shape stops extending and generates a second standard shape outside the standard shape; generate according to steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the areas of all standard shapes, and obtain the grid area of ​​the COPD area of ​​each layer and each lung lobe; wherein the areas of the standard shape, the second standard shape, and the standard shape corresponding to the set number decrease in sequence.

[0167] In an embodiment of the present invention, the specific method for gridding the small airway lesion area of ​​each layer and each lung lobe to obtain the grid area of ​​the small airway lesion area of ​​each layer and each lung lobe is: Step 1: Determine the edge of the small airway lesion area of ​​each layer and each lung lobe respectively; Determine a first standard shape in the small airway lesion area of ​​each layer and each lung lobe respectively; Step 2: The first standard shape extends to the edge of the small airway lesion area of ​​each layer and each lung lobe; Step 3: When several set points of the standard shape touch the edge of the small airway lesion area of ​​the left lung of each layer and each lung lobe, the first standard shape stops extending and generates a second standard shape outside the standard shape; Generate in accordance with steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the areas of all standard shapes, and obtain the grid area of ​​the small airway lesion area of ​​each layer and each lung lobe; wherein the areas of the standard shape, the second standard shape, and the standard shape corresponding to the set number decrease in sequence.

[0168] Among them, the first specification shape is preferably in the COPD area of ​​each lobe and at the geometric center of the small airway lesion area of ​​each lobe. The specification shape can be selected from any one of circle, ellipse, rectangle and square. For example, if the specification shape is selected as a circle, a number of set points can be evenly set on the circumference of the circle, such as 20 set points. When several set points of the specification shape touch the edge of the COPD area of ​​the right lung or the edge of the small airway lesion area of ​​the right lung, the first specification shape stops extending and generates a second specification shape outside the specification shape; the second specification shape can also be selected from any one of circle, ellipse, rectangle and square, but the second specification shape is smaller than the first specification shape to achieve the purpose of fine division. For the setting of the value of the set number, those skilled in the art can set it by themselves according to the accuracy requirements.

[0169] In COPD and small airway lesions, based on the level of COPD, the trend level of the first disease in the corresponding lobe is determined according to the ratio of the first volume to the second volume of each lobe and a preset threshold.

[0170] For example, on the left upper lobe, based on the level of COPD, the trend level of COPD in the left upper lobe can be determined according to the ratio of the first volume of the left upper lobe to the second volume of the left upper lobe and a preset threshold.

[0171] The method for determining the trend level of the first disease of the corresponding lung lobe based on the level of the first disease according to the ratio of the first area and the second area of ​​each lung lobe to a preset threshold is: if the ratio is greater than the preset threshold, the trend level is the next level of the level of the first disease; otherwise, the trend level is the previous level of the level of the first disease; wherein the previous level is more serious than the level of the first disease.

[0172] For example, in the left upper lobe, the grade of COPD is GOLD 2 (moderate), and the ratio obtained by dividing the first volume of the COPD area in the left upper lobe by the second volume of the small airway lesion area in the left upper lobe is less than or equal to the preset threshold, indicating that although the COPD area in the left upper lobe is small, the small airway lesion area is large, and the possibility of the small airway lesion area being transformed into the COPD area is high, and the grade of COPD in the left upper lobe can be predicted to the previous grade GOLD 3 (severe). If the ratio is greater than the preset threshold, it is considered that drug treatment can be performed, and the grade of COPD in the left upper lobe can be predicted to the next grade GOLD 1 (mild). Among them, when the first disease is lung cancer, the preset threshold can be 0.4.

[0173] In addition, the embodiment of the present invention also provides a disease trend level determination device, referring to Figure 4 , the disease trend level determining device comprises:

[0174] An acquisition module 10 is used to acquire a lung lobe segmentation image;

[0175] A first determination module 20 is used to determine a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, where the second disease can be unidirectionally converted into the first disease;

[0176] The second determination module 30 is used to determine the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease.

[0177] Optionally, the second determining module 30 includes:

[0178] a positioning unit, used for positioning a first area of ​​the first disease and a second area of ​​the second disease in each lung lobe respectively;

[0179] A determination unit is used to determine the trend level of the first disease in each lung lobe based on the known level of the first disease and according to the ratio of the first area to the second area of ​​each lung lobe.

[0180] Optionally, the determining unit is specifically configured to obtain a first volume of the first disease on each lung lobe according to the first region, and to obtain a second volume of the second disease on each lung lobe according to the second region;

[0181] calculating a ratio of the first volume to the second volume for each lung lobe respectively;

[0182] Based on the level of the first disease, the trend level of the first disease in the corresponding lung lobe is determined according to the ratio of the first volume to the second volume of each lung lobe.

[0183] Optionally, the determining unit is further configured to determine, in response to a ratio of the first area to the second area being greater than a preset threshold, a next level lighter than a known level of the first disease as a trend level of the first disease;

[0184] In response to the ratio of the first area to the second area being less than or equal to a preset threshold, a previous level that is more serious than the known level of the first disease is determined as the trend level of the first disease.

[0185] Optionally, the first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is chronic obstructive pulmonary disease, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is small airway lesions; or

[0186] The first disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung cancer, and the second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image is lung nodule.

[0187] Furthermore, the first determination module is also used to input the segmented images of each lung lobe into a preset neural network to extract image features through the residual network of the preset neural network, and determine the known level of the first disease based on the classification network of the preset neural network.

[0188] Optionally, the acquisition module is specifically used to acquire lung images, which include full inhalation phase lung images and full expiratory phase lung images; and perform lung lobe segmentation on the full inhalation phase lung images and the full expiratory phase lung images, respectively, to obtain a first lung lobe segmentation image and a second lung lobe segmentation image.

[0189] Optionally, the first determination module 20 is specifically used to determine the first disease based on the first lobe segmentation image or the second lobe segmentation image; and to determine the second disease based on the first lobe segmentation image and the second lobe segmentation image.

[0190] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a disease trend level determination program is stored. When the disease trend level determination program is executed by a processor, the steps of the disease trend level determination method described below are implemented.

[0191] The various embodiments of the disease trend level determination device and the computer-readable storage medium of the present invention can all refer to the various embodiments of the disease trend level determination method of the present invention, and will not be described in detail here.

[0192] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0193] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0195] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for determining a disease trend level, It is characterized in that include: Acquire lung lobe segmentation images; Determine a first disease and a second disease in the lesion area of ​​each lobe in the lobe segmentation image, wherein the second disease can be unidirectionally transformed into the first disease; wherein the first disease is chronic obstructive pulmonary disease and the second disease is small airway lesions; or the first disease is lung cancer and the second disease is lung nodules; Locating first areas of the first disease and second areas of the second disease in each lung lobe respectively; Based on the known level of the first disease, determining the trend level of the first disease in each lung lobe according to the ratio of the first area to the second area in each lung lobe; The determining, based on the known level of the first disease and according to the ratio of the first area to the second area of ​​each lung lobe, the trend level of the first disease of each lung lobe comprises: obtaining a first volume of the first disease on each lung lobe according to the first region, and obtaining a second volume of the second disease on each lung lobe according to the second region; calculating a ratio of the first volume to the second volume for each lung lobe respectively; Based on the level of the first disease, determining the trend level of the first disease in the corresponding lung lobe according to the ratio of the first volume to the second volume of each lung lobe; Or the determining, based on the known level of the first disease, of the trend level of the first disease in each lobe of the lung according to the ratio of the first area to the second area of ​​each lobe of the lung, comprises: In response to the ratio of the first area to the second area being greater than a preset threshold, determining the next level lighter than the known level of the first disease as the trend level of the first disease; In response to the ratio of the first area to the second area being less than or equal to a preset threshold, a previous level that is more serious than the known level of the first disease is determined as the trend level of the first disease.

2. The method according to claim 1, It is characterized in that Before determining the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease, the method further includes: The segmented images of each lung lobe are input into a preset neural network to extract image features through a residual network of the preset neural network, and the known level of the first disease is determined based on a classification network of the preset neural network.

3. The method according to claim 1, It is characterized in that The step of acquiring a lung lobe segmentation image comprises: Acquiring a lung image, wherein the lung image includes a full inspiratory phase lung image and a full expiratory phase lung image; Performing lung lobe segmentation on the full inhalation phase lung image and the full exhalation phase lung image respectively to obtain a first lung lobe segmentation image and a second lung lobe segmentation image; The determining of the first disease and the second disease of the lesion area of ​​each lung lobe in the lung lobe segmentation image comprises: determining the first disease according to the first lung lobe segmentation image or the second lung lobe segmentation image; The second disease is determined according to the first lung lobe segmentation image and the second lung lobe segmentation image.

4. A device for determining a disease trend level, It is characterized in that The disease trend level determination device comprises: An acquisition module, used for acquiring a lung lobe segmentation image; A first determination module is used to determine a first disease and a second disease in the lesion area of ​​each lung lobe in the lung lobe segmentation image, wherein the second disease can be unidirectionally converted into the first disease; wherein the first disease is chronic obstructive pulmonary disease and the second disease is small airway lesions; or the first disease is lung cancer and the second disease is lung nodules; A second determination module, configured to determine the trend level of the first disease in each lung lobe based on the known level of the first disease and the second disease; The second determining module comprises: a positioning unit, used for positioning a first area of ​​the first disease and a second area of ​​the second disease in each lung lobe respectively; A determination unit is used to determine the trend level of the first disease in each lobe based on the known level of the first disease and respectively according to the ratio of the first area to the second area of ​​each lobe. The determination unit is specifically used to obtain the first volume of the first disease in each lobe based on the first area, and obtain the second volume of the second disease in each lobe based on the second area; calculate the ratio of the first volume to the second volume for each lobe; based on the level of the first disease, determine the trend level of the first disease in the corresponding lobe based on the ratio of the first volume to the second volume of each lobe; or the determination unit is specifically used to determine the next level lighter than the known level of the first disease as the trend level of the first disease in response to the ratio of the first area to the second area being greater than a preset threshold; determine the previous level more serious than the known level of the first disease as the trend level of the first disease in response to the ratio of the first area to the second area being less than or equal to the preset threshold.

5. A device for determining the level of disease trend, It is characterized in that The disease trend level determination device includes: a memory, a processor, and a disease trend level determination program stored in the memory and executable on the processor. When the disease trend level determination program is executed by the processor, the steps of the disease trend level determination method as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a disease trend level determination program, which, when executed by a processor, implements the steps of the disease trend level determination method according to any one of claims 1 to 3.

Images