A lung nodule detection and semantic attribute rating method based on multi-task learning

Abstract

The application discloses a lung nodule detection and semantic attribute rating method based on multi-task learning, which comprises two sub-tasks of lung nodule detection and semantic attribute rating. In order to realize feature sharing of joint learning between the two sub-tasks, the application connects a lung nodule detection sub-network and a semantic attribute rating sub-network together to form an end-to-end joint model. The lung nodule detection sub-network acquires position information of the nodule, and the output thereof serves as input of the semantic attribute rating sub-network. The two sub-networks share bottom layer features in a down-sampling stage of a U-Net network, so that feature sharing of the multi-task model is realized. In the process of joint learning training of the two tasks, since the training difficulty and convergence speed of different sub-tasks can be different, the application adopts a dynamic weight average method to adjust loss weights of different tasks. The method can not only effectively detect lung nodules, but also identify semantic attributes of the lung nodules as an additional supervision signal.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided diagnosis of medical images, and more specifically, relates to a method for lung nodule detection and semantic attribute rating based on multi-task learning. Background Technology

[0002] Lung cancer, characterized by its high incidence and low survival rate, is one of the leading causes of cancer death worldwide. Diagnostic methods based on pulmonary nodule imaging are effective for the early diagnosis of lung cancer. The foundation of lung cancer diagnosis lies in the detection and classification of pulmonary nodules in CT images for benign or malignant characteristics. When reviewing lung CT images, doctors primarily rely on the medical features of the nodules, analyzing semantic attributes such as the presence of spiculations and lobulations at the nodule's edges to determine their benign or malignant nature.

[0003] Existing research typically treats the detection of lung nodules or the classification of benign and malignant nodules as independent tasks, designing and training network models separately. However, designing and training two models independently can lead to redundancy in low-level feature extraction, resulting in low algorithm efficiency and affecting accuracy. Summary of the Invention

[0004] To address the shortcomings and improvement needs of existing technologies, this invention provides a lung nodule detection and semantic attribute rating method based on multi-task learning. This method aims to solve the technical problem that existing technologies treat lung nodule detection or benign / malignant classification as independent tasks, resulting in low algorithm efficiency and accuracy.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for lung nodule detection and semantic attribute rating based on multi-task learning is provided, comprising the following steps:

[0006] A lung nodule detection subnetwork and a semantic attribute rating subnetwork are constructed. The lung nodule detection subnetwork includes an encoder, a decoder, and a region proposal network. The semantic attribute rating subnetwork includes a feature extraction network and a regression network.

[0007] The preprocessed lung CT image is input into the encoder to extract the low-level features, and then the low-level features are mapped back into the input space by the decoder to obtain the first feature map.

[0008] The first feature map is input into the region proposal network to generate candidate regions for lung nodules.

[0009] The candidate lung nodule region and the preprocessed lung CT image are input into the feature extraction network to obtain a second feature map, and then the second feature map is connected with the bottom layer features.

[0010] The concatenated features are input into the regression network to obtain semantic attribute rating results.

[0011] Furthermore, with joint loss L total With the goal of minimizing the impact, the lung nodule detection subnetwork and the semantic attribute rating subnetwork are trained.

[0012] L total =λ1L1+λ2L2

[0013]

[0014]

[0015] Where L1 and L2 are the loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively, and λ1 and λ2 are the weight coefficients of the loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively; w i (t-1) represents the relative decline rate of task i in the t-th epoch of training, defined as the ratio of the loss functions corresponding to the first two training rounds, where t is the iteration exponent, i = 1, 2, T represents the looseness between tasks, and K is the total number of tasks.

[0016] Furthermore, the encoder includes two 3×3×3 convolutional layers, four residual dense structures, and four max pooling layers; the decoder includes two 2×2×2 deconvolutional layers with a stride of 2 and two residual dense structures, with two convolutional layers having a kernel of 1×1×1 and channels of 15 and 64 respectively.

[0017] Furthermore, the feature extraction network includes three convolutional layers, each followed by a max pooling layer; the regression network includes two fully connected layers, representing the final regression values ​​of the output features and semantic attributes.

[0018] Further, the step of inputting the first feature map into a region proposal network to generate candidate regions for lung nodules includes:

[0019] The first feature map is input into the region proposal network, and several anchor boxes are obtained by sliding a sliding window on the first feature map. Then, the anchor boxes are filtered and fine-tuned to generate candidate regions for lung nodules.

[0020] To achieve the above objectives, according to a second aspect of the present invention, a lung nodule detection and semantic attribute rating system based on multi-task learning is provided, comprising:

[0021] The model building module is used to build a lung nodule detection subnetwork and a semantic attribute rating subnetwork. The lung nodule detection subnetwork includes an encoder, a decoder, and a region proposal network. The semantic attribute rating subnetwork includes a feature extraction network and a regression network.

[0022] The first feature extraction module is used to input the preprocessed lung CT image into the encoder, extract the low-level features, and then map the low-level features back into the input space through the decoder to obtain the first feature map.

[0023] The candidate region generation module is used to input the first feature map into the region proposal network to generate candidate regions for lung nodules;

[0024] The second feature extraction module is used to input the candidate lung nodule region and the preprocessed lung CT image into the feature extraction network to obtain a second feature map, and then connect the second feature map with the bottom layer features;

[0025] The semantic attribute rating module is used to input the concatenated features into the regression network to obtain the semantic attribute rating result.

[0026] To achieve the above objectives, according to a third aspect of the present invention, a machine-readable storage medium is provided, the machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the lung nodule detection and semantic attribute rating method based on multi-task learning as described in the first aspect.

[0027] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0028] (1) This invention connects a lung nodule detection subnetwork and a semantic attribute rating subnetwork to form an end-to-end joint model. The lung nodule detection subnetwork acquires the location information of lung nodules, and its output serves as the input to the semantic attribute rating subnetwork. The two subnetworks share low-level features during the downsampling stage of the U-Net network, achieving feature sharing in a multi-task model. Thus, not only can lung nodules be effectively detected, but the semantic attributes of lung nodules are also identified as an additional supervisory signal, thereby simultaneously achieving correct detection of lung nodules and rating and describing the semantic attributes of candidate lung nodules.

[0029] (2) In view of the fact that the training difficulty and convergence speed of different sub-tasks may be different during the joint learning and training of two sub-tasks, in order to find appropriate weights for each sub-task, the importance of them is balanced by the multi-task loss function, and the loss weights of different tasks are adjusted by the dynamic weight averaging method to optimize the training of the shared network structure. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the lung nodule detection and semantic attribute rating method based on multi-task learning provided in this embodiment of the invention.

[0031] Figure 2This is a schematic diagram of the lung nodule detection subnetwork and semantic attribute rating subnetwork structure provided in an embodiment of the present invention.

[0032] Figure 3 These are specific application example diagrams provided in the embodiments of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0035] To achieve accurate detection and semantic attribute rating of lung nodules, sub-networks are designed for lung nodule detection and semantic attribute rating tasks, respectively. Multi-task learning is achieved by sharing the same feature extractor across different tasks. To find appropriate weights for each sub-task, a dynamic weight averaging method is used to adjust the loss weights of different tasks and optimize the training of the shared network structure, balancing their importance through a multi-task loss function. This invention not only effectively detects lung nodules but also simultaneously uses the semantic attributes of lung nodules as supervisory signals for identification, rating and describing the semantic attributes of candidate lung nodules.

[0036] See Figure 1 , combined Figure 2 and 3 The present invention provides a method for lung nodule detection and semantic attribute rating based on multi-task learning, including operations S1 to S5.

[0037] Operation S1 constructs a lung nodule detection subnetwork and a semantic attribute rating subnetwork. The lung nodule detection subnetwork includes an encoder, a decoder, and a region proposal network. The semantic attribute rating subnetwork includes a feature extraction network and a regression network.

[0038] The first subtask, lung nodule detection, aims to identify lung nodules from CT images with the highest possible recall, while reducing false positives to lower the false detection rate. This embodiment designs a lung nodule detection subnetwork based on the 3D Faster R-CNN framework, introducing a U-Net backbone network with a dense residual structure as the main framework for feature extraction. Candidate regions are directly generated through the RPN to complete the tasks of lung nodule localization and reducing false positives.

[0039] The second subtask, semantic attribute rating, involves scoring the semantic attributes of candidate nodules obtained from the lung nodule detection subnetwork. This embodiment uses a multi-task CNN-based semantic attribute rating subnetwork for lung nodules, including feature extraction and semantic attribute regression.

[0040] Operation S2 inputs the preprocessed lung CT image into the encoder to extract the low-level features, and then the decoder maps the low-level features back into the input space to obtain the first feature map.

[0041] In this embodiment, preprocessed lung CT images are first input into the network model for training, with an improved U-Net backbone network serving as the main framework for feature extraction. During training, feature representations of lung nodules are automatically learned and feature maps are generated through feature extraction.

[0042] In the lung nodule detection subnetwork, the encoder in U-Net starts with two 3×3×3 convolutional layers, each with 24 channels; followed by four residual dense structures interleaved with four max pooling layers (pooling size of 2×2×2, stride of 2).

[0043] In the decoder, the feature map is first processed through a deconvolutional layer and a residual dense network structure, and then cascaded with the corresponding layers in the encoder network. Finally, two convolutional layers with kernels of 1×1×1 and channels of 15 and 64 respectively convert the image data size to 32×32×32×3×5.

[0044] Operation S3 involves inputting the first feature map into the region proposal network to generate candidate regions for lung nodules.

[0045] In this embodiment, the feature extraction result (i.e., the first feature map) is fed into the back-end detection network of the Region Proposal Network (RPN). In the RPN, using the first feature map as input, a sliding window is used to slide across the first feature map to obtain several anchor boxes. In the last layer, based on the scale distribution of lung nodules, three anchor boxes of sizes 5, 10, and 20 are designed. The (o,x,y,z,d) values ​​at each position represent the confidence level, three-dimensional coordinates, and diameter of the lung nodule, and the sigmoid activation function is used.

[0046] By integrating the output of the decoder's upsampled feature map with the encoder's feature map of the same resolution through skip connections, and then inputting it into the next submodule of the decoder to combine low-level and high-level features, the output of the feature map is enhanced and optimized.

[0047] After operations such as reducing false positives, anchor box bounding box regression, and semantic attribute rating, candidate regions for lung nodules with semantic attribute scores are obtained. Using the first feature map and the candidate regions for lung nodules as input, the positions of the candidate regions are fine-tuned through region of interest pooling to complete the lung nodule detection subtask.

[0048] During lung nodule detection, the model generates a set of candidate regions. IoU is used to compare the overlap between these candidate regions and the ground truth regions. If an anchor box overlaps with a Ground Truth bounding box and the IoU is greater than 0.5, it is considered a positive sample (p=1); if the IoU between an anchor box and all ground truth bounding boxes (the actual location of the lung nodule) is less than 0.02, it is considered a negative sample (p=0).

[0049] For each anchor box participating in training, the loss function includes a classification loss L. cls and position loss L reg Classification loss L cls Indicates whether the current bounding box contains a nodule; position loss L reg This represents the loss in terms of nodule coordinates (x, y, z) and nodule diameter (d). For L... cls Using the binary cross-entropy loss function, for L reg Using the Smooth L1 regression loss function, the L1 loss of the lung nodule detection subnetwork is expressed as:

[0050] L1 = L cls +pL reg

[0051] Where p is the label of the anchor box, p∈{0,1} (0 for negative samples, 1 for positive samples), and the position loss L reg Only applicable to positive samples.

[0052] In operation S4, the candidate lung nodule region and the preprocessed lung CT image are input into the feature extraction network to obtain a second feature map, and then the second feature map is connected with the bottom layer features.

[0053] In this embodiment, the anchor box generated by the lung nodule detection subnetwork is used as input, and the lung nodule image is cropped with the center of the anchor box as the detection center. The cropped image with a size of 64×64×64 is used as the input of the CNN.

[0054] In the feature extraction network, the designed CNN contains three convolutional layers, each followed by a max-pooling layer to improve translation tolerance. The regression network contains two fully connected layers, representing 32-dimensional output features and the final regression values ​​for 9 semantic attributes.

[0055] In the model, neurons in all convolutional and fully connected layers use the stochastic ReLU activation function to reduce overfitting. From the input image patch of the lung nodule to the output feature layer, the size of the feature map continuously decreases, removing potentially redundant information from the original nodule image patch and obtaining discriminative features from the fully connected layer for assessing the malignancy of the nodules.

[0056] Operation S5 inputs the concatenated features into the regression network to obtain semantic attribute rating results.

[0057] In this embodiment, the regression network first uses a single fully connected layer to explicitly extract feature representations of semantic attributes. Then, another fully connected layer is added to generate the final semantic attribute ratings. In this network, the role of the fully connected layer is to transfer the features extracted by the CNN into the final attribute labels.

[0058] The semantic attribute ratings of lung nodules, as labeled by doctors, are used to determine the average score of each nodule across nine semantic attributes. Each semantic attribute rating task has its own loss function. The weighted sum of the loss functions of these sub-tasks yields the loss function of the semantic attribute rating sub-network.

[0059]

[0060] Among them, L j (X,Y j ) is based on X as input and Y as input. j Let λ be the loss function for the j-th semantic attribute rating task. j These are the weighting coefficients.

[0061] The loss for regression of each semantic attribute is the output score y. j and tags The error between the model's output score and the average score annotated by physicians is considered. The final regression loss of the multi-task CNN is a weighted sum of nine losses. The mean squared error between the model's output score and the average score annotated by physicians is used as the loss function L for nodule semantic attribute rating. i :

[0062]

[0063] Where N is the total number of lung nodules in the training set. It is a semantic attribute tag, y j It is the model's score for the semantic attributes of lung nodules.

[0064] Furthermore, in the joint learning of lung nodule detection and semantic attribute rating, the lung nodule detection subnetwork and the semantic attribute rating subnetwork are connected together to form an end-to-end joint model.

[0065] The lung nodule detection subnetwork uses an improved U-Net backbone network to obtain nodule location information. The two subnetworks share low-level features during the downsampling stage of the U-Net network, realizing feature sharing in the multi-task model, thereby improving the efficiency and accuracy of the model. The semantic attribute rating subnetwork can also extract high-level semantic features.

[0066] The multi-task network architecture uses the output of the detection subnetwork as the input of the semantic attribute rating subnetwork, trains the model using semantic attribute labels, and obtains the semantic attribute score of the candidate lung nodule through each regression branch in the semantic attribute rating subnetwork.

[0067] A multi-task loss function was constructed based on dynamic weight averaging, and the loss functions of the lung nodule detection network and the semantic attribute rating network were optimized for end-to-end training of the model.

[0068] Step ①: The loss function of the multi-task network model is the joint loss of the lung nodule detection subnetwork (L1) and the semantic attribute rating subnetwork (L2).

[0069]

[0070] Where λ1 and λ2 are weight coefficients, and L1 is the class loss L cls and position loss L reg The L2 loss consists of the losses from nine semantic attribute regression tasks. If the current bounding box does not contain a lung nodule (i.e., a negative sample), the loss is only the class loss L. cls At this time, the position of the lung nodule is lost by L reg The loss L2 for both the semantic attribute rating task and the semantic attribute rating task is 0.

[0071] Step 2: During network model training, calculate the loss ratio for each subtask corresponding to the previous epoch, then divide it by a fixed value T for exponential mapping, and calculate the weight of each loss. Define the relative decline rate w of task i in the t-th epoch of training. i (t-1) represents the ratio of the loss functions for the first two training rounds:

[0072]

[0073] Step 3: Calculate the weight λ of task i. i :

[0074]

[0075] Among them, w i λ represents the relative rate of decline, t is the iteration exponent, and T represents the looseness between tasks. When T is sufficiently large, λ i≈1 and the weights of the tasks are equal. Multiplying the softmax operator by K ensures that ∑ i λ i (t) = K, where K is the total number of tasks. Multiplying by K ensures that the sum of the weights of each task is K, thus ensuring that the weights are scaled on the same dimension.

[0076] Step 4: After obtaining the loss weights for each subtask, the multi-task network structure can be optimized and trained based on the overall loss function. The dynamic weight averaging loss function weight calculation optimization method reduces the weights for tasks where the loss decreases rapidly, and vice versa. This method ensures that the loss of multiple subtasks decreases synchronously. Backpropagation is then used to adjust the feature extraction network to obtain features more relevant to the task.

[0077] In summary, for preprocessed lung CT images, a shared feature extraction module is used to extract features from the input image. These features are then connected to a semantic attribute rating subnetwork to rate multiple semantic attributes of the nodules. For any input lung CT image, a well-trained multi-task learning model can predict the location and confidence information of lung nodules, as well as the semantic attribute rating of candidate lung nodules. In multi-task learning, different tasks share the same feature extractor, and multiple related tasks share some or all parameters in the neural network to achieve model parameter reuse, thereby improving the efficiency and accuracy of model training and decision-making.

[0078] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for lung nodule detection and semantic attribute rating based on multi-task learning, characterized in that, The method comprises the following steps: constructing a lung nodule detection subnetwork and a semantic attribute rating subnetwork, the lung nodule detection subnetwork comprising an encoder, a decoder and a region proposal network, and the semantic attribute rating subnetwork comprising a feature extraction network and a regression network; inputting a preprocessed lung CT image into the encoder to extract a bottom-level feature, and then mapping the bottom-level feature back to an input space through the decoder to obtain a first feature map; inputting the first feature map into the region proposal network to generate a lung nodule candidate region; inputting the lung nodule candidate region and the preprocessed lung CT image into the feature extraction network to obtain a second feature map, and then connecting the second feature map with the bottom-level feature; inputting the connected feature into the regression network to obtain a semantic attribute rating result; The lung nodule detection sub-network and the semantic attribute rating sub-network are trained by the following manner: training the lung nodule detection sub-network and the semantic attribute rating sub-network in a joint loss The lung nodule detection sub-network and the semantic attribute rating sub-network are trained by the following manner: training the lung nodule detection sub-network and the semantic attribute rating sub-network in a joint loss wherein, and are loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively, and are weight coefficients of the loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively; denotes the task The relative drop rate in the first epoch training is defined as the ratio of the corresponding loss functions in the first two rounds of training, is the iteration index, , denotes the looseness between tasks, is the total number of tasks.

2. The method of claim 1, wherein, the encoder comprises two 3x3x3 convolution layers, four staggered residual dense structures and a maximum pooling layer; the decoder comprises two 2x2x2 deconvolution layers with a step of 2 and two residual dense structures, two convolution layers with a convolution kernel of 1x1x1 and a channel number of 15 and 64.

3. The method of claim 1, wherein, the feature extraction network comprises three convolution layers, each of which is connected with a maximum pooling layer; the regression network comprises two fully connected layers, which represent the final regression values of the output features and semantic attributes.

4. The method of claim 1, wherein, the step of inputting the first feature map into the region proposal network to generate a lung nodule candidate region comprises: inputting the first feature map into the region proposal network, performing sliding window operation on the first feature map to obtain a plurality of anchor boxes, and then screening and fine-tuning the plurality of anchor boxes to generate a lung nodule candidate region. 5.A system for lung nodule detection and semantic attribute rating based on multi-task learning, characterized in that, comprise: a model construction module configured to construct a lung nodule detection subnetwork and a semantic attribute rating subnetwork, the lung nodule detection subnetwork comprising an encoder, a decoder and a region proposal network, and the semantic attribute rating subnetwork comprising a feature extraction network and a regression network; a first feature extraction module configured to input a preprocessed lung CT image into the encoder to extract a bottom-level feature, and then map the bottom-level feature back to an input space through the decoder to obtain a first feature map; a candidate region generation module configured to input the first feature map into the region proposal network to generate a lung nodule candidate region; a second feature extraction module configured to input the lung nodule candidate region and the preprocessed lung CT image into the feature extraction network to obtain a second feature map, and then connect the second feature map with the bottom-level feature; a semantic attribute rating module configured to input the connected feature into the regression network to obtain a semantic attribute rating result. The lung nodule detection sub-network and the semantic attribute rating sub-network are trained by the following manner: training the lung nodule detection sub-network and the semantic attribute rating sub-network in a joint loss The lung nodule detection sub-network and the semantic attribute rating sub-network are trained by the following manner: training the lung nodule detection sub-network and the semantic attribute rating sub-network in a joint loss wherein, and are loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively, and are weight coefficients of the loss functions of the lung nodule detection subnetwork and the semantic attribute rating subnetwork, respectively; denotes the task The relative drop rate in the first epoch training is defined as the ratio of the corresponding loss functions in the first two rounds of training, is the iteration index, , denotes the looseness between tasks, is the total number of tasks.

6. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by the processor, the machine-executable instructions cause the processor to implement the lung nodule detection and semantic attribute rating method based on multi-task learning according to any one of claims 1 to 4.

Images