Pulmonary nodule recognition model training method and equipment, pulmonary nodule recognition equipment

Abstract

The present invention provides a lung nodule recognition model training method and device, and a lung nodule recognition device. The method is applied to the field of image processing. The method comprises: using a lung nodule recognition model to recognize a lung image, the lung nodule recognition model comprises a backbone network module, a dual-feature pyramid module and an output head module, the backbone network module is used to extract a multi-level feature map from an input lung image, the dual-feature pyramid module is used to fuse the multi-level feature maps, the output head module is used to classify and regress the multi-scale feature map generated by the dual-feature pyramid module to obtain classification results and regression results, and calculate the loss function to obtain classification loss and regression loss; weighted sum the classification loss and regression loss to obtain the total loss, and use the back propagation algorithm to calculate the gradient of the total loss; and update the network parameters of the lung nodule recognition model according to the gradient. The present invention improves the detection and positioning accuracy of lung nodules.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a lung nodule recognition model training method and equipment, and a lung nodule recognition equipment. Background Art

[0002] With the development of the information age, the application of deep learning is becoming more and more extensive, especially in the automatic auxiliary detection of lung nodules. In recent years, computer-aided diagnosis systems based on convolutional neural networks (CNNs) have shown strong capabilities to autonomously learn the advanced features of nodules, help doctors make clinical diagnoses, and reduce reliance on manual screening.

[0003] Although deep learning methods have performed well in CT image analysis, developing robust nodule detection models remains a challenge due to the morphological heterogeneity of nodules and the complexity of their surroundings. Although many deep learning-based algorithms have made great progress in improving the accuracy of nodule detection, the high false positive rate remains a challenging problem, which limits automatic diagnosis in routine clinical practice. Summary of the invention

[0004] In view of this, the present invention provides a lung nodule recognition model training method, comprising:

[0005] A lung image is identified using a lung nodule recognition model, wherein the lung nodule recognition model includes a backbone network module, a dual-feature pyramid module, and an output head module, wherein the backbone network module is used to extract a multi-level feature map from an input lung image, the dual-feature pyramid module is used to fuse the multi-level feature maps to enhance the multi-scale representation of features, and the output head module is used to classify and regress the multi-scale feature maps generated by the dual-feature pyramid module to obtain a classification result and a regression result, wherein the classification result indicates the probability of a nodule predicted by the lung nodule recognition model, and the regression result indicates the location information of the nodule predicted by the lung nodule recognition model;

[0006] Calculate the loss function according to the classification result and the regression result to obtain the classification loss and the regression loss;

[0007] Perform a weighted summation of the classification loss and the regression loss to obtain a total loss, and calculate the gradient of the total loss using a back propagation algorithm;

[0008] The network parameters of the pulmonary nodule recognition model are updated according to the gradient.

[0009] Optionally, the backbone network module includes: an initial convolution layer, a second multi-scale convolution layer, a third multi-scale convolution layer, a fourth multi-scale convolution layer, and a fifth multi-scale convolution layer. The initial convolution layer is used to extract preliminary features from the input lung image, and the second multi-scale convolution layer, the third multi-scale convolution layer, the fourth multi-scale convolution layer and the fifth multi-scale convolution layer are used to perform convolution operations of different scales on the preliminary features to gradually extract and enhance intermediate to high-level feature information in the image.

[0010] Optionally, the second multi-scale convolution layer, the third multi-scale convolution layer, the fourth multi-scale convolution layer and the fifth multi-scale convolution layer respectively include a multi-scale bottleneck unit, and the multi-scale bottleneck unit includes: a convolution branch, a multi-scale convolution branch, a feature splicing layer, a dimension adjustment convolution layer, a residual connection layer, and a non-linear activation layer. The convolution branch is used to extract feature information of a standard scale from the input feature, and the multi-scale convolution branch is used to extract and fuse multi-scale feature information from the input feature. The feature splicing layer is used to splice the features of the convolution branch and the multi-scale convolution branch. The dimension adjustment convolution layer is used to adjust the dimension of the spliced ​​features. The residual connection layer is used to add the dimension adjusted features to the input features to form a residual connection. The non-linear activation layer is used to convert the feature map after the residual connection into a non-linear feature map using a non-linear activation function.

[0011] Optionally, the multi-scale convolution branch includes a multi-scale self-calibration module, which includes a segmentation unit, a self-calibration processing unit and a splicing unit. The segmentation unit is used to divide the input feature map into four feature maps on average to obtain feature maps. , feature map , feature map , feature map The self-calibration processing unit includes three self-calibration branches of different scales and a standard convolution branch. The three self-calibration branches of different scales are used to calibrate the feature map , feature map and feature map Perform self-calibration operation to obtain the characteristic map , feature map , feature map , the standard convolution branch is used to map the feature map conduct Convolution operation to obtain feature map , the concatenation unit is used for feature map , feature map , feature map and feature map Splice to get the feature map .

[0012] Optionally, the self-calibration branch includes a feature calibration module and a feature fusion module, wherein the feature calibration module is used to calibrate the input feature map. Perform R times downsampling and then After the convolution operation, upsample by R times to restore the feature map size, and compare the restored feature map with the input feature map The result after addition is used to calculate the activation function and generate feature weights;

[0013] The feature fusion module is used to input feature maps conduct Convolution operation is used to extract preliminary features and then The convolution result is multiplied by the feature weight to perform weighting, and the weighted feature map is processed. Convolution operation is performed to obtain the output feature map.

[0014] Optionally, the dual-feature pyramid module includes a semantic feature propagation unit and a feature positioning enhancement unit. The semantic feature propagation unit is used to propagate high-level semantic information from top to bottom according to the multi-level feature map, and transfer rich semantic features to all feature levels layer by layer to obtain a multi-scale semantic feature map. The feature positioning enhancement unit is used to propagate the bottom-level precise positioning information layer by layer from bottom to top according to the multi-scale semantic feature map to obtain a multi-scale positioning enhanced feature map.

[0015] Optionally, the dual feature pyramid module also includes a balanced feature fusion module, which is used to balance and fuse multi-scale positioning enhancement feature maps, integrate feature information at different levels, and improve the model's processing capabilities for multi-scale features, wherein the balanced feature fusion module includes a first linear interpolation unit, a feature aggregation unit, and a second linear interpolation unit, the first linear interpolation unit is used to adjust the resolution of the multi-scale positioning enhancement feature map, the feature aggregation unit is used to aggregate the feature map processed by the first linear interpolation unit, and the second linear interpolation unit is used to restore the aggregated feature map to the resolution of the original positioning enhancement feature map.

[0016] Optionally, the output head module includes a classification head and a regression head, the classification head is used to perform a classification operation on the multi-scale feature map to obtain a probability distribution of each candidate box on each feature point, and the regression head is used to perform a regression operation on the multi-scale feature map to obtain position information of each candidate box on each feature point.

[0017] Optionally, a loss function is calculated according to the classification result and the regression result to obtain a classification loss and a regression loss, including:

[0018] Use the adaptive training sample selection method to perform positive and negative sample matching on all generated candidate boxes to obtain the positive sample set and negative sample set of the candidate boxes;

[0019] Performing difficult negative sample sampling according to the classification results, the positive sample set and the negative sample set, and adjusting the ratio of the positive sample set to the negative sample set within a preset range;

[0020] The cross entropy loss function is calculated on the data after sampling the difficult negative samples to obtain the classification loss function, and the weighted intersection-over-union loss function is calculated on the data after sampling the difficult negative samples to obtain the regression loss function.

[0021] Optionally, an adaptive training sample selection method is used to perform positive and negative sample matching on all generated candidate boxes to obtain a positive sample set and a negative sample set of the candidate boxes, including:

[0022] Calculate the distance between the center point of each candidate box and the center point of the current pre-labeled box, select the K candidate boxes closest to each level of the candidate box as candidate positive samples, and form a set of all candidate positive samples in the current pre-labeled box ;

[0023] Calculate the intersection-over-union ratio of each candidate positive sample and the pre-labeled box, calculate the mean and standard deviation of the intersection-over-union ratio, and set the threshold of the intersection-over-union ratio based on the mean and standard deviation;

[0024] If the intersection-over-union ratio of the candidate positive sample is greater than or equal to the threshold, it is marked as a positive sample, and the set of positive samples of all pre-annotated boxes is counted as , the remaining candidate boxes are counted as negative sample sets .

[0025] The second aspect of the present invention provides a lung nodule recognition model training device, which includes: a processor and a memory connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor executes the above-mentioned lung nodule recognition model training method.

[0026] A third aspect of the present invention provides a lung nodule identification device, which includes: a processor and a memory connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor uses the lung nodule recognition model trained by the method described to identify lung images and obtain classification results and regression results, wherein the classification result represents the probability of nodules predicted by the lung nodule recognition model, and the regression result represents the location information of nodules predicted by the lung nodule recognition model.

[0027] The present invention trains a lung nodule recognition model by using a backbone network module, a dual-feature pyramid module and an output head module. The backbone network module extracts multi-level features of lung images, the dual-feature pyramid module fuses features to enhance multi-scale representation, and the output head module performs classification and regression prediction. Then, the classification loss and regression calculation are calculated according to the output results and gradient updates are performed to optimize the network parameters of the lung nodule recognition model, thereby improving the model's detection and positioning accuracy of lung nodules and effectively reducing the false positive rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is a flow chart of a lung nodule recognition model training method in an embodiment of the present invention;

[0030] Figure 2 is a network structure diagram of a lung nodule recognition model in an embodiment of the present invention;

[0031] Figure 3 is a specific network structure diagram of the lung nodule recognition model in an embodiment of the present invention;

[0032] Figure 4 A network structure diagram of a backbone network module in an embodiment of the present invention;

[0033] Figure 5 A network structure diagram of a multi-scale bottleneck unit in an embodiment of the present invention;

[0034] Figure 6 is a network structure diagram of a self-calibration branch in an embodiment of the present invention;

[0035] Figure 7 FIG. 4 is another specific network structure diagram of the pulmonary nodule recognition model in an embodiment of the present invention;

[0036] Figure 8 is a network structure diagram of a classification head and a regression head in an embodiment of the present invention;

[0037] Fig. 9 A comparison diagram of the pre-marked box and the candidate box in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0040] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0041] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] like Figure 1 As shown, an embodiment of the present invention provides a lung nodule recognition model training method, which is executed by an electronic device such as a computer or a server, and specifically includes:

[0043] S1. Lung images are identified using a lung nodule recognition model. The lung nodule recognition model includes a backbone network module, a dual-feature pyramid module, and an output head module. The backbone network module is used to extract multi-level feature maps from the input lung images. The dual-feature pyramid module is used to fuse the multi-level feature maps to enhance the multi-scale representation of features. The output head module is used to classify and regress the multi-scale feature maps generated by the dual-feature pyramid module to obtain classification results and regression results. The classification result represents the probability of nodules predicted by the lung nodule recognition model, and the regression result represents the location information of nodules predicted by the lung nodule recognition model.

[0044] The lung image is a lung CT image. A certain lung CT image is preprocessed, specifically, the lung CT image is statistically analyzed to obtain statistical information, including spacing, mean, and standard deviation, and resampling and standardization operations are performed based on the statistical information.

[0045] Figure 2 The network structure of the lung nodule recognition model is as follows: the preprocessed lung CT image is input into the backbone network module 1 (Backbone) for multi-level feature map extraction, and then input into the dual feature pyramid module 2 (DualFeaturePyramid) for feature fusion, and finally input into the output head module 3 (Head) for probability and position prediction.

[0046] S2, calculate the loss function based on the classification results and regression results to obtain the classification loss and regression loss. Calculate the gap between the classification results and regression results and the true value.

[0047] S3, the classification loss and regression loss are weighted summed to obtain the total loss, and the gradient of the total loss is calculated using the back-propagation algorithm. Starting from the output layer, the gradient of the total loss is back-propagated to the input layer, and the gradient of each parameter is calculated layer by layer.

[0048] S4, update the network parameters of the lung nodule recognition model according to the gradient.

[0049] This embodiment trains the lung nodule recognition model by using the backbone network module, the dual-feature pyramid module and the output head module. The backbone network module extracts multi-level features of the lung image, the dual-feature pyramid module fuses features to enhance multi-scale representation, and the output head module performs classification and regression prediction. Then, the classification loss and regression calculation are calculated according to the output results and gradient updates are performed to optimize the network parameters of the lung nodule recognition model, thereby improving the model's detection and positioning accuracy of lung nodules and effectively reducing the false positive rate.

[0050] like Figure 3 As shown, the backbone network module 1 includes: an initial convolution layer 11, a second multi-scale convolution layer 12, a third multi-scale convolution layer 13, a fourth multi-scale convolution layer 14, and a fifth multi-scale convolution layer 15. The initial convolution layer 11 is used to extract preliminary features from the input lung image, and the second multi-scale convolution layer 12, the third multi-scale convolution layer 13, the fourth multi-scale convolution layer 14 and the fifth multi-scale convolution layer 15 are used to perform convolution operations of different scales on the preliminary features, and gradually extract and enhance the intermediate to high-level feature information in the image;

[0051] The initial convolution layer 11 consists of a convolution operation, batch normalization and a ReLu activation function. The second multi-scale convolution layer 12 consists of three multi-scale bottleneck units. The third multi-scale convolution layer 13, the fourth multi-scale convolution layer 14 and the fifth multi-scale convolution layer 15 are respectively composed of multiple multi-scale bottleneck units. The number of multi-scale units in each layer can be determined according to the complexity of the task and the hardware resources.

[0052] Specifically, the initial convolution layer 11 is a common CBA structure, and the second multi-scale convolution layer 12, the third multi-scale convolution layer 13, the fourth multi-scale convolution layer 14, and the fifth multi-scale convolution layer 15 are composed of different numbers of MSC-BottleNecks, which are 3, 4, 6, and 3 respectively.

[0053] The initial convolution layer 11 is a standard convolution operation, which is responsible for extracting preliminary features and obtaining a feature map C1; the second multi-scale convolution layer 12, the third multi-scale convolution layer 13, the fourth multi-scale convolution layer 14 and the fifth multi-scale convolution layer 15 gradually extract and enhance the intermediate to high-level feature information in the image, and obtain feature maps C2, C3, C4 and C5 respectively.

[0054] The backbone network module of this embodiment gradually transforms the preliminary features in the lung image into more complex high-level features by extracting and enhancing features of the input image layer by layer. The multi-scale convolution operation and bottleneck unit design of each layer help to improve the feature extraction capability, thereby more accurately identifying and analyzing important information in the image.

[0055] Furthermore, the multi-scale bottleneck unit includes: a convolution branch, a multi-scale convolution branch, a feature splicing layer, a dimension adjustment convolution layer, a residual connection layer, and a non-linear activation layer. The convolution branch is used to extract feature information of a standard scale from the input feature, the multi-scale convolution branch is used to extract and fuse multi-scale feature information from the input feature, the feature splicing layer is used to splice the features of the convolution branch and the multi-scale convolution branch, the dimension adjustment convolution layer is used to adjust the dimension of the spliced ​​features, the residual connection layer is used to add the dimension adjusted features to the input features to form a residual connection, and the non-linear activation layer is used to convert the feature map after the residual connection into a non-linear feature map using a non-linear activation function.

[0056] like Figure 4 As shown in the figure, the multi-scale bottleneck unit (MSC-BottleNeck) is improved on the basis of Resnet's BottleNeck. MSC-BottleNeck adds a branch on the basis of Native BottleNeck, in which the convolution operation is replaced by the MSC operation (multi-scale self-calibration), and the convolution branch ( Figure 4 Left branch) and multi-scale convolution branch ( Figure 4The right branch), then the two branches are concat together through the feature splicing layer, and the dimension is changed by 1*1*1 convolution through the dimension adjustment convolution layer. The result after dimension change and the input feature are added together to form a residual structure, and then activated by the nonlinear activation function ReLU of the nonlinear activation layer to get the output.

[0057] Specifically, the convolution branch first performs a 1*1*1 convolution operation on the input features and generates a feature map of 64 output channels, then performs batch normalization and ReLU activation on the feature map, then performs a 3*3*3 convolution operation and generates a feature map of 64 output channels, and finally performs ReLU activation to obtain the output of the convolution branch; the multi-scale convolution branch first performs a 1*1*1 convolution operation on the input features and generates a feature map of 64 output channels, then performs batch normalization and ReLU activation on the feature map, then performs an MSC operation, and finally performs ReLU activation to obtain the output of the multi-scale convolution branch; then the output of the convolution branch is concat- ed with the output of the multi-scale convolution branch through a feature concatenation layer, and a 1*1*1 convolution operation is performed through a dimension adjustment convolution layer to generate a feature map of 256 output channels, the result after the dimension change is added to the initial input feature to form a residual structure, and then passes through the nonlinear activation function of the nonlinear activation layer to obtain the final output.

[0058] The multi-scale bottleneck unit in this embodiment improves the traditional bottleneck unit by combining standard convolution and multi-scale convolution operations, introducing mechanisms such as feature splicing, dimension adjustment, residual connection and nonlinear activation. The network designed in this way can not only extract richer features, but also improve the expression ability and training efficiency of the model, which has significant advantages in processing complex image data.

[0059] Furthermore, the multi-scale convolution branch includes a multi-scale self-calibration module, which includes a segmentation unit, a self-calibration processing unit and a splicing unit. The segmentation unit is used to divide the input feature map into four feature maps on average to obtain the feature map , feature map , feature map , feature map The self-calibration processing unit includes three self-calibration branches of different scales and one standard convolution branch. The three self-calibration branches of different scales are used to calibrate the feature maps. , feature map and feature map Perform self-calibration operation to obtain the characteristic map , feature map , feature map , the standard convolution branch is used to map the feature map Perform convolution operation to obtain feature map , the concatenation unit is used for feature maps , feature map , feature map and feature map Splice to get the feature map .

[0060] like Figure 5 As shown in the figure, the multi-scale self-calibration module (MSC) includes a segmentation unit, a self-calibration processing unit and a splicing unit. The segmentation unit consists of four parts. The feature map X is evenly divided into four feature maps according to the dimension, and the feature map is obtained. , feature map , feature map , feature map , whose sizes are ; Then they are input into the four branches of the self-calibration processing unit, the feature map After four-fold double self-calibration operation (2-self-calibration), the feature map is obtained , feature map After a four-fold self-calibration operation (4-self-calibration), the feature map is obtained , feature map After eight self-calibration operations (8-self-calibration), the feature map is obtained , feature map will pass Convolution operation to obtain feature map , feature map , feature map , feature map , feature map The size is ; Finally, the feature map , feature map , feature map , feature map , feature map The input concatenation units are concatenated together according to the dimension to obtain the feature map .

[0061] The multi-scale self-calibration module of this embodiment improves the richness of feature representation and the expressiveness of the model by processing and fusing feature maps from different scales. Through self-calibration operations and splicing strategies, this module can effectively capture and combine multi-scale information and optimize the quality of the final feature map, thereby improving the performance of the network in the task.

[0062] Furthermore, the self-calibration branch includes a feature calibration module and a feature fusion module. The feature calibration module is used to calibrate the input feature map. Perform R times downsampling and then After the convolution operation, upsample by R times to restore the feature map size, and compare the restored feature map with the input feature map The result after addition is used to calculate the activation function and generate feature weights;

[0063] The feature fusion module is used to input feature maps conduct Convolution operation is used to extract preliminary features and then The convolution result is multiplied by the feature weight to perform weighting, and the weighted feature map is processed. Convolution operation is performed to obtain the output feature map.

[0064] like Figure 6 As shown, the self-calibration branch (R-self-calibration) consists of two branches, namely the feature calibration module ( Figure 6 A branch located on the upper side of the module) and feature fusion module ( Figure 6 In order to obtain a larger receptive field, the feature calibration module performs a First, downsample by R times, and then go through a Convolution operation, then restore the size of the feature map by upsampling R times, and then compare the restored feature map with the input feature map The result of the addition is subjected to the activation function Sigmoid operation to obtain the feature weight; the feature fusion module first performs a Sigmoid operation on the feature map. Convolution operation, and multiply the convolution result with the feature weight generated by the feature calibration module, and perform it again The convolution operation obtains the output of the self-calibration branch. Thanks to this heterogeneous convolution and communication between convolution kernels, the receptive field of each spatial position can be effectively expanded.

[0065] The convolution kernel sizes used in different branches of the multi-scale self-calibration module (MSC) vary according to the different downsampling rates in each branch. The smaller the downsampling rate, the larger the convolution kernel required to obtain a larger receptive field; conversely, the larger the downsampling rate, the smaller the convolution kernel can be used to obtain a sufficiently large receptive field. R is the downsampling rate of the R-self-calibration branch. If the branch does not contain downsampling operations, R is set to 1. Convolution operations in each branch of MSC The corresponding convolution kernel The calculation formula is as follows:

[0066] ,

[0067] The receptive field of traditional convolution is controlled by the preset convolution kernel. A large receptive field requires the design of a larger convolution kernel, which increases the training parameters and prevents the establishment of long-range spatial and inter-channel dependencies. MSC uses a novel multi-scale self-calibration operation to enable the module to create inter-channel and long-range spatial dependencies around each spatial position, while extracting multi-scale information from the feature map, expanding the receptive field while generating more discriminative feature expressions and improving detection sensitivity.

[0068] like Figure 2 As shown, the dual feature pyramid module 2 includes a semantic feature propagation unit 21 and a feature positioning enhancement unit 22. The semantic feature propagation unit 21 is used to propagate high-level semantic information from top to bottom according to the multi-level feature map, and transfer the rich semantic features to all feature levels layer by layer to obtain a multi-scale semantic feature map. The feature positioning enhancement unit 22 is used to propagate the bottom-level precise positioning information layer by layer from bottom to top according to the multi-scale semantic feature map to obtain a multi-scale positioning enhanced feature map.

[0069] Specifically, the semantic feature propagation unit 21 is a top-down path, and the specific network structure is FPN, and finally the feature maps P2, P3, P4, P5, and P6 are obtained, among which P6 is the final output feature map C5 of the backbone network module generated by convolution with kernel_size=3, stride=2, and P2, P3, P4, and P5 are the output feature maps C2, C3, C4, and C5 of the backbone network module, which are respectively convolved with kernel_size=1, stride=1 and then aggregated with the upsampled feature maps.

[0070] The feature location enhancement unit 22 is a bottom-top path (a bottom-up path), which eventually generates feature maps N2, N3, N4, N5, and N6. The specific process is as follows:

[0071] in,

[0072] in, It is a downsampling operation, which can be achieved by convolution with kernel_size=3, stride=2; Aggregate different inputs together along the dimension direction; Specifically refers to the convolution with kernel_size=3, stride=1, in order to eliminate the aliasing effect caused by the aggregation of different feature maps.

[0073] In a neural network, high layers contain richer semantic information, and low layers contain richer detail information. Therefore, it is necessary to propagate feature information with strong semantics to all feature levels through a Top-down path, thereby enhancing the semantic classification capabilities of all features. This is also what is done in FPN, but the positioning information is not transmitted. In view of this, the present invention adds a Bottom-top path after FPN. Through the bottom-up path, the precise positioning information of the bottom layer is fully utilized, and the positioning information is propagated layer by layer to all feature layers, thereby enhancing the positioning capability of the network. The present invention enhances the expression capability of the backbone network by fusing the top-down and bottom-up paths.

[0074] An alternative, such as Figure 7 As shown, the dual feature pyramid module 2 includes a semantic feature propagation unit 21, a feature positioning enhancement unit 22 and a balanced feature fusion module 23. The balanced feature fusion module 23 is used to balance and fuse multi-scale positioning enhancement feature maps, integrate feature information at different levels, so as to improve the model's processing capability for multi-scale features. The balanced feature fusion module 23 includes a first linear interpolation unit 231, a feature aggregation unit 232 and a second linear interpolation unit 233. The first linear interpolation unit is used to adjust the resolution of the multi-scale positioning enhancement feature map, the feature aggregation unit is used to aggregate the feature map processed by the first linear interpolation unit, and the second linear interpolation unit is used to restore the aggregated feature map to the resolution of the original positioning enhancement feature map.

[0075] like Figure 7 As shown, in this embodiment Figure 2 A balanced feature fusion module 23 (Balanced Feature Fusion) is added on the basis of , in which the first linear interpolation unit 231 is used to make features of different levels become features of the same size for subsequent feature aggregation; taking feature map P4 as an example, feature maps P2 and P3 can be downsampled, and feature maps P5 and P6 can be upsampled to the same resolution as P4 before feature aggregation. The feature aggregation unit 232 can use the concat method, the Add method, or a mixture of the two; the aggregation operation is generally followed by a convolution operation to further fuse the features. The second linear interpolation unit 233 is used to restore the aggregated features to the same resolution as feature maps P2, P3, P4, P5, and P6, and obtain feature maps B2, B3, B4, B5, and B6.

[0076] Whether it is FPN or dual feature pyramid module, the information is transmitted layer by layer, usually in a top-down or bottom-up manner. This layer-by-layer fusion method makes the comprehensive features focus more on the information of adjacent layers, and pay less attention to the semantic information of non-adjacent layers. In the information flow, the semantic information of non-adjacent layers will gradually dilute the feature information far away from the current layer in each fusion process, which may lead to insufficient differentiation of feature representation. Adding a balanced feature fusion module will make each layer contain information from other layers. This balanced information flow makes the extracted features more distinctive.

[0077] like Figure 2 As shown, the output head module 3 includes a classification head 31 and a regression head 32. The classification head 31 is used to perform a classification operation on the multi-scale feature map to obtain the probability that each candidate box on each feature point is recognized as a target. The regression head 32 is used to perform a regression operation on the multi-scale feature map to obtain the position information of each candidate box on each feature point.

[0078] Specifically, the classification head (ClassificationHead) 31 and the regression head (RegressionHead) 32 classify and regress the feature maps N2, N3, N4, N5, N6 or the feature maps B2, B3, B4, B5, B6 generated by the dual feature pyramid module 2, respectively. Figure 8 As shown, the classification head ( Figure 8 The upper branch in the Figure 8 The network structures of the two parts are the same except for the size of the output layer. That is, after a series of output head processing operations (CGA operations), the CGA operation consists of convolution, group normalization and activation function Act, the final feature map is converted into output. The output dimension of the classification head is ,in, Represents the number of candidate boxes set for each feature point on the feature map. represents the number of categories, that is, the number of types predicted as pulmonary nodules, such as solid nodules, ground glass nodules, etc. The output dimension of the regression head is ,in, Represents the information of the candidate box (center point representation or vertex representation). It represents the number of candidate boxes set for each feature point on the feature map. Optionally, when using the adaptive training sample to select the ATSS scheme for positive and negative sample matching, A can be set to 1, that is, each feature point is assigned only one candidate box to achieve a good detection effect.

[0079] The output head module of this embodiment provides the probability of each candidate box in each lung CT image through the classification head, and the regression head provides the precise position of the candidate box.

[0080] In one embodiment, step S2 calculates the loss function according to the classification result and the regression result to obtain the classification loss and the regression loss, including:

[0081] S21, using an adaptive training sample selection method to perform positive and negative sample matching on all generated candidate boxes to obtain a positive sample set and a negative sample set of the candidate boxes.

[0082] S22, performing hard negative sample sampling according to the classification results, the positive sample set and the negative sample set, and adjusting the ratio of the positive sample set to the negative sample set within a preset range.

[0083] During the training process, a large number of candidate frames will be generated, among which the number of positive sample frames is only a few dozen at most, or even only a few. This imbalance between positive and negative samples will cause the training process to be overwhelmed by negative samples, which is not conducive to training. In addition, most of the negative sample frames are not in the transition area between the foreground and the background, and the classification is very clear (this is called a simple sample or an easy sample). The loss generated by such samples is very small, and the corresponding gradient will also be very small. The small gradient will have a limited effect on the convergence of the model, so difficult negative samples that can generate greater losses and greater gradients are more meaningful for the training model. Therefore, before calculating the classification loss and regression loss, use the difficult negative samples to adjust the model to control the ratio of positive and negative samples within a preset range so that the positive samples are not overwhelmed.

[0084] S23, performing cross entropy loss function calculation on the data after sampling the difficult negative samples to obtain a classification loss function, and performing weighted intersection-over-union loss function calculation on the data after sampling the difficult negative samples to obtain a regression loss function.

[0085] The classification loss uses cross entropy loss, and the regression loss function is WIoU Loss. When Smooth L1 is used to calculate the bounding box loss of target detection, it is necessary to independently calculate the Loss of 4 points, and then add them together to get the final bounding box loss. This operation assumes that the 4 points are independent of each other. In fact, in a candidate box, these 4 points are strongly correlated. The losses of multiple candidate boxes calculated using Smooth L1 Loss may be the same, but the intersection over union (IoU) is very different. Therefore, it is more reasonable to use IoU Loss for target detection. In the present invention, the latest version of IoU Loss, WIoU Loss, is selected.

[0086] Furthermore, step S21 uses an adaptive training sample selection method to perform positive and negative sample matching on all generated candidate boxes to obtain a positive sample set and a negative sample set of the candidate boxes, including:

[0087] S211, calculate the distance between the center point of each candidate box and the center point of the current pre-annotated box, select the K candidate boxes with the closest distance from the candidate boxes at each level as candidate positive samples, and form a set of all candidate positive samples in the current pre-annotated box .

[0088] like Fig. 9 As shown in the figure, a1 is a lung nodule in a lung CT image, a2 is a pre-labeled box, and a3 is a candidate box predicted by the model. The distance between the center point of the candidate box a3 and the center point of the current pre-labeled box a2 can be calculated.

[0089] S212, calculating the intersection-over-union ratio of each candidate positive sample and the pre-labeled box, calculating the mean and standard deviation of the intersection-over-union ratio, and setting a threshold of the intersection-over-union ratio according to the mean and standard deviation.

[0090] S213, if the intersection-over-union ratio of the candidate positive sample is greater than or equal to the threshold, it is marked as a positive sample, and the set of positive samples of all pre-annotated boxes is counted as , the remaining candidate boxes are counted as negative sample sets .

[0091] Specifically, assuming is the set of all pre-annotated boxes in a lung CT image. is the set of all candidate boxes at all levels, is the set of all candidate boxes in each level, is the set of candidate positive samples, is the final set of positive samples, is the final set of negative samples.

[0092] For the pre-labeled boxes, calculate the set The center point of each candidate box and the center point of the current pre-annotated box distance, and from each set separately Select K candidate boxes that are closest to the center point of the current pre-annotated box as candidate positive samples. All candidate positive samples of the current pre-annotated box form a set ; Calculate the current pre-annotated box and all candidate positive sample sets The intersection over union (IoU) between them is calculated, and the mean and standard deviation (std) of this group of intersection over union (IoU) are calculated; the threshold value of the intersection over union (IoU) is set to ; For each candidate positive sample, when IoU>= , the candidate positive sample is set as the positive sample; according to the above steps, the statistical set The set of positive samples assigned to all pre-annotated boxes in , then the negative sample set = - .

[0093] This embodiment first uses the principle of close center distance to screen candidate positive samples, and then finally confirms the positive samples based on the intersection-and-union principle. When calculating the intersection-and-union threshold, the statistical results of all candidate positive samples are comprehensively considered. The closer the center point of the candidate box is to the center point of the pre-labeled box, the greater the intersection-and-union ratio between the candidate box and the pre-labeled box. Therefore, a high intersection-and-union ratio indicates that the candidate box can cover the lung nodules more accurately, indicating that the model has better detection performance in this area. The mean represents the average level of matching with the pre-labeled box, and the standard deviation std reflects the difference in the degree of matching between these candidate boxes and the pre-labeled boxes. The larger the standard deviation std, the better the positive sample candidate box on a certain feature layer matches the pre-labeled box, and the higher the quality of this batch of positive sample candidate boxes, so setting the threshold to the mean mean plus the standard deviation std can screen out higher quality positive sample candidate boxes.

[0094] Under the intersection-union matching principle, small targets may have no or only a very small number of positive samples corresponding to them, while large targets will be assigned more positive samples. During the sampling process, the probability of positive samples corresponding to small targets being sampled will be much lower than the probability of positive samples corresponding to large targets, which ultimately leads to low detection performance of small targets. The adaptive training sample selection method dynamically selects positive samples for each target (pre-labeled box) based on the mean and variance calculated by the intersection-union ratio, which can ensure that the number of positive samples assigned to large and small targets is on the same order of magnitude, effectively improving the problems under the intersection-union matching principle and helping to improve the detection performance of small targets. In addition, the adaptive training sample selection method calculates the threshold based on statistical features, which is an adaptive strategy with strong robustness to different detection scenarios.

[0095] The embodiment of the present invention further provides a method for identifying lung nodules, which is executed by an electronic device such as a computer or a server, and specifically includes:

[0096] The lung nodule recognition model trained by the lung nodule recognition model training method is used to recognize lung images to obtain classification results and regression results. The classification result represents the probability of nodules predicted by the lung nodule recognition model, and the regression result represents the location information of nodules predicted by the lung nodule recognition model. Fig. 9For example, it is the recognition result of the lung nodule recognition model on the lung image, and the candidate box a3 is marked with the classification result (the probability of the lung nodule) and the regression result (the position of the candidate box). The candidate box a3 may completely include the lung nodule or may partially include the lung nodule.

[0097] This embodiment uses a trained lung nodule recognition model to identify lung nodules in lung images. The automated recognition process significantly improves the efficiency of lung nodule detection and reduces the false positive rate.

[0098] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A pulmonary nodule recognition model training method, characterized in that: include: A lung image is identified using a lung nodule recognition model, wherein the lung nodule recognition model includes a backbone network module, a dual-feature pyramid module, and an output head module, wherein the backbone network module is used to extract a multi-level feature map from an input lung image, the dual-feature pyramid module is used to fuse the multi-level feature maps to enhance the multi-scale representation of features, and the output head module is used to classify and regress the multi-scale feature maps generated by the dual-feature pyramid module to obtain a classification result and a regression result, wherein the classification result indicates the probability of a nodule predicted by the lung nodule recognition model, and the regression result indicates the location information of the nodule predicted by the lung nodule recognition model; The backbone network module includes: an initial convolution layer, a second multi-scale convolution layer, a third multi-scale convolution layer, a fourth multi-scale convolution layer, and a fifth multi-scale convolution layer. The initial convolution layer is used to extract preliminary features from the input lung image. The second multi-scale convolution layer, the third multi-scale convolution layer, the fourth multi-scale convolution layer, and the fifth multi-scale convolution layer are used to perform convolution operations of different scales on the preliminary features to gradually extract and enhance the intermediate to high-level feature information in the image. The second multi-scale convolution layer, the third multi-scale convolution layer, the fourth multi-scale convolution layer, and the fifth multi-scale convolution layer respectively include multi-scale bottleneck units, and the multi-scale bottleneck units include: convolution Branch, multi-scale convolution branch, feature splicing layer, dimension adjustment convolution layer, residual connection layer, non-linear activation layer, the convolution branch is used to extract standard scale feature information of input feature, the multi-scale convolution branch is used to extract and fuse multi-scale feature information of input feature, the feature splicing layer is used to splice the features of convolution branch and multi-scale convolution branch, the dimension adjustment convolution layer is used to adjust the dimension of the spliced ​​features, the residual connection layer is used to add the dimension adjusted features to the input features to form a residual connection, and the non-linear activation layer is used to convert the feature map after residual connection into a non-linear feature map using a non-linear activation function; Calculate the loss function according to the classification result and the regression result to obtain the classification loss and the regression loss; Perform a weighted summation of the classification loss and the regression loss to obtain a total loss, and calculate the gradient of the total loss using a back propagation algorithm; The network parameters of the pulmonary nodule recognition model are updated according to the gradient.

2. The method according to claim 1, characterized in that The multi-scale convolution branch includes a multi-scale self-calibration module, which includes a segmentation unit, a self-calibration processing unit and a splicing unit. The segmentation unit is used to divide the input feature map into four feature maps on average to obtain the feature map. , feature map , feature map , feature map The self-calibration processing unit includes three self-calibration branches of different scales and a standard convolution branch. The three self-calibration branches of different scales are used to calibrate the feature map , feature map and feature map Perform self-calibration operation to obtain the characteristic map , feature map , feature map , the standard convolution branch is used to map the feature map Perform convolution operation to obtain feature map , the concatenation unit is used for the feature map , feature map , feature map and feature map Splice to get the feature map .

3. The method according to claim 2, characterized in that The self-calibration branch includes a feature calibration module and a feature fusion module. The feature calibration module is used to calibrate the input feature map. Perform R times downsampling and then After the convolution operation, upsample by R times to restore the feature map size, and compare the restored feature map with the input feature map The result after addition is used to calculate the activation function and generate feature weights; The feature fusion module is used to input feature maps conduct Convolution operation is used to extract preliminary features and then The convolution result is multiplied by the feature weight to perform weighting, and the weighted feature map is processed. Convolution operation is performed to obtain the output feature map.

4. The method according to claim 1, characterized in that: The dual-feature pyramid module includes a semantic feature propagation unit and a feature positioning enhancement unit. The semantic feature propagation unit is used to propagate high-level semantic information from top to bottom according to the multi-level feature map, and transfer rich semantic features to all feature levels layer by layer to obtain a multi-scale semantic feature map. The feature positioning enhancement unit is used to propagate the bottom-level precise positioning information layer by layer from bottom to top according to the multi-scale semantic feature map to obtain a multi-scale positioning enhanced feature map.

5. The method according to claim 4, characterized in that The dual feature pyramid module also includes a balanced feature fusion module, which is used to balance and fuse multi-scale positioning enhancement feature maps, integrate feature information at different levels, and improve the model's processing capabilities for multi-scale features, wherein the balanced feature fusion module includes a first linear interpolation unit, a feature aggregation unit, and a second linear interpolation unit, wherein the first linear interpolation unit is used to adjust the resolution of the multi-scale positioning enhancement feature map, the feature aggregation unit is used to aggregate the feature map processed by the first linear interpolation unit, and the second linear interpolation unit is used to restore the aggregated feature map to the resolution of the original positioning enhancement feature map.

6. The method according to claim 1, characterized in that The output head module includes a classification head and a regression head. The classification head is used to perform a classification operation on the multi-scale feature map to obtain the probability distribution of each candidate box on each feature point. The regression head is used to perform a regression operation on the multi-scale feature map to obtain the position information of each candidate box on each feature point.

7. The method according to claim 6, characterized in that The loss function is calculated according to the classification result and the regression result to obtain the classification loss and the regression loss, including: Use the adaptive training sample selection method to perform positive and negative sample matching on all generated candidate boxes to obtain the positive sample set and negative sample set of the candidate boxes; Performing difficult negative sample sampling according to the classification results, the positive sample set and the negative sample set, and adjusting the ratio of the positive sample set to the negative sample set within a preset range; The cross entropy loss function is calculated on the data after sampling the difficult negative samples to obtain the classification loss function, and the weighted intersection-over-union loss function is calculated on the data after sampling the difficult negative samples to obtain the regression loss function.

8. The method according to claim 7, characterized in that The adaptive training sample selection method is used to match positive and negative samples of all generated candidate boxes to obtain the positive sample set and negative sample set of the candidate boxes, including: Calculate the distance between the center point of each candidate box and the center point of the current pre-labeled box, select the K candidate boxes closest to each level of the candidate box as candidate positive samples, and form a set of all candidate positive samples in the current pre-labeled box ; Calculate the intersection-over-union ratio of each candidate positive sample and the pre-labeled box, calculate the mean and standard deviation of the intersection-over-union ratio, and set the threshold of the intersection-over-union ratio based on the mean and standard deviation; If the intersection-over-union ratio of the candidate positive sample is greater than or equal to the threshold, it is marked as a positive sample, and the set of positive samples of all pre-annotated boxes is counted as , the remaining candidate boxes are counted as negative sample sets .

9. A lung nodule recognition model training device, characterized in that: include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor executes the lung nodule recognition model training method as described in any one of claims 1-8.

10. A lung nodule identification device, characterized in that: include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor uses a lung nodule recognition model trained by the method of any one of claims 1 to 8 to recognize lung images and obtain classification results and regression results, wherein the classification result represents the probability of nodules predicted by the lung nodule recognition model, and the regression result represents the location information of nodules predicted by the lung nodule recognition model.

Images