A multi-domain lung nodule detection method, device, equipment and readable storage medium

Abstract

The application relates to a multi-domain lung nodule detection method, device and equipment and a readable storage medium, which comprises a segmentation module, which performs axial, coronal and sagittal segmentation on a 3D feature map to obtain a plurality of slice groups in each direction; a projection sub-module, which performs nonlinear projection on each slice group to obtain a splicing adjustment value of each slice group; a domain perception sub-module, which processes each slice group to obtain a domain perception weight value of each slice group, combines the domain perception weight value and the splicing adjustment value to obtain a response vector of each slice group; a splicing and fusion sub-module, which adjusts the channel response of the slice group through the response vector to obtain a target slice group, splices the target slice groups in the same direction to obtain a target splicing map in each direction, fuses the target splicing maps in all directions based on a three-way cross attention mechanism to obtain a fusion feature map, and a detection head performs lung nodule detection and recognition on the fusion feature map, thereby improving the accuracy and robustness of multi-center lung nodule detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image lesion detection, and in particular to a multi-domain lung nodule detection method, device and equipment and readable storage medium. BACKGROUND

[0002] With the rapid development of deep learning and the development of domestic multi-center medical big data platform, artificial intelligence technology has been widely applied in the field of medical image lesion detection. In the case of increasing image data year by year, it can help clinicians reduce reading time and improve diagnostic accuracy. Compared with the traditional method of feature extraction by using morphological operation and low-level descriptor operation, which has the problem of relying on artificial design, poor generalization and robustness, deep learning as a multi-layer neural network algorithm can adjust layer by layer, automatically extract multi-level low-level features, and convert them into more complex abstract features to complete the learning task. In particular in the medical field, deep learning has great advantages in image analysis, and its effect greatly exceeds that of traditional methods, and the algorithm diagnosis accuracy is very high.

[0003] Among many medical image lesion intelligent detection scenarios, lung nodule detection is undoubtedly the most demanding and highest value scenario task. Among them, the examination amount of chest CT is the largest in imaging examination, and other imaging examinations are not in the same order of magnitude, and lung nodules, especially small lung nodules, are not easy to distinguish, which is the most assisted scenario for doctors. In addition, since early lung cancer often lacks obvious clinical symptoms, most patients are found to be in the middle and late stages, and the 5-year survival rate is only 16%-18%, but if it can be diagnosed in the early stage of the disease, the 5-year survival rate will be greatly improved. Therefore, timely diagnosis of lung nodules can achieve early screening of lung cancer.

[0004] At present, great breakthroughs have been made in the development of lung nodule intelligent detection technology, but there are still the following challenges: medical multi-center clinical verification has high requirements for the robustness of lung nodule detection algorithms, but the robustness of current lung nodule detection algorithms is poor; and there is a great domain difference between different medical center data, that is, the difference between CT image data collected by different medical centers forms different domains, such as CT equipment, doctor's labeling habits, and the difference in the distribution of benign and malignant nodules in the data set and the difference in the field caused by the difference in the radiation dose, etc., so that the accuracy of lung nodule detection is low. SUMMARY

[0005] The present application provides a multi-domain lung nodule detection method, device, equipment and readable storage medium to solve the problem of poor robustness and low accuracy of multi-center lung nodule detection algorithm in related technologies.

[0006] In a first aspect, a multi-domain lung nodule detection method is provided, comprising the following steps:

[0007] inputting a CT image to be detected into a preset lung nodule detection model, the lung nodule detection model comprising a backbone network, a domain adaptation module and a detection head, the domain adaptation module comprising a segmentation sub-module, a projection sub-module, a domain perception sub-module and a splicing fusion sub-module;

[0008] the backbone network performing feature extraction on the CT image to be detected to obtain a 3D feature map;

[0009] the segmentation sub-module performing segmentation on the 3D feature map along an axial direction, a coronal direction and a sagittal direction respectively to obtain a plurality of slice groups corresponding to each direction;

[0010] the projection sub-module performing nonlinear projection on each slice group respectively to obtain a splicing adjustment value corresponding to each slice group;

[0011] the domain perception sub-module performing global pooling, full connection and normalization processing on each slice group to obtain a domain perception weight value corresponding to each slice group, and combining the domain perception weight value corresponding to each slice group and the splicing adjustment value to obtain a response vector corresponding to each slice group;

[0012] the splicing fusion sub-module adjusting the channel response of the slice groups through the response vector to obtain a target slice group, splicing the target slice groups in the same direction to obtain a target splicing map corresponding to each direction, and fusing the target splicing maps of all directions based on a three-way cross-attention mechanism to obtain a fusion feature map;

[0013] the detection head performing detection and recognition on the fusion feature map to output a lung nodule detection result corresponding to the CT image to be detected.

[0014] In some embodiments, the projection sub-module performs nonlinear projection on each slice group respectively to obtain a splicing adjustment value corresponding to each slice group, comprising:

[0015] projecting each slice group onto N slice grouping adapters respectively, wherein N is a positive integer;

[0016] performing 3D global pooling and full connection processing on the slice groups on each slice grouping adapter respectively according to a first calculation formula to obtain N initial adjustment values corresponding to each slice group;

[0017] splicing the N initial adjustment values corresponding to each slice group according to a second calculation formula to obtain a splicing adjustment value of each slice group in the corresponding direction;

[0018] the first calculation formula is:

[0019]

[0020] wherein, denotes the i-th slice group in the direction, denotes the initial adjustment value corresponding to the i-th slice group in the direction on the j-th slice packet adapter, denotes one of the axial, coronal and sagittal directions, denotes the i-th slice group in the direction, denotes the i-th slice group in the direction, F SE denotes the squeezing and excitation function, F avg denotes the 3D global pooling operation, and denotes the fully connected layer, and denotes the ReLU activation function;

[0021] The second calculation formula is:

[0022]

[0023] In the formula, denotes the i-th slice group in the direction, denotes the i-th slice group in the direction.

[0024] In some embodiments, the domain-aware sub-module obtains the domain-aware weight value corresponding to each slice group after global pooling, full connection and normalization processing of each slice group, including:

[0025] The domain-aware weight value corresponding to each slice group is obtained by global pooling, full connection and normalization processing of each slice group through a third calculation formula;

[0026] The third calculation formula is:

[0027]

[0028] In the formula, denotes the i-th slice group in the direction, denotes the i-th slice group in the direction, denotes one of the axial, coronal and sagittal directions, F DA denotes the domain-aware function, denotes the i-th slice group in the direction, denotes the i-th slice group in the direction, and softmax denotes the normalized exponential function, denotes the fully connected layer, F avg denotes the 3D global pooling operation.

[0029] In some embodiments, the concatenation fusion sub-module obtains the target slice group by adjusting the channel response of the slice group through the response vector, including:

[0030] The adjusted target slice group is obtained by adjusting the channel response of the slice group through the response vector according to a fourth calculation formula;

[0031] The fourth calculation formula is:

[0032]

[0033] In the formula, represents a target slice group corresponding to the i-th slice group in the direction, represents one of the axial direction, the coronal direction and the sagittal direction, represents the i-th slice group in the direction, F scale represents a channel-by-channel multiplication, represents a response vector corresponding to the i-th slice group in the direction, and σ represents a sigmoid activation function.

[0034] In some embodiments, the splicing of the target slice groups in the same direction to obtain a target splicing image corresponding to each direction comprises:

[0035] splicing the target slice groups in the same direction to obtain a target splicing image corresponding to each direction by a fifth calculation formula;

[0036] The fifth calculation formula is:

[0037]

[0038] In the formula, represents a target splicing image in the direction, and G represents There are G target slice groups in the direction.

[0039] In some embodiments, the fusing of the target splicing images of all directions to obtain a fused feature map based on a three-way cross-attention mechanism comprises:

[0040] fusing the target splicing images of all directions to obtain a fused feature map by a sixth calculation formula;

[0041] The sixth calculation formula is:

[0042]

[0043] In the formula, represents a fused feature map, represents a target splicing image in the axial direction, represents a target splicing image in the coronal direction, represents a target splicing image in the sagittal direction, W CA , W θ , W φ and W gConvolutional layers, and softmax represents a normalized exponential function.

[0044] In some embodiments, the 3D feature map to be detected includes information of four dimensions of depth, height, width, and channel number.

[0045] In a second aspect, a multi-domain lung nodule detection device is provided, comprising: an image input unit and a lung nodule detection model, wherein the lung nodule detection model comprises a backbone network, a domain adaptation module, and a detection head, and the domain adaptation module comprises a segmentation sub-module, a projection sub-module, a domain perception sub-module, and a splicing fusion sub-module.

[0046] The image input unit is configured to input a CT image to be detected into the lung nodule detection model.

[0047] The backbone network is configured to extract features of the CT image to be detected to obtain a 3D feature map.

[0048] The segmentation sub-module is configured to segment the 3D feature map along an axial direction, a coronal direction, and a sagittal direction respectively to obtain a plurality of slice groups corresponding to each direction.

[0049] The projection sub-module is configured to perform nonlinear projection on each slice group to obtain a splicing adjustment value corresponding to each slice group.

[0050] The domain perception sub-module is configured to perform global pooling, full connection, and normalization processing on each slice group to obtain a domain perception weight value corresponding to each slice group, and combine the domain perception weight value corresponding to each slice group and the splicing adjustment value to obtain a response vector corresponding to each slice group.

[0051] The splicing fusion sub-module is configured to adjust the channel response of the slice group through the response vector to obtain a target slice group, splice the target slice groups in the same direction to obtain a target splicing map corresponding to each direction, and fuse the target splicing maps of all directions to obtain a fusion feature map.

[0052] The detection head is configured to detect and identify the fusion feature map to output a lung nodule detection result corresponding to the CT image to be detected.

[0053] In a third aspect, a multi-domain lung nodule detection device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the multi-domain lung nodule detection method described above.

[0054] In a fourth aspect, a computer readable storage medium is provided, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-domain lung nodule detection method described above is implemented.

[0055] The application provides a multi-domain lung nodule detection method, device, equipment and readable storage medium. The method comprises inputting a CT image to be detected into a preset lung nodule detection model. The lung nodule detection model comprises a backbone network, a domain self-adaptive module and a detection head. The domain self-adaptive module comprises a segmentation sub-module, a projection sub-module, a domain perception sub-module and a splicing and fusion sub-module. The backbone network extracts features of the CT image to be detected to obtain a 3D feature map. The segmentation sub-module segments the 3D feature map along an axial direction, a coronal direction and a sagittal direction respectively to obtain a plurality of slice groups corresponding to each direction. The projection sub-module performs nonlinear projection on each slice group to obtain a splicing adjustment value corresponding to each slice group. The domain perception sub-module performs global pooling, full connection and normalization processing on each slice group to obtain a domain perception weight value corresponding to each slice group, and combines the domain perception weight value and the splicing adjustment value corresponding to each slice group to obtain a response vector corresponding to each slice group. The splicing and fusion sub-module adjusts the channel response of the slice group through the response vector to obtain a target slice group, splices the target slice groups in the same direction to obtain a target splicing map corresponding to each direction, fuses the target splicing maps of all directions based on a three-way cross attention mechanism to obtain a fusion feature map, and the detection head detects and identifies the fusion feature map to output a lung nodule detection result corresponding to the CT image to be detected. The application introduces the domain self-adaptive module into the lung nodule detection model to enhance the generalization performance of the network, can adaptively perform sufficient feature extraction on the domain differences of different medical centers, that is, can fully utilize the CT images of different medical centers and the shared knowledge of multiple medical centers, and thus improves the accuracy and robustness of multi-center lung nodule detection. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0057] Figure 1 A flowchart of a multi-domain lung nodule detection method provided by an embodiment of the application;

[0058] Figure 2 A structure diagram of a lung nodule detection model provided by an embodiment of the application;

[0059] Figure 3 A specific flowchart of multi-domain lung nodule detection provided by an embodiment of the application;

[0060] Figure 4 A structure diagram of a multi-domain lung nodule detection device provided by an embodiment of the application. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0062] The embodiments of the present application provide a multi-domain lung nodule detection method, device, equipment and readable storage medium, which can solve the problems of poor robustness and low precision of multi-center lung nodule detection algorithms in related technologies.

[0063] Figure 1 The multi-domain lung nodule detection method provided by the embodiments of the present application comprises the following steps:

[0064] Step S10: inputting a to-be-detected CT image into a preset lung nodule detection model, wherein the lung nodule detection model comprises a backbone network, a domain adaptation module, and a detection head, and the domain adaptation module comprises a segmentation sub-module, a projection sub-module, a domain perception sub-module, and a splicing and fusion sub-module.

[0065] Illustratively, a conventional deep learning lung nodule detection network is usually composed of a backbone network and a detection head, wherein the backbone network is used for feature extraction of a 3D CT image, and the detection head is used for lung nodule detection on the 3D feature map extracted by the backbone network. It should be understood that the to-be-detected CT image in the embodiments of the present application is obtained from a multi-center lung nodule dataset, and there is a domain difference between the CT images in the multi-center lung nodule dataset. However, since the existing backbone network is directly trained on medical images in multiple centers, a selective learning behavior occurs, that is, the network only effectively learns the “dominant” data part in the multi-medical center data, and ignores the domain knowledge (i.e., the part where the data is different) brought by the remaining data, thereby causing the model to show inconsistency or performance decline in a single center / field. Therefore, the conventional deep learning lung nodule detection network cannot achieve accurate lung nodule detection on the to-be-detected CT image.

[0066] The embodiments of the present application introduce a domain adaptation module in the residual layer of the backbone network, which can fully utilize the CT images of different medical centers and the shared knowledge of multiple medical centers, so that the final network has better precision and robustness for multi-medical center lung nodule detection, and avoids inconsistency or performance decline in a single center / field. Specifically, referring to Figure 2As shown, the lung nodule detection model constructed in the embodiment mainly consists of a backbone network layer, a domain adaptation module and a detection head. The backbone network consists of a 3D neural network layer and a residual layer constructed by a 3D neural network layer. When a 3D CT image is input into the backbone network, a 3D feature map will be obtained after feature extraction of each layer. The domain adaptation module is a plug-and-play module suitable for various lung nodule detection algorithms, mainly including a segmentation sub-module, a projection sub-module, a domain perception sub-module and a splicing and fusion sub-module. It can adaptively perform sufficient feature extraction on different medical center data and output a modulated 3D feature map based on the original 3D feature map. The detection head consists of a classification layer and a regression layer, which performs lung nodule detection on the modulated 3D feature map output by the domain adaptation module.

[0067] Step S20: The backbone network performs feature extraction on the to-be-detected CT image to obtain a 3D feature map; wherein the to-be-detected 3D feature map includes information of four dimensions of depth, height, width and channel number;

[0068] Exemplarily, in the embodiment, the backbone network consists of a 3D neural network layer and a residual layer constructed by a 3D neural network layer. When a 3D CT image to be detected is input into the backbone network, a 3D feature map will be obtained after feature extraction of each layer, and the dimension of the 3D feature map X is CxDxHxW. Wherein C, D, H and W respectively represent the channel number, depth, height and width of the feature map X.

[0069] Step S30: The segmentation sub-module performs segmentation on the 3D feature map along the axial direction, the coronal direction and the sagittal direction respectively to obtain a plurality of slice groups corresponding to each direction;

[0070] Exemplarily, in the embodiment, the segmentation sub-module divides the input 3D feature map X into G groups along the axial direction, the coronal direction and the sagittal direction to obtain slice groups in each direction: X a (i), X c (i) and X s (i), a, c and s respectively represent the axial direction, the coronal direction and the sagittal direction, i = 1, …, G, which is used to identify the i-th slice group; wherein X a (i), X c (i) and X s (i) are also of the dimensions of depth, height, width and channel number, i.e. X a (i) is of the dimension of CxD'HxW, D' = D / H, X c (i) is of the dimension of CxDxH'xW, H' = H / G, X s (i) is of the dimension of CxDxHxW', W' = W / G. It should be noted that G is a preset value, which can be determined according to actual requirements, and is not limited herein, for example, the value of G can be 4.

[0071] Step S40: The projection sub-module respectively performs nonlinear projection on each slice group to obtain a splicing adjustment value corresponding to each slice group;

[0072] Exemplarily, in the embodiment, the projection sub-module is a general adapter which is mainly composed of a plurality of slice grouping adapters and can capture feature subspaces of a field covered by data of all medical centers in groups in axial, coronal and sagittal directions, i.e., each group of slices is projected nonlinearly onto N slice grouping adapters (i.e., field feature subspaces) driven by data to obtain a splicing adjustment value corresponding to N projections of each group of slices in three directions wherein, indicates the axial, coronal or sagittal direction, and N is a preset value which can be determined according to actual requirements and is not limited herein, for example, the value of N can be 3.

[0073] Further, the projection sub-module respectively performs nonlinear projection on each slice group to obtain a splicing adjustment value corresponding to each slice group, including:

[0074] projecting each slice group onto N slice grouping adapters, wherein N is a positive integer;

[0075] performing 3D global pooling and full connection processing on the slice group on each slice grouping adapter according to a first calculation formula to obtain N initial adjustment values corresponding to each slice group;

[0076] splicing the N initial adjustment values corresponding to each slice group according to a second calculation formula to obtain a splicing adjustment value of each slice group in the corresponding direction;

[0077] The first calculation formula is:

[0078]

[0079] wherein, indicates an initial adjustment value corresponding to the i-th slice group in the direction on the j-th slice grouping adapter, indicates one of the axial, coronal and sagittal directions, indicates the i-th slice group in the direction, and F SE indicates a squeezing and excitation function, F avg indicates a 3D global pooling operation, and indicates a full connection layer, and δ indicates a ReLU activation function;

[0080] The second calculation formula is:

[0081]

[0082] wherein, denotes the initial adjustment value of the i-th slice group in the direction of the j-th slice grouping adapter.

[0083] Exemplarily, in the present embodiment, each group of slices the output of the j-th slice grouping adapter in the universal adapter can be obtained by the following formula:

[0084]

[0085] wherein, denotes the initial adjustment value of the i-th slice group in the direction of the j-th slice grouping adapter, denotes one of the axial, coronal and sagittal directions, denotes the i-th slice group in the direction of the j-th slice grouping adapter, F SE denotes the squeeze-and-excitation function, F avg denotes the 3D global pooling operation, is a fully connected layer of , and is a fully connected layer of , r denotes a channel size reduction factor, and δ denotes a ReLU activation function. Since there are N slice grouping adapters, each group of slices outputs N initial adjustment values in its corresponding direction

[0086] For each group of slices the N initial adjustment values in its corresponding direction are spliced, i.e. the spliced adjustment value of each group of slices in its corresponding direction is obtained. Wherein, the spliced adjustment value of each group of slices in its corresponding direction can be obtained by the following formula:

[0087]

[0088] wherein, denotes the spliced adjustment value of the i-th slice group in the direction of the j-th slice grouping adapter. It can be understood that the corresponding dimension of the spliced adjustment value is CxN.

[0089] Step S50: The domain-aware submodule performs global pooling, full connection and normalization on each slice group to obtain the domain-aware weight value corresponding to each slice group, and combines the domain-aware weight value and the splicing adjustment value corresponding to each slice group to obtain the response vector corresponding to each slice group.

[0090] As an example, in this embodiment, the stitching corresponds to N projections of each group of slices in each direction. The input will be fed into the domain-aware submodule in the corresponding direction. The domain-aware submodule generates corresponding domain-aware weights for each group of slices through a domain allocation mechanism. Furthermore, for any slice group, the projection output by the slice grouping adapter in the corresponding direction of that slice group (i.e., ) and domain-aware weights By combining them, we can obtain the neighborhood-adapted response vector of this slice group. Wherein, DA (Domain Attentive / Attention) represents domain awareness or domain attention, used to indicate It refers to the weights specific to a given domain.

[0091] Furthermore, the domain-aware submodule performs global pooling, fully connected processing, and normalization on each slice group to obtain the domain-aware weight value corresponding to each slice group, including:

[0092] The domain-aware weight value for each slice group is obtained by performing global pooling, full connection, and normalization on each slice group using the third calculation formula.

[0093] The third calculation formula is:

[0094]

[0095] In the formula, express The domain-aware weight value corresponding to the i-th slice group in the direction. F represents one of the following directions: axial, posterior, and sagittal. DA Represents the domain-aware function. express The i-th slice group in the direction, where softmax represents the normalized exponential function. F represents a fully connected layer. avg This indicates a 3D global pooling operation.

[0096] As an example, in this embodiment, the slice group exist Domain-aware weights in direction It can be obtained through the following formula:

[0097]

[0098] wherein, denotes the field-aware weight value corresponding to the i-th slice group in the direction F DA denotes the field-aware function, and softmax denotes the normalized exponential function, denotes a fully connected layer with a dimension of NxC. Then the field-aware weight value F is used to combine the output of the i-th slice group of the general adapter in the corresponding direction (i.e. ), and thus the response vector of the slice group after field adaptation is obtained:

[0099]

[0100] wherein, denotes the response vector corresponding to the i-th slice group in the direction.

[0101] Step S60: The splicing and fusion sub-module adjusts the channel response of the slice group through the response vector to obtain a target slice group, splices the target slice groups in the same direction to obtain a target splicing map corresponding to each direction, fuses the target splicing maps of all directions based on a three-way cross attention mechanism to obtain a fused feature map;

[0102] Exemplarily, in the present embodiment, the response vector of each group of slices is used to adjust the channel response of the slice group to obtain the modulated target slice group corresponding to the slice group, and then the modulated target slice groups in the same direction are spliced together, i.e., the target splicing maps in three directions and The three target splicing maps in three directions and are fused through a three-way cross attention mechanism, thereby obtaining the modulated fused feature map corresponding to the 3D feature map As can be seen, the field adaptation module of the present embodiment realizes the automatic reasoning of the field by soft allocating the feature maps projected by the general adapter from three directions in groups through the field allocation mechanism, and fuses the result feature maps in three directions through a three-way cross attention mechanism, which takes into account the difficult situation of large field difference of multi-center medical image data, so as to realize the robust and accurate detection of multi-center lung nodules by the detection head.

[0103] Further, the splicing and fusion sub-module adjusts the channel response of the slice group through the response vector to obtain a target slice group, comprising:

[0104] According to a fourth calculation formula and by adjusting the channel responses of the slice groups through the response vectors, an adjusted target slice group is obtained;

[0105] The fourth calculation formula is:

[0106]

[0107] In the formula, F represents the i-th slice group in the direction, and F represents the target slice group corresponding to the i-th slice group in the direction. represents the target slice group corresponding to the i-th slice group in the direction, represents the i-th slice group in the direction, represents one of the axial direction, the coronal direction, and the sagittal direction, represents the target slice group corresponding to the i-th slice group in the direction, represents the i-th slice group in the direction, and F represents the target slice group corresponding to the i-th slice group in the direction. scale represents the channel-by-channel multiplication, represents the target slice group corresponding to the i-th slice group in the direction, represents the response vector corresponding to the i-th slice group in the direction, and σ represents the sigmoid activation function.

[0108] The target slice groups in the same direction are spliced to obtain a target splicing image corresponding to each direction, including:

[0109] The target slice groups in the same direction are spliced to obtain a target splicing image corresponding to each direction through a fifth calculation formula;

[0110] The fifth calculation formula is:

[0111]

[0112] In the formula, F represents the i-th slice group in the direction, and F represents the target slice group corresponding to the i-th slice group in the direction. represents the target splicing image in the direction, and G represents there are G target slice groups in the direction. Exemplarily, in the present embodiment, the response vector

[0113] is applied to the corresponding slice group to adjust the channel response of the corresponding slice group, and then the adjusted target slice group in the direction corresponding to each slice group is calculated and

[0114]

[0115] In the formula, F represents the i-th slice group in the direction, and F represents the target slice group corresponding to the i-th slice group in the direction. represents the target slice group corresponding to the i-th slice group in the direction, represents the i-th slice group in the direction, represents the target slice group corresponding to the i-th slice group in the direction, represents the i-th slice group in the direction, and F represents the target slice group corresponding to the i-th slice group in the direction. scale ​(i.e., channel-wise) represents a per-channel multiplication, i.e., each weight is directly multiplied onto an entire DxHxW of each channel, and represents a sigmoid activation function.

[0116] Then all the adjusted target slice groups in the same direction are spliced to obtain a target splicing map in the corresponding direction and

[0117]

[0118] It can be understood that, The dimension of is also CxDxHxW.

[0119] Further, the three-way cross attention mechanism based on fuses the target splicing maps in all directions to obtain a fused feature map, including:

[0120] The target splicing maps in all directions are fused to obtain a fused feature map by a sixth calculation formula;

[0121] The sixth calculation formula is:

[0122]

[0123] In the formula, represents a fused feature map, represents an axial target splicing map, represents a coronal target splicing map, represents a sagittal target splicing map, W CA , W θ , W φ , and W g all represent convolution layers, and softmax represents a normalized exponential function.

[0124] Exemplarily, in the present embodiment, the result feature maps in three directions are fused by the three-way cross attention mechanism to further obtain a modulated fused feature map corresponding to the 3D feature map input to the domain adaptation module Thus, the modulated fused feature map can be used to realize domain compensation. The calculation formula of the modulated fused feature map is:

[0125]

[0126]

[0127] In the formula, W CA , W θ , W φ , and W g ​are all 1x1x1 convolutional layers. It should be noted that, in order to further reduce the amount of calculation, the number of channels of W θ , W φ and W g may be set to half of the number of channels of W , and W CA is then restored to the original number of channels to realize matrix addition. A max-pooling layer can also be added after W φ and W g to further reduce the amount of calculation.

[0128] Step S70: The detection head detects and identifies the fused feature map to output a lung nodule detection result corresponding to the CT image to be detected.

[0129] Illustratively, in the present embodiment, the detection head classifies and regresses the anchor placed in advance on each voxel point of the modulated 3D fused feature map output by the domain adaptation module, and outputs the classification probability and boundary box of the lung nodule; and the two-stage detection head also uses the lung nodule boundary box output in the first stage to crop the 3D fused feature map extracted by the domain adaptation module, and further classifies and regresses the cropped region of interest to output the final lung nodule classification probability and boundary box.

[0130] As can be seen, the present embodiment introduces a multi-center slice grouping domain adaptation module into the lung nodule detection network to enhance the generalization performance of the network. The domain characteristic subspaces covered by multiple medical center data are captured from the axial, coronal and sagittal directions by the universal adapter of the domain adaptation module, and then the combination of the projections output by the universal adapter is soft-assigned in groups from the three directions by a domain assignment mechanism. Both of these two steps are data-driven and do not require prior information of the domain, and can realize automatic reasoning; then the three-direction result feature maps are fused by a three-way cross-attention mechanism, so that robust and accurate detection of lung nodules in multiple medical centers can be realized. In summary, the present application can make the lung nodule detection model fully utilize the CT images of different medical centers and the shared knowledge of multiple medical centers, so that the model has better precision and robustness for lung nodule detection in multiple medical centers.

[0131] The specific implementation process of the present embodiment will be explained below in conjunction with Figure 3 .

[0132] Step N10: Obtain the preprocessed CT image, and obtain a 3D feature map through the backbone network of the lung nodule detection model.

[0133] In order to ensure the accuracy of the lung nodule detection and prevent unnecessary interference, the CT images in the data set are preprocessed in this embodiment: the original CT data is converted into a standard quantitative value HU (Hounsfield Unit) value describing radiation density, and then the data is windowed and normalized; and in order to improve the detection accuracy and reduce the amount of calculation, some original CT data can also be resampled (that is, the voxel spacing of the CT image is converted to 1x1x1mm) and lung mask (that is, mask) extraction operation, and the preprocessed CT image (that is, the normalized CT image) is obtained after the above operation.

[0134] It should be understood that when preprocessing the CT image in the training stage, because of the limitation of the display memory, it is not feasible to use the whole CT image as input for model training, so the normalized CT image is cut into a patch block of 1x128x128x128 according to each nodule as the input of the training model, and if the cropped patch exceeds the range of the CT image, the exceeding part is completed by using a padding value of 170. In the test stage, in order to avoid the occurrence of odd size of the CT image, the image can be completed by using a padding value of 170 as the input of the test stage.

[0135] Therefore, in specific implementation, at the beginning of the process, it can be first judged whether it is a training stage, if so, the normalized CT image is cut into a patch block (that is, an image block) of 1x128x128x128 according to each nodule in the normalized CT image as the input of the model, and data augmentation can be set according to the model training requirements; if not, the normalized CT image is directly used as the input of the model. Among them, when the model is trained, the batch size is generally 8 or 16, and when the model is tested, the batch size is generally 1.

[0136] Step N20: obtaining the slice group in each direction, that is, dividing the 3D feature map into G groups along the axial, coronal and sagittal directions, and then obtaining the slice group in each direction

[0137] Step N30: obtaining the general representation of each group of slices in multiple fields in three directions, that is, obtaining the general representation of each group of slices in multiple fields in three directions Through the general adapter, nonlinearly project onto N field feature subspaces to obtain the splicing adjustment value corresponding to the N projections of each group of slices in three directions

[0138] Step N40: obtaining the field-aware weight, that is, obtaining the field-aware weight of each group of slices Input the corresponding field-aware submodule in the direction to generate the corresponding field-aware weight

[0139] Step N50: obtaining a response vector of each slice group, that is, a splicing adjustment value corresponding to each group of N projections of each slice group respectively applying a domain awareness weight obtaining a domain-adapted response vector corresponding to each slice group

[0140] Step N60: obtaining a modulated target slice group corresponding to each slice group, that is, by the response vector of each slice group adjusting the channel response of the corresponding slice group to obtain a slice group corresponding to the target slice group

[0141] Step N70: obtaining a target splicing map in three directions, that is, splicing the modulated target slice group in the same direction to obtain a target splicing map in three directions and

[0142] Step N80: obtaining a fusion feature map, that is, by a cross-attention mechanism to fuse the target splicing map in three directions and obtaining a fusion feature map

[0143] Step N90: continuing to input the fusion feature map into the lung nodule detection model, determining whether a domain adaptation module is encountered, if so, returning to continue to execute steps N20 to N80 until the tracking ends, otherwise continuing to input the fusion feature map into the detection head of the lung nodule detection model until a lung nodule detection result is obtained.

[0144] The embodiment of the present application also provides a multi-domain lung nodule detection device, comprising: an image input unit and a lung nodule detection model, wherein the lung nodule detection model comprises a backbone network, a domain adaptation module and a detection head, and the domain adaptation module comprises a slicing sub-module, a projection sub-module, a domain awareness sub-module and a splicing and fusion sub-module.

[0145] The image input unit is used to input a CT image to be detected into the lung nodule detection model;

[0146] The backbone network is used to extract features of the CT image to be detected to obtain a 3D feature map;

[0147] The slicing sub-module respectively slices the 3D feature map along the axial direction, the coronal direction and the sagittal direction to obtain a plurality of slice groups corresponding to each direction;

[0148] The projection submodule is configured to perform nonlinear projection on each slice group respectively to obtain a splicing adjustment value corresponding to each slice group.

[0149] The domain perception submodule is configured to perform global pooling, full connection and normalization processing on each slice group to obtain a domain perception weight value corresponding to each slice group, and combine the domain perception weight value and the splicing adjustment value corresponding to each slice group to obtain a response vector corresponding to each slice group.

[0150] The splicing fusion submodule is configured to adjust the channel response of the slice group through the response vector to obtain a target slice group, splice the target slice groups in the same direction to obtain a target splicing image corresponding to each direction, and fuse the target splicing images of all directions to obtain a fusion feature map.

[0151] The detection head is configured to detect and identify the fusion feature map to output a lung nodule detection result corresponding to the to-be-detected CT image.

[0152] Further, the projection submodule is specifically configured to:

[0153] project each slice group onto N slice grouping adapters respectively, wherein N is a positive integer;

[0154] perform 3D global pooling and full connection processing on each slice group on each slice grouping adapter according to a first calculation formula to obtain N initial adjustment values corresponding to each slice group;

[0155] splice the N initial adjustment values corresponding to each slice group according to a second calculation formula to obtain a splicing adjustment value of each slice group in a corresponding direction;

[0156] The first calculation formula is:

[0157]

[0158] In the formula, represents an initial adjustment value of an i-th slice group in a direction on a j-th slice grouping adapter, represents one of an axial direction, a coronal direction and a sagittal direction, represents an i-th slice group in a direction, F SE represents a squeezing and excitation function, F avg represents a 3D global pooling operation, and represents a full connection layer, and δ represents a ReLU activation function.

[0159] The second calculation formula is:

[0160]

[0161] wherein, denotes the i-th slice group in the direction corresponds to a splicing adjustment value.

[0162] Further, the domain perception sub-module is specifically configured to:

[0163] After global pooling, full connection and normalization processing of each slice group are performed through a third calculation formula, a domain perception weight value corresponding to each slice group is obtained.

[0164] The third calculation formula is:

[0165]

[0166] wherein, denotes the i-th slice group in the direction corresponds to a domain perception weight value, denotes one of the axial direction, the coronal direction and the sagittal direction, F DA denotes a domain perception function, denotes the i-th slice group in the direction, and softmax denotes a normalization exponential function, denotes a full connection layer, F avg denotes a 3D global pooling operation.

[0167] Further, the splicing fusion sub-module is specifically configured to:

[0168] According to a fourth calculation formula and through a response vector, a channel response of the slice group is adjusted to obtain an adjusted target slice group.

[0169] The fourth calculation formula is:

[0170]

[0171] wherein, denotes the i-th slice group in the direction corresponds to a target slice group, denotes one of the axial direction, the coronal direction and the sagittal direction, denotes the i-th slice group in the direction, F scale denotes a channel-by-channel multiplication, denotes the i-th slice group in the direction corresponds to a response vector, and σ denotes a sigmoid activation function.

[0172] Further, the splicing fusion sub-module is specifically configured to:

[0173] The target slice groups in the same direction are spliced by a fifth calculation formula to obtain a target splicing graph corresponding to each direction;

[0174] The fifth calculation formula is:

[0175]

[0176] In the formula, represents target splicing graph in the direction, G represents There are G target slice groups in the direction.

[0177] Further, the splicing and fusing sub-module is specifically configured to:

[0178] All the target splicing graphs in all directions are fused by a sixth calculation formula to obtain a fused feature graph;

[0179] The sixth calculation formula is:

[0180]

[0181] In the formula, represents the fused feature graph, represents the target splicing graph in the axial direction, represents the target splicing graph in the coronal direction, represents the target splicing graph in the sagittal direction, W CA , W θ , W φ and W g all represent convolution layers, and softmax represents a normalization exponential function.

[0182] Further, the 3D feature graph to be detected includes information of four dimensions of depth, height, width and channel number.

[0183] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described device and each unit can refer to the corresponding processes in the foregoing multi-domain lung nodule detection method embodiments, which will not be described herein.

[0184] The device provided in the foregoing embodiments can be implemented in the form of a computer program, which can run on a multi-domain lung nodule detection device as shown in Figure 4 .

[0185] The multi-domain lung nodule detection device provided in the embodiments of the present application includes a memory, a processor and a network interface connected through a system bus, the memory stores at least one instruction, the at least one instruction is loaded and executed by the processor to implement all steps or part of steps of the foregoing multi-domain lung nodule detection method.

[0186] The network interface is configured to perform network communication, such as sending the assigned task. Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0187] The processor can be a CPU, and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0188] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a video playing function, an image playing function, etc.), etc. The data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device.

[0189] The computer readable storage medium provided by the embodiments of the present application has a computer program stored thereon. When the computer program is executed by the processor, all steps or part of steps of the multi-domain lung nodule detection method are realized.

[0190] The embodiments of the present application realize all or part of the foregoing processes, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer readable storage medium, and the computer program can realize the steps of the above methods when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0191] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, a server or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program codes.

[0192] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that realizes the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that realizes the functions specified in one flow or multiple flows and / or blocks

[0193] It has to be noted that, as used herein, the terms "includes", "including", "has", "having" or "has" are intended to be open-ended terms that specifically permit the inclusion of one or more elements, steps, features, components, or the like. As used herein, the term "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. The terms "comprises", "comprising", "includes", "including", "has", "having", "contains", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, contains or includes, but does not consist of, one or more elements, steps, features, components, or the like, is and is not limited to those elements, steps, features, components, or the like. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or restricted way unless expressly so defined herein.

[0194] The foregoing detailed description has set forth various embodiments of the devices and / or methods via the use of specific terminology. However, embodiments thereof should not be construed as limited to the foregoing aspects, and the terminology used herein should not be read to limit the descriptive scope of the application. The description is intended to cover any and all modifications and equivalents, and is limited only by the claims.

Claims

1. A method for detecting multi-domain pulmonary nodules, characterized in that, Includes the following steps: The CT image to be detected is input into a preset lung nodule detection model, which includes a backbone network, a domain adaptive module, and a detection head. The domain adaptive module includes a slicing module, a projection submodule, a domain perception submodule, and a stitching and fusion submodule. The backbone network extracts features from the CT image to be detected, obtaining a 3D feature map. The slicing module segments the 3D feature map along the axial, coronal, and sagittal directions to obtain multiple slice groups corresponding to each direction; The projection submodule performs nonlinear projection on each slice group to obtain the splicing adjustment value corresponding to each slice group; The domain-aware submodule performs global pooling, full connection, and normalization on each slice group to obtain the domain-aware weight value corresponding to each slice group. Then, it combines the domain-aware weight value and the concatenation adjustment value corresponding to each slice group to obtain the response vector corresponding to each slice group. The stitching and fusion submodule adjusts the channel response of the slice group through the response vector to obtain the target slice group. The target slice groups in the same direction are stitched together to obtain the target stitched image corresponding to each direction. Based on the three-way cross attention mechanism, the target stitched images in all directions are fused to obtain the fused feature map. The detection head detects and identifies the fused feature map to output the lung nodule detection result corresponding to the CT image to be detected; The projection submodule performs nonlinear projection on each slice group to obtain the splicing adjustment value corresponding to each slice group, including: Project each slice group onto N slice grouping adapters, where N is a positive integer; According to the first calculation formula, 3D global pooling and full connection processing are performed on the slice groups on each slice group adapter to obtain N initial adjustment values ​​corresponding to each slice group; According to the second calculation formula, the N initial adjustment values ​​corresponding to each slice group are spliced ​​together to obtain the splicing adjustment value of each slice group in the corresponding direction; The first calculation formula is: In the formula, express The initial adjustment value corresponding to the i-th slice group in the direction on the j-th slice group adapter. It indicates one of the three directions: axial, cerebral, and sagittal. express The i-th slice group in the direction, Represents the squeezing and excitation functions, This indicates a 3D global pooling operation. and Indicates a fully connected layer. Represents the ReLU activation function; The second calculation formula is: In the formula, express The splicing adjustment value corresponding to the i-th slice group in the direction.

2. The multi-domain pulmonary nodule detection method as described in claim 1, characterized in that, The domain-aware submodule performs global pooling, full connection, and normalization on each slice group to obtain the domain-aware weight value corresponding to each slice group, including: The domain-aware weight value for each slice group is obtained by performing global pooling, full connection, and normalization on each slice group using the third calculation formula. The third calculation formula is: In the formula, express The domain-aware weight value corresponding to the i-th slice group in the direction. It indicates one of the three directions: axial, cerebral, and sagittal. Represents the domain-aware function. express The i-th slice group in the direction, Represents the normalized exponential function, Indicates a fully connected layer. This indicates a 3D global pooling operation.

3. The multi-domain pulmonary nodule detection method as described in claim 1, characterized in that, The splicing and fusion submodule obtains the target slice group by adjusting the channel response of the slice group through the response vector, including: The adjusted target slice group is obtained by adjusting the channel response of the slice group according to the fourth calculation formula and the response vector. The fourth calculation formula is: In the formula, express The target slice group corresponding to the i-th slice group in the direction. It indicates one of the three directions: axial, cerebral, and sagittal. express The i-th slice group in the direction, This represents channel-wise multiplication. express The response vector corresponding to the i-th slice group in the direction. This represents the sigmoid activation function.

4. The multi-domain pulmonary nodule detection method as described in claim 3, characterized in that, The step of stitching together target slices in the same direction to obtain a target mosaic image for each direction includes: The fifth calculation formula is used to stitch together the target slices in the same direction to obtain the target mosaic image corresponding to each direction; The fifth calculation formula is: In the formula, express Target mosaic in the direction, G represents There are a total of G target slice groups in the direction.

5. The multi-domain pulmonary nodule detection method as described in claim 4, characterized in that, The method based on the three-way cross-attention mechanism fuses the target stitched images from all directions to obtain a fused feature map, including: The sixth calculation formula is used to fuse the target stitching images from all directions to obtain a fused feature map; The sixth calculation formula is: In the formula, Represents the fused feature map. This represents the target mosaic along the axial direction. The image shows a target mosaic with the crown facing upwards. A mosaic of targets in the vector direction. , , and Both represent convolutional layers. This represents the normalized exponential function.

6. The multi-domain lung nodule detection method as described in claim 1, characterized in that: The 3D feature map includes information in four dimensions: depth, height, width, and number of channels.

7. A multi-domain lung nodule detection device, characterized in that, include: An image input unit and a lung nodule detection model, wherein the lung nodule detection model includes a backbone network, a domain adaptive module and a detection head, wherein the domain adaptive module includes a slicing module, a projection submodule, a domain awareness submodule and a stitching and fusion submodule; The image input unit is used to input the CT image to be detected into the lung nodule detection model; The backbone network is used to extract features from the CT image to be detected, resulting in a 3D feature map. The slicing module segments the 3D feature map along the axial, coronal, and sagittal directions to obtain multiple slice groups corresponding to each direction; The projection submodule is used to perform non-linear projection on each slice group to obtain the splicing adjustment value corresponding to each slice group; The domain-aware submodule is used to perform global pooling, full connection and normalization on each slice group to obtain the domain-aware weight value corresponding to each slice group, and to combine the domain-aware weight value and the concatenation adjustment value corresponding to each slice group to obtain the response vector corresponding to each slice group. The stitching and fusion submodule is used to adjust the channel response of the slice group through the response vector to obtain the target slice group, stitch the target slice groups in the same direction to obtain the target stitched image corresponding to each direction, and fuse the target stitched images in all directions to obtain the fused feature map. The detection head is used to detect and identify the fused feature map to output the lung nodule detection result corresponding to the CT image to be detected; Specifically, the projection submodule is used for: Project each slice group onto N slice grouping adapters, where N is a positive integer; According to the first calculation formula, 3D global pooling and full connection processing are performed on the slice groups on each slice group adapter to obtain N initial adjustment values ​​corresponding to each slice group; According to the second calculation formula, the N initial adjustment values ​​corresponding to each slice group are spliced ​​together to obtain the splicing adjustment value of each slice group in the corresponding direction; The first calculation formula is: In the formula, express The initial adjustment value corresponding to the i-th slice group in the direction on the j-th slice group adapter. It indicates one of the three directions: axial, cerebral, and sagittal. express The i-th slice group in the direction, Represents the squeezing and excitation functions, This indicates a 3D global pooling operation. and Indicates a fully connected layer. Represents the ReLU activation function; The second calculation formula is: In the formula, express The splicing adjustment value corresponding to the i-th slice group in the direction.

8. A multi-domain lung nodule detection device, characterized in that, include: A memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the multi-domain lung nodule detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multi-domain lung nodule detection method according to any one of claims 1 to 6.

Images