Lightweight method, system and device for lung nodule detection network based on shift convolution

Abstract

The application relates to a lung nodule detection network lightweight method, system and equipment based on shift convolution, relates to the technical field of medical image processing, and comprises the following steps: constructing a lung nodule detection model based on an anchor-free target detection algorithm; training the lung nodule detection model based on a preprocessed target lung CT data set to generate a trained lung nodule detection model; updating and replacing standard 3D convolution operation in the trained lung nodule detection model with shift convolution operation to obtain a lightweight lung nodule detection model, wherein the shift convolution operation comprises a shift operation and 2D convolution operation; and training the lightweight lung nodule detection model based on a preprocessed self-defined lung CT data set to generate a final lightweight lung nodule detection model. Through the application, the complexity of the lung nodule detection model can be effectively reduced, the size and calculation amount of the model are greatly reduced, and the lung nodule detection network is lightened.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a lung nodule detection network lightweight method, system and equipment based on shift convolution. BACKGROUND

[0002] Cancer is a malignant disease that seriously threatens people's health, and brings great harm to people's life and society, and lung cancer is the highest incidence and mortality among all cancers, and is the main cause of cancer death in the world. With the increasing progress of social economy and the continuous development of medical imaging technology, people's health concept is also constantly enhanced, and more and more people choose regular physical examination, and the screening of lung cancer is also gradually valued. Lung cancer has no obvious characteristics in the early stage, and generally appears in the form of lung nodules. In medical images, the contrast of the lesion is not obvious, so when lung cancer is diagnosed, it is often in the middle or late stage, which is the main reason for the high mortality rate of lung cancer. Although lung cancer is a malignant disease with high incidence and mortality. However, clinical research shows that if lung cancer can be found in the early stage and treated in time, the number of patients who survive for more than 5 years will increase from 18% to 56%.

[0003] Among them, lung nodule is an important marker for measuring lung cancer in the early stage, and timely diagnosis and treatment of lung nodule is the best treatment plan for preventing lung cancer. In order to realize the early diagnosis of lung nodule, chest computed tomography (CT) is widely used and proved to be an effective tool. However, manual identification of nodules in CT images is still a time-consuming and laborious task, which requires experienced radiologists to analyze hundreds of slices each time. Therefore, it is necessary to introduce artificial intelligence deep learning to automatically and accurately identify and diagnose lung nodules. However, although current deep learning algorithms have achieved great success in lung nodule automatic detection, these algorithms often integrate models to improve accuracy; and lung CT as a 3D image, compared with natural images, its size and data are larger, and the 3D model itself is more complex, which leads to the fact that the existing lung nodule detection products in the hospital scene cannot run in time, and doctors cannot see the detection results in time.

[0004] In this context, how to realize the lightweight of the model without affecting the detection performance is particularly important. The current lightweight model direction performs outstanding algorithms including MobileNet, VoVNet, etc. Among them, MobileNet optimizes the convolution and proposes to use deep separable convolution to reduce the parameter quantity and calculation quantity of the model. VoVNet proposes an OSA (one-shot-aggregation) module, which inherits the multi-receptive field representation of DenseNet (dense convolutional network) to represent multiple features, while solving the problem of low efficiency of dense connection. The network performance is better than DenseNet, and the speed is 2 times faster than DenseNet. The above algorithms can achieve the effect of lightweight model. However, although the methods such as MobileNet and VoVNet are effective in real-time target detection tasks, they cannot be directly applied to the specific needs of medical image processing, and may not always achieve the best performance for medical image processing tasks. Therefore, how to realize the lightweight of the lung nodule detection network is a problem that needs to be solved at present. SUMMARY

[0005] The present application provides a lung nodule detection network lightweight method, system and device based on shift convolution, to solve the problem of inability to realize the lightweight of the lung nodule detection network in related technologies.

[0006] In a first aspect, a lung nodule detection network lightweight method based on shift convolution is provided, comprising the following steps:

[0007] Constructing a lung nodule detection model based on an Anchor-Free target detection algorithm;

[0008] Training the lung nodule detection model based on a preprocessed target lung CT data set to generate a trained lung nodule detection model;

[0009] Updating and replacing the standard 3D convolution operation in the trained lung nodule detection model with a shift convolution operation to obtain a lightweight lung nodule detection model, the shift convolution operation comprising a shift operation and a 2D convolution operation;

[0010] Training the lightweight lung nodule detection model based on a preprocessed custom lung CT data set to generate a final lightweight lung nodule detection model.

[0011] In some embodiments, the shift convolution operator in the lightweight lung nodule detection model is:

[0012]

[0013] In the formula, X represents the input feature, X iF represents a feature of the i-th slice of the Z-axis, F represents a convolution operation, W represents a convolution kernel parameter, and Y represents a feature obtained after convolution, represents a concatenation operator, C - represents up-shifting, C + represents down-shifting, and C represents no shifting.

[0014] In some embodiments, the shifting operation is information exchange between slices in the Z-axis direction of the lung CT image along the channel dimension, and the 2D convolution operation is feature learning on the slices after the shifting operation.

[0015] In some embodiments, the lung nodule detection model includes an encoder and a decoder, feature concatenation is performed between corresponding layers of the encoder and the decoder based on a skip connection, the encoder includes a plurality of resolution modules for generating features of different resolution levels, each resolution module includes two calculation blocks, and each calculation block includes a convolution operation, instance normalization, and a leaky rectified linear unit.

[0016] In some embodiments, before the step of training the lung nodule detection model based on the preprocessed target lung CT data set, the method further includes:

[0017] Preprocessing a plurality of public lung CT image data sets and custom lung CT image data sets to generate a preprocessed target lung CT data set.

[0018] In some embodiments, the preprocessing of the plurality of public lung CT image data sets and the custom lung CT image data sets to generate the preprocessed target lung CT data set includes:

[0019] Uniformly formatting the lung CT images in the plurality of public lung CT image data sets and the custom lung CT image data sets and screening out thin-slice lung CT images with a spacing less than a preset size;

[0020] Converting the original data of the thin-slice lung CT images into Hounsfield Unit (HU) values, and performing pixel range truncation and normalization processing on the lung region images in the thin-slice lung CT images to obtain the preprocessed target lung CT data set.

[0021] In some embodiments, the method further includes:

[0022] Labeling input images of the lung nodule detection model as positive samples, negative samples, and negligible samples to adjust the proportions of image foreground and background.

[0023] In a second aspect, a lung nodule detection network lightweight system based on shifting convolution is provided, including:

[0024] a model construction unit configured to construct a lung nodule detection model based on an Anchor-Free target detection algorithm;

[0025] a first training unit configured to train the lung nodule detection model based on the preprocessed target lung CT data set to generate a trained lung nodule detection model;

[0026] an update replacement unit configured to replace a standard 3D convolution operation in the trained lung nodule detection model with a shift convolution operation to obtain a lightweight lung nodule detection model, the shift convolution operation including a shift operation and a 2D convolution operation;

[0027] a second training unit configured to train the lightweight lung nodule detection model based on the preprocessed custom lung CT data set to generate a final lightweight lung nodule detection model.

[0028] In some embodiments, the shift convolution operator in the lightweight lung nodule detection model is:

[0029]

[0030] In the formula, X represents the input feature, X i represents the feature of the i-th slice of the Z-axis, F represents the convolution operation, W represents the convolution kernel parameter, Y represents the feature obtained after convolution, represents a concatenation operator, C - represents upward shift, C + represents downward shift, and C represents no shift.

[0031] In a third aspect, a lung nodule detection network lightweight device based on shift convolution is provided, including a memory and a processor, at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the foregoing lung nodule detection network lightweight method based on shift convolution.

[0032] This application provides a lightweight method, system, and device for lung nodule detection networks based on shifted convolution. The method includes constructing a lung nodule detection model based on an anchor-free target detection algorithm; training the lung nodule detection model on a preprocessed target lung CT dataset to generate a trained lung nodule detection model; replacing the standard 3D convolution operations in the trained lung nodule detection model with shifted convolution operations to obtain a lightweight lung nodule detection model, wherein the shifted convolution operations include shift operations and 2D convolution operations; and training the lightweight lung nodule detection model on a preprocessed custom lung CT dataset to generate the final lightweight lung nodule detection model. This application effectively reduces the complexity of the lung nodule detection model, significantly reduces the model size and computational load, and achieves lightweight lung nodule detection networks. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a lightweight method for lung nodule detection network based on shifted convolution provided in this application embodiment;

[0035] Figure 2 A schematic diagram of the overall framework of the Anchor-Free lung nodule detection model provided in the embodiments of this application;

[0036] Figure 3 A schematic diagram providing an overview of the dataset provided in this application embodiment;

[0037] Figure 4 This is an overall schematic diagram of the shift convolution operation provided in the embodiments of this application;

[0038] Figure 5 This is an overall schematic diagram of the shifting operation provided in the embodiments of this application;

[0039] Figure 6 This is a schematic diagram of the structure of a lightweight device for lung nodule detection network based on shift convolution, provided in an embodiment of this application. Detailed Implementation

[0040] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] The embodiment of the present application provides a lung nodule detection network lightweight method, system and device based on shift convolution, which can solve the problem that the lung nodule detection network lightweight cannot be realized in the related art.

[0042] Figure 1 The lung nodule detection network lightweight method based on shift convolution provided by the embodiment of the present application comprises the following steps:

[0043] Step S10: constructing a lung nodule detection model based on an anchor-free target detection algorithm;

[0044] Exemplarily, in the embodiment, a Python (an object-oriented language) platform is adopted and is implemented based on a Pytorch library (an open source Python machine learning library), and a Python medical image reading and writing function is taken as an implementation basis. For example, the Python medical image reading and writing function is called, and a medical image file name is input to be read into a matrix with a size of XxYxZ, each element in the matrix being a corresponding pixel value of each dimension, wherein Z is the number of slices of the medical image, and X and Y are the length and width of the medical image slice respectively.

[0045] It can be understood that the embodiment will construct an anchor-free lung nodule detection model with good performance on cross-domain datasets to solve the problem of poor model cross-domain performance and further improve the performance of lung nodule detection. The input of the lung nodule detection model is a 3D lung CT image, and the output is two matrices with the same size as the input, as shown in FIG. 1. Figure 2 As shown in FIG. 1, the first matrix is a predicted nodule positioning heat map generated at the center point of the lung nodule, and the position of the lung nodule can be located through the heat map; the second matrix is a predicted nodule long diameter regression map, and the predicted value at the center point of the lung nodule in the matrix represents the long diameter value of the lung nodule.

[0046] Understandingly, Anchor-Free refers to a network using keypoint estimation to find nodal center points and regress diameters, eliminating the need for anchor box design. Keypoint estimation treats the target as a single point, representing it with the center point of the target detection box, and predicts the target's center point offset and dimensions to obtain the actual object detection box. Specifically, if the object's center point exists at a certain coordinate, a keypoint (represented by a Gaussian circle) is generated at that coordinate. Figure 2 The location of the nodules is represented by a heat map.

[0047] Furthermore, the lung nodule detection model includes an encoder and a decoder. The corresponding layers of the encoder and the decoder are used for feature concatenation based on skip connections. The encoder includes multiple resolution modules for generating features at different resolution levels. Each resolution module includes two computation blocks. Each computation block includes convolution operations, instance normalization, and leaky rectified linear units.

[0048] As an example, in this embodiment, the lung nodule detection model consists of an encoder and a decoder, with skip connections used to concatenate features between corresponding layers of the encoder and decoder. In this process, the encoder provides necessary high-resolution features, and the decoder extracts high-level information. Preferably, the encoder has four different resolution levels, meaning it generates features at four different resolution levels from four different resolution modules. For details, see [link to documentation]. Figure 2 As shown, each resolution module contains two repeating computational blocks (i.e. Figure 2 The computational blocks in this model consist of four levels, each representing a computational block containing a 3×3×3 convolution operation (Conv), instance normalization (IN), and a leaky rectified linear unit (LeakyReLU). Furthermore, downsampling in this model is achieved through 2×2×2 max pooling, while upsampling is achieved through deconvolution, which also has four levels. Each level is a computational block, and each computational block contains a 3×3×3 deconvolution operation (ConvTranspose), instance normalization, and a leaky rectified linear unit.

[0049] Step S20: Train the lung nodule detection model based on the preprocessed target lung CT dataset to generate a trained lung nodule detection model;

[0050] Exemplarily, when the lung nodule detection model is trained by the preprocessed target lung CT data set, the Adam (Adaptive Moment Estimation) optimizer is used, the batch size is preferably set to 4, the initial learning rate is preferably set to 0.0001, and the learning rate is reduced to 0.00003 after 45 epochs (all training samples are processed once in the neural network, and the process is called 1 epoch). It can be understood that the lung nodule detection model needs to be trained to convergence. Specifically, when the detection performance of the validation set decreases instead of increasing, the model is considered to be converged, and the trained lung nodule detection model is generated.

[0051] In addition, after the model training is completed, the trained lung nodule detection model is also subjected to performance detection verification in the same environment as the above optimizer, batch size and initial learning rate setting. Specifically, the lung CT images in the test set are subjected to unified format processing, thin layer CT image screening, data conversion, pixel range truncation and normalization processing respectively to obtain the preprocessed test set; then the preprocessed test set is input into the trained single segment lung nodule detection model to predict the position of the possible lung nodule and the long diameter of the nodule, and compared with the true label. In the present embodiment, the evaluation index of the model detection performance is the official evaluation index of the LUNA16 data set: FROC (Free-response Receiver Operating Characteristic, Free-response Receiver Operating Characteristic curve), the horizontal coordinate of the FROC curve is the false positive rate, and the vertical coordinate is the recall rate. It can be understood that the false positive rate is also called the false positive rate, and the calculation formula is:

[0052]

[0053] The recall rate is also called the true positive rate, and the calculation formula is:

[0054]

[0055] In the formula, TP represents true positive (True Positive), that is, the center coordinates of the nodule predicted by the model fall within the radius range of a real nodule; FP represents false positive (False Positive), that is, the center coordinates of the nodule predicted by the model fall outside the radius range of any real nodule; TN represents true negative (True Negative), that is, the non-nodule center coordinates predicted by the model fall within the radius range of no lung nodule; and FN represents false negative, that is, the real nodule that is not detected by the model.

[0056] Based on the above evaluation indicators, the trained lung nodule detection model (i.e. AFNet) constructed in this embodiment is compared with the current two-stage method SANet, NoduleNet and segmentation method 3D-UNet based on Anchor-Based (a target detection algorithm). Among them, the experiments of SANet, NoduleNet and 3D-UNet are consistent with the settings of the AFNet proposed in this example. The comparison results are shown in Table 1, where FPs represents the working point, and Average FROC represents the average FROC.

[0057] Table 1 Comparison of lung nodule detection performance

[0058] Method FPs = 1 FPs = 2 FPs = 4 FPs = 8 AverageFROC 3D-Unet 45.5 63.3 74.3 80.7 65.95 NoduleNet 52.8 64.6 74.6 81.3 68.33 SANet 50.9 61.2 70.9 78.9 65.48 AFNet 51.5 64.4 76.2 85.1 69.30

[0059] From Table 1, it can be seen that the average FROC of this embodiment has a certain degree of improvement in the lung nodule detection task of multiple data sets, which is 0.97% higher than the best NoduleNet result at present.

[0060] Further, before the step of training the lung nodule detection model based on the preprocessed target lung CT data set, it further includes:

[0061] Preprocessing multiple public lung CT image data sets and self-defined lung CT image data sets to generate a preprocessed target lung CT data set.

[0062] For example, as shown in Figure 3 In this embodiment, the public lung CT image data set includes PN9, TC800, LUNA16 and RU500. Among them, PN9 is a large-scale data set proposed by Pushing Technology in cooperation with Nankai University, which contains 6708 CT images in the training set and 300 CT images in the test set in this embodiment; TC800 comes from the Tianchi Medical AI Competition, which contains 800 CT images in this embodiment; LUNA16 data set is a subset of the largest public lung nodule data set LIDC-IDRI, in which CT images with a slice thickness greater than 3mm are discarded, and LUNA16 contains 601 CT images in this embodiment; RU500 comes from the Moscow Lung Cancer Screening Data Set, which contains 450 CT images in this embodiment.

[0063] And the self-defined lung CT image data set refers to the internal data set NAP-1, which is collected from multiple hospitals and includes 2947 lung CT images in physical examination scenes and 1667 lung CT images in preoperative scenes. After preprocessing the above data sets, a preprocessed target lung CT data set can be generated.

[0064] Further, the preprocessing of the plurality of public lung CT image data sets and the self-defined lung CT image data set generates a preprocessed target lung CT data set, including:

[0065] The lung CT images in the plurality of public lung CT image data sets and the self-defined lung CT image data set are uniformly formatted and thin-slice lung CT images with a spacing less than a preset size are screened out.

[0066] The original data of the thin-slice lung CT images are converted into HU (Hounsfield) values, and the lung region images in the thin-slice lung CT images are subjected to pixel range truncation and normalization processing to obtain the preprocessed target lung CT data set.

[0067] Exemplarily, since the formats of the acquired lung CT images are generally inconsistent, in this embodiment, first, the lung CT images of different formats are uniformly formatted, such as being uniformly formatted into nii (NIFTI format extension) file format; second, CT images with a slice thickness greater than 3 mm are removed to screen out thin-slice CT images with a spacing less than 3 mm; third, the original data are converted into HU (Hounsfield) values, wherein HU refers to a standard quantitative scale for describing radioactivity density, each tissue has its specific HU range, and this range is the same for different people, so the lung CT images can be read in Python using the SimpleITK (a tool for processing medical images) interface to convert the original data into HU values; fourth, the pixel range of the lung region images is truncated using the range of [-1000, 400], that is, only the pixel values between -1000 and 400 are retained, and values less than -1000 are set to -1000 and values greater than 400 are set to 400; and finally, normalization processing is performed to linearly convert the data range to the interval [0, 1].

[0068] Further, the method further includes marking the input images of the lung nodule detection model with positive samples, negative samples and negligible samples to adjust the ratio of image foreground and background.

[0069] Exemplarily, due to the memory limitation of GPU (Graphic Processing Unit, graphic processor), in this embodiment, the lung CT images are cropped to extract small 3D images therefrom, which are individually input into the network and standardized to [0, 255]. The input 3D image block size is 128x128x128x1 (depthxheightxwidthxchannel number). If the cropping area of an image block exceeds the range of the CT image, a padding operation is performed, and the area to be padded is set to 0.

[0070] It should be noted that in the traditional method, when a CT image is cut into many small pieces for network training, only positive samples and negative samples are usually divided, but due to the large number of negative samples in lung nodule detection, the proportion of positive and negative samples is destroyed, and many negative samples are air, and the empty background is not worth all being included in the calculation, so there is a problem of data imbalance. In the embodiment, the lung CT image input into the model is regarded as a chessboard, and is marked as a positive sample, a negative sample and an ignorable sample according to the actual situation of the image, so as to adjust the proportion of image foreground and background by selectively allocating positive, negative and ignorable samples, which is beneficial to the training of the model. Among them, the positive sample represents an image block containing a nodule, the negative sample represents an image block not containing a nodule, and the ignorable sample is not learned by the network.

[0071] In addition, the traditional training method adopts a uniform sampling method, so that samples of different difficulty levels are treated equally in training, resulting in poor training effect. In order to solve this problem, the training method of difficult sample mining is introduced in the embodiment, that is, by sorting the loss of negative samples in the training process, difficult samples difficult to classify are mined and added to the training data set, thereby effectively improving the accuracy and robustness of the trained model. As can be seen, the training method based on random mask adjustment of positive and negative sample proportion and difficult sample mining is used to solve the problem of sample data imbalance, which can effectively optimize the training of the model.

[0072] It should be understood that the data imbalance problem is embodied in two aspects: on the one hand, the PN9 data set occupies a large part, each data set comes from different sources and has different characteristics, so too many PN9 data will affect the training effect; on the other hand, the size and number of nodules are also imbalanced in each data set, TC800 and LUNA16 have less than 2 nodules per scan, while PN9 has nearly 5 nodules per scan, and the number and diameter of nodules in these data sets are quite different, for example, the diameter of most PN9 nodules is 6mm-8mm, while other data sets have a large proportion of larger nodules, such as 32mm.

[0073] In the traditional target detection task, the training data is usually randomly cropped, which may result in that the training data only contains a small amount of targets, and a large amount of background will reduce the detection ability of the model to the targets. Especially for the lung nodule detection task, the lung nodule is a small target in the image, and the number of positive samples containing nodules is very small, and the number of pixel points containing nodules in the lung CT image is less than 5% of the total number of pixels in the CT image, that is, most of the image is background irrelevant to the detection task.

[0074] To solve this problem, the embodiment is based on a positive sample priority cropping method to preferentially select a region containing a target for cropping, and then adjust the ratio of image foreground and background through a random mask-based adjustment of positive and negative ratios. During training, the positive sample priority cropping method can make the model pay more attention to the image region containing the target, reduce the interference of the background, and thus improve the accuracy and robustness of target detection. Moreover, the embodiment also uses data enhancement, which generates more training samples by performing a series of random transformations on the cropped images, which can include rotation, mirror flipping, scaling, and adding Gaussian noise. These operations can simulate different scenarios, thereby increasing the diversity and quantity of training data and improving the generalization ability of the model. As can be seen, the embodiment uses a positive sample priority cropping strategy and data enhancement method to solve the problem of small target lung nodule detection.

[0075] Step S30: updating and replacing the standard 3D convolution operation in the trained lung nodule detection model with a shift convolution operation to obtain a lightweight lung nodule detection model, the shift convolution operation including a shift operation and a 2D convolution operation; wherein the shift operation is a small-range information exchange between slices in the Z-axis direction of the lung CT image along the channel dimension, and the 2D convolution operation is feature learning on the slices after the shift operation;

[0076] Exemplarily, it should be understood that traditional three-dimensional detection tasks usually use 3D convolution kernels to extract and integrate features, which can effectively capture three-dimensional spatial information and help locate the target position, but the cost of this method is to increase the parameter quantity and computational complexity, for example, the parameter quantity of a 2D convolution kernel is 9 (3x3), while the parameter quantity of a 3D convolution kernel is 27 (3x3x3). In addition, in the field of medical image analysis, the three-dimensional information learning process requires a large amount of GPU memory and large model size, however, using 2D convolution also brings some problems, for example, in the case of TLCS (thin layer CT for lung cancer screening), the distance between slices can vary from 1mm to 5mm, and the network based on 2D convolution is difficult to work effectively.

[0077] In the embodiment, shift convolution is introduced to replace the standard 3D convolution in the trained lung nodule detection model to obtain a lightweight lung nodule detection model, thereby realizing the lightweight of the lung nodule detection model. As shown in the following figure, Figure 4 The method of shift convolution in the embodiment is composed of a shift operation and a 2D convolution, wherein the shift operation exchanges information between slices in the Z-axis direction of the CT image along the channel dimension, and then a 2D convolution kernel is used for feature learning, so as to capture three-dimensional spatial information like 3D convolution.

[0078] It can be understood that the CT image is composed of a series of DICOM (Digital imaging and Communication in Medicine) slices, and using 2D convolution, although the feature map of a certain slice in the Z axis can be obtained, only the features of the current slice can be obtained, and the lesion information in the entire three-dimensional space cannot be learned. Therefore, in order to overcome this problem, referring to Figure 4 and Figure 5 It is proposed that a shift operation is performed, that is, before performing 2D convolution, the feature maps extracted from adjacent slices in the Z axis are exchanged in the channel dimension by a certain proportion, so that part of the features of the current slice are moved to the adjacent slice before the convolution operation is performed, and the information exchange is completed.

[0079] Specifically, assuming that a 128-channel input feature is given, and the shift ratio is set to 1 / 8, after the shift operation, 16 slice channel features are shifted upwards, and 16 slice channel features are shifted downwards, and then two-dimensional convolution processing is performed on each slice, and then the shift convolution is completed.

[0080] Further, the shift convolution operator in the lightweight lung nodule detection model is:

[0081]

[0082] In the formula, X represents the input feature, X i represents the feature of the i-th slice in the Z axis, F represents the convolution operation, W represents the convolution kernel parameter, Y represents the feature obtained after convolution, represents the concatenation operator, C - represents upward shift, C + represents downward shift, and C represents the feature of the current slice.

[0083] Exemplarily, the original 2D convolution operator can be written as:

[0084] Y=F(W,X i )

[0085] And after the shift operation, the convolution operator becomes:

[0086]

[0087] In the formula, X represents the input feature, X i represents the feature of the i-th slice in the Z axis, X i-1 , X i and X i+1 respectively represent three adjacent Z axis slice features, wherein X i represents the current slice feature, Xi-1 represents the upper slice feature of the current slice, X i+1 represents the lower slice feature of the current slice; F represents a convolution operation, W represents a convolution kernel parameter, Y represents a feature obtained after convolution; the symbol represents a splicing operator, and the features extracted from adjacent Z-axis slices are combined together along the channel dimension through the splicing operation; C represents a channel that needs to perform a shift operation, wherein C - represents upward shift, C + represents downward shift, and C represents no shift.

[0088] As can be seen from the above two formulas, the original 2D convolution operation can only obtain the feature of the current single slice, and after the shift operation, the 2D convolution can also be used to extract the spatial information between the upper and lower adjacent slices, such as Figure 5 As shown in the shift operation, the second slice contains information from the first slice and the third slice. As can be seen, the shift convolution method proposed in this embodiment can ingeniously combine spatial features, while maintaining the calculation cost and the number of parameters of two-dimensional convolution, and still realize approximate three-dimensional spatial feature extraction. Therefore, the shift operation in this embodiment can realize positioning the lesion position and extracting the feature in the 3D space without a large amount of calculation.

[0089] In addition, after the lightweight lung nodule detection model is constructed, the lightweight degree thereof can be quantified, such as using the parameter quantity and the calculation quantity to measure the lightweight degree. The calculation formula of the parameter quantity Params can be written as:

[0090] Params=C out ×(K h ×K d ×K w ×C in )+bias

[0091] And the calculation formula of the calculation quantity FLOPs can be written as:

[0092] FLOPs=(C in ×2×K h ×K d ×K w )×S out ×C out

[0093] In the formula, C in represents the number of input channels, C out represents the number of output channels; K h , K d , and K w respectively represent the size of the convolution kernel; bias represents a bias, and C outthe values of S out represents the size of the output feature map.

[0094] Assuming that the size of the convolution kernel is set to 3, for 3D convolution and shift convolution, C in and C out do not change, and after changing the regular 3D convolution to shift convolution, K h will be reduced from 3 to 1, and the size of the convolution kernel will change from 3x3x3 to 3x3. Therefore, compared with 3D convolution, the parameter amount and the calculation amount will be reduced to 1 / 3 by using the shift convolution method of the present embodiment. As can be seen, the present embodiment provides a more efficient and smaller calculation amount solution for medical image analysis, reduces the complexity of the lung nodule detection model, and to some extent, reduces the inference time of the model, thereby reducing the time for doctors to wait for the detection results.

[0095] Step S40: training the lightweight lung nodule detection model based on the preprocessed custom lung CT data set to generate a final lightweight lung nodule detection model.

[0096] Exemplarily, in the present embodiment, the custom lung CT data set NAP-1 will be preprocessed to obtain a preprocessed custom lung CT data set. Since the preprocessing method and principle here are the same as those in the above-mentioned embodiments, for the sake of brevity, they will not be described again. The NAP-1 includes 4614 CT images, which are collected from 2947 physical examination scenes and 1667 preoperative scenes in multiple hospitals.

[0097] Then, the Adam optimizer is used, and under the condition that the batch size is 4, the initial learning rate is set to 0.0001, and the learning rate is reduced to 0.00003 after 45 epochs, the lightweight lung nodule detection model is trained by the preprocessed custom lung CT data set until convergence, and then a final lightweight lung nodule detection model is generated. It can be understood that when the detection performance of the validation set does not increase but decreases, the model is considered to be converged. The validation set also needs to be preprocessed in the same way as described above.

[0098] After obtaining the final lightweight lung nodule detection model based on shift convolution, the present embodiment also performs performance detection and verification on the final lightweight lung nodule detection model under the same environment as the above-mentioned optimizer, batch size, and initial learning rate setting. First, the test CT image is preprocessed, and then input into the final lightweight lung nodule detection model to output the predicted lung nodule position and the predicted lung nodule long diameter, and compare them with the true label.

[0099] The shift convolution-based lightweight lung nodule detection model (i.e., AFNet+Shift) constructed in this embodiment is compared with the current classic lightweight model methods: MobileNet, VoVNet and SlimNet in performance comparison experiments. Among them, MobileNet applies depth separable convolution to compress the network; VoVNet optimizes the network structure, which is constructed by One-Shot Aggregation (OSA) and aggregates all features at once in the last feature map; SlimNet strengthens the sparsity of the channel dimension and automatically identifies and removes unimportant channels. It should be noted that the experimental settings of the above three networks are the same as the shift convolution-based lung nodule detection model of this embodiment. The comparison experiments are carried out on the NAP-1 dataset, and the evaluation method is the average FROC value of the work points 1, 2, 4 and 8, as well as the parameter amount and the calculation amount. For specific comparison experimental results, refer to Table 2.

[0100] Table 2 Performance comparison of classic lung nodule detection models

[0101]

[0102]

[0103] As can be seen from Table 2, the shift convolution-based lightweight lung nodule detection model provided in this embodiment is the only model that can improve the model performance while reducing the parameter amount and the calculation amount.

[0104] In addition, comparison experiments are carried out before and after replacing the original 3D convolution in AFNet with shift convolution. For specific results, refer to Table 3. Through the compression ratio in Table 3, it is found that after replacing the original 3D convolution in AFNet with shift convolution, the parameter amount is reduced by 66.62% and the calculation amount is reduced by 62.40% compared with before replacing the original 3D convolution in AFNet with shift convolution, and the lung nodule detection performance is improved by 3.08%.

[0105] Table 3 Comparison of effects of different networks using shift convolution

[0106]

[0107] As can be seen, replacing the original 3D convolution in AFNet with shift convolution can effectively reduce the parameter amount and the calculation amount of the model, thereby reducing the cost and improving the efficiency. In summary, compared with 3D convolution, the shift convolution is used in this embodiment, and the parameter amount and the calculation amount of the model are reduced, which to some extent alleviates the problem of large and high calculation amount of the current lung nodule detection model without reducing the model performance.

[0108] The embodiment of the application further provides a lung nodule detection network lightweight system based on shift convolution, comprising:

[0109] a model construction unit configured to construct a lung nodule detection model based on an anchor-free target detection algorithm;

[0110] a first training unit configured to train the lung nodule detection model based on a preprocessed target lung CT data set, so as to generate a trained lung nodule detection model;

[0111] an update replacement unit configured to replace standard 3D convolution operation in the trained lung nodule detection model with shift convolution operation, so as to obtain a lightweight lung nodule detection model, wherein the shift convolution operation comprises a shift operation and 2D convolution operation;

[0112] a second training unit configured to train the lightweight lung nodule detection model based on a preprocessed self-defined lung CT data set, so as to generate a final lightweight lung nodule detection model.

[0113] Further, the shift convolution operator in the lightweight lung nodule detection model is as follows:

[0114]

[0115] In the formula, X represents input features, X i represents features of the i-th slice of the Z-axis, F represents convolution operation, W represents convolution kernel parameters, Y represents features obtained after convolution, represents a splicing operator, C - represents upward shift, C + represents downward shift, and C represents no shift.

[0116] Further, the shift operation is information exchange between slices in the Z-axis direction of the lung CT image along the channel dimension, and the 2D convolution operation is feature learning on the slices after the shift operation.

[0117] Further, the lung nodule detection model comprises an encoder and a decoder, feature splicing is performed between corresponding layers of the encoder and the decoder based on a skip connection, the encoder comprises a plurality of resolution modules for generating features of different resolution levels, each resolution module comprises two calculation blocks, and each calculation block comprises convolution operation, instance normalization and leaky rectified linear unit.

[0118] Further, the system further comprises a preprocessing unit configured to:

[0119] Preprocess a plurality of public lung CT image data sets and self-defined lung CT image data sets to generate a preprocessed target lung CT data set.

[0120] Further, the preprocessing unit is specifically used for:

[0121] The lung CT images in the plurality of public lung CT image data sets and the self-defined lung CT image data sets are uniformly formatted and thin-layer lung CT images with a spacing less than a preset size are screened out.

[0122] The original data of the thin-layer lung CT images are converted into Hounsfield Unit (HU) values, and the lung region images in the thin-layer lung CT images are subjected to pixel range truncation and normalization processing to obtain the preprocessed target lung CT data set.

[0123] Further, the first training unit is further used for:

[0124] The input images of the lung nodule detection model are labeled as positive samples, negative samples and negligible samples to adjust the proportion of image foreground and background.

[0125] It should be noted that, for the convenience and brevity of description, the specific working processes of the system and each unit described above can refer to the corresponding processes in the foregoing embodiment of the lung nodule detection network lightweight method based on shift convolution, which will not be described here.

[0126] The system provided in the foregoing embodiments can be implemented in the form of a computer program, which can run on a lung nodule detection network lightweight device based on shift convolution as shown in Figure 6 .

[0127] The embodiment of the present application also provides a lung nodule detection network lightweight device based on shift convolution, which comprises a memory, a processor and a network interface connected through a system bus, at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to realize all steps or part of steps of the foregoing lung nodule detection network lightweight method based on shift convolution.

[0128] The network interface is used for network communication, such as sending an assigned task. Those skilled in the art can understand that, Figure 6 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] The processor can be a CPU, and can also be other general-purpose processors, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0130] The memory can be used to store computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a video playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, an SMC (Smart Media Card), an SD (Secure Digital) card, a flash card, at least one disk storage device, a flash memory device or other volatile solid-state memory device.

[0131] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another same element in the process, method, article or system including the element.

[0132] The foregoing detailed description of the application has been presented for purposes of illustration and description. Various modifications and changes can be made to these embodiments without departing from the spirit and scope of the application. It is intended that the scope of the application should not be limited by the particular representative embodiments described above.

Claims

1. A shift convolution-based lung nodule detection network lightweight method, characterized in that, The method comprises the following steps: constructing a lung nodule detection model based on an anchor-free target detection algorithm; training the lung nodule detection model based on a preprocessed target lung CT data set to generate a trained lung nodule detection model; updating and replacing standard 3D convolution operations in the trained lung nodule detection model with shift convolution operations to obtain a lightweight lung nodule detection model, the shift convolution operations comprising a shift operation and a 2D convolution operation; training the lightweight lung nodule detection model based on a preprocessed self-defined lung CT data set to generate a final lightweight lung nodule detection model; wherein the shift convolution operator in the lightweight lung nodule detection model is: wherein, represents an input feature, represents features of the i-th slice of the axis, represents a convolution operation, represents a convolution kernel parameter, represents a feature obtained after convolution, represents a concatenation operator, represents an up-shift, represents a down-shift, represents no shift.​ 2. The shift convolution-based lung nodule detection network lightweight method of claim 1, wherein: the shift operation is information exchange between slices in the Z-axis direction of the lung CT image along the channel dimension, and the 2D convolution operation is feature learning on the slices after the shift operation. 3.The method of claim 1, wherein: The lung nodule detection model comprises an encoder and a decoder, the corresponding layers of the encoder and the decoder are connected based on a skip connection for feature splicing, the encoder comprises a plurality of resolution modules for generating features of different resolution levels, each resolution module comprises two calculation blocks, and each calculation block comprises convolution operation, instance normalization and leaky rectified linear unit. 4.The method of claim 1, wherein, Before the step of training the lung nodule detection model based on the preprocessed target lung CT data set, the method further comprises: preprocessing a plurality of public lung CT image data sets and self-defined lung CT image data sets to generate a preprocessed target lung CT data set.

5. The shift convolution-based lung nodule detection network lightweight method of claim 4, wherein, The preprocessing of the plurality of public lung CT image data sets and the self-defined lung CT image data set to generate the preprocessed target lung CT data set comprises: performing unified format processing on the lung CT images in the plurality of public lung CT image data sets and the self-defined lung CT image data set, and screening out thin-layer lung CT images with a spacing less than a preset size; converting the original data of the thin-layer lung CT images into Hounsfield unit HU values, and performing pixel range truncation and normalization processing on the lung region images in the thin-layer lung CT images to obtain the preprocessed target lung CT data set. 6.The method of claim 1, wherein, The method further comprises: labeling positive samples, negative samples and negligible samples of the input image of the lung nodule detection model to adjust the proportion of image foreground and background.

7. A shift convolution-based lung nodule detection network lightweight system, characterized in that, It comprises: a model construction unit configured to construct a lung nodule detection model based on an anchor-free target detection algorithm; a first training unit configured to train the lung nodule detection model based on a preprocessed target lung CT data set to generate a trained lung nodule detection model; an updating and replacing unit configured to update and replace standard 3D convolution operations in the trained lung nodule detection model with shift convolution operations to obtain a lightweight lung nodule detection model, the shift convolution operations comprising a shift operation and a 2D convolution operation; a second training unit configured to train the lightweight lung nodule detection model based on the preprocessed customized lung CT dataset to generate a final lightweight lung nodule detection model; wherein the shift convolution operator in the lightweight lung nodule detection model is: In the formula, Represents the features of the input. express The first axis Features of each slice This represents the convolution operation. Represents the convolution kernel parameters. This represents the features obtained after convolution. This indicates the concatenation operator. Indicates shifting upwards. Indicates shifting down. This indicates that no shifting will occur.

8. A shift convolution based lung nodule detection network lightweight device, characterized in that, comprising: a memory and a processor, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the shift convolution based lung nodule detection network lightweight method according to any one of claims 1 to 6.

Images