Chest multi-class disease detection method based on lightweight neural network model

Abstract

The invention discloses a method for detecting various chest diseases based on a lightweight neural network model, and belongs to the technical field of crossing of biomedicine and information technology. According to the invention, through the designed FSConv module, the N-EffS module and the HFC module, a Mobile Vit framework is improved, a Mobile Vit-mix model is obtained, the lightweight level of the network can be maintained, deep feature extraction can be realized, local features and global features are fused, and the Mobile Vit-Mix can better capture information in an image so as to achieve lightweight and accurate detection; according to the method, a user can obtain a detection result in a short time through a simple operation process, the user is helped to know the health condition of the chest in time, potential disease risks are prevented and found in advance, and meanwhile the detection cost is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of intersectional technology of biomedicine and information technology, and in particular to a method for detecting multiple types of chest diseases based on a lightweight neural network model. Background Art

[0002] Because the traditional deep learning reasoning process requires a lot of computing resources, including high-performance computers, large amounts of storage and bandwidth, etc. Some researchers have used chest X-ray images to build a convolutional neural network (CNN) model for detecting lung diseases, and have achieved better results than the baseline model. Other researchers have proposed a method of using confidence fusion to fuse MobileNetV2 and VGG16 to achieve better classification accuracy on two currently public datasets. Some researchers have used a VGG16-based architecture that has been fine-tuned with additional custom layers, which is very useful for classification. Some researchers have used a pre-trained model VGG19 and three CNN blocks as feature extraction and fully connected networks, and experimental results show that their networks outperform other existing work. This makes the model's reasoning cost high and the detection and classification process complex. Especially in the medical field, these costs may seriously hinder the widespread application of smart medical care, and large-model deep learning medical care requires the target population to go to a designated location for testing and consumes a lot of time waiting for the results, which is time-consuming and laborious.

[0003] However, there are still some challenges. The above models are all pure lightweight convolutional models and are not combined with the currently popular Transformer architecture. Therefore, developing models that maintain low computing power and low memory requirements and take into account performance and accuracy requirements is one of the urgent problems to be solved. First of all, lightweight models can significantly reduce the number of model parameters by reducing some model accuracy; and accuracy is very important for the diagnosis and treatment of patient diseases in the medical field. Therefore, it is necessary to reduce the complexity of the model while ensuring accuracy. This is the further research and improvement required for the current deployment of lightweight models on mobile terminals. In addition, the computing power of mobile devices is limited. Although the CPU, GPU, and NPU of current mobile devices can provide powerful computing power, the computing power of mobile devices is still insufficient compared to AI models. Therefore, AI models still need to be made more lightweight so that they can be directly deployed on mobile terminals.

[0004] By using lightweight neural network models to replace traditional deep learning models, not only can information mining of key features of medical image data be achieved, but it can also be deployed on mobile terminals with a smaller number of parameters and model complexity. This can greatly improve the efficiency of patients' medical treatment under resource-constrained conditions and enable timely detection and treatment of diseases. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to solve the problems existing in the prior art such as the large computing power of medical equipment resulting in low efficiency of hospital visits, and provide a method for detecting multiple types of chest diseases based on a lightweight neural network model, which uses a module based on deep separable convolution to focus on and extract features of abnormal chest areas, and aligns attention according to their category labels; so that the characteristics of chest diseases are better matched with the labels, so as to achieve lightweight and accurate detection, which is mainly implemented in primary medical services such as rural areas.

[0006] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0007] S1: Model construction

[0008] Based on the basic architecture of the Mobile Vit framework, the Mobile Vit-mix model is constructed and trained by combining the designed FSConv module, N-EffS module, and HFC module;

[0009] S2: Compile model framework

[0010] Compile the trained Mobile Vit-mix model into a model framework that can be deployed on the server;

[0011] S3: Deployed on the server

[0012] Deploy the compiled Mobile Vit-mix model in step S2 to the server;

[0013] S4: Detection and classification of chest diseases

[0014] The patient uploads the chest X-ray medical image to the mobile terminal, and the mobile terminal transmits the chest X-ray medical image to the server through the network. The Mobile Vit-mix model in the server is used to detect and classify chest diseases in the chest X-ray medical image, and the results are sent back to the mobile terminal.

[0015] Furthermore, in the step S1, the Mobile Vit framework includes layer1, layer2, layer3, layer 4 and layer5. In the Mobile Vit-mix model, the three-layer repeated MV2 module in layer2 is replaced by the FSConv module, and the Mobile Vit Block in layer3, layer 4 and layer5 is replaced by the Mobile Vit MixBlock, wherein the local feature extraction part in the Mobile Vit Mix Block is the N-EffS module, and the Fusion part is the HFC module.

[0016] Furthermore, the FSConv module is used as an input coding layer to divide the input image into a series of image blocks, map each image block into a vector representation of a fixed dimension through linear transformation, and perform feature extraction on the input image; the N-EffS module is used to extract local features of the image; and the HFC module uses an improved efficient residual connection to perform feature fusion operations on the features.

[0017] Furthermore, the specific processing process of the FSConv module is as follows:

[0018] S101: The input feature map is processed by a 3×3 DW layer to obtain image features, which are connected to the input feature map. The expression is as follows:

[0019] F'=DW(I1)

[0020] Among them, I1 represents the image features output by layer1, that is, the input feature map, and F' represents the image features obtained by processing the 3×3 DW layer;

[0021] S102: Use two branches to jointly combine hierarchical information. In the first branch, the Squeeze layer and the expand layer in the Fire module are used to extract local information. The expression is as follows:

[0022] F1'=Fire(F')

[0023] Among them, F1' represents the feature obtained by the Fire module;

[0024] In the second branch, the group convolution module is used to divide the number of channels into groups, and the channel rearrangement module is used to shuffle the number of channels to obtain information exchange and feature fusion between different channels. The expression is as follows:

[0025] F2'=GC(F')

[0026] Among them, F2' represents the features obtained through the second branch, and GC represents the features obtained through the group convolution module and the channel rearrangement module;

[0027] S103: Perform ConCat fusion on the features obtained from the two branches. The expression is as follows:

[0028] F”=F1'+F2'

[0029] Among them, F” is the feature obtained after fusion;

[0030] S104: Then use 1×1 point-by-point convolution to restore the number of channels;

[0031] S105: The input feature map and the output after 1×1 point-by-point convolution are connected by truncation branches to promote feature fusion and multi-scale representation at different levels. The expression is as follows:

[0032] I2=I1+F”

[0033] Among them, I2 represents the image features after passing through layer2.

[0034] Furthermore, in step S102, the Fire module includes a Squeeze layer and an expand layer. The Squeeze convolution layer with a 1×1 convolution kernel is used to realize parameter sharing between channels and is input into the expand layer. The expand layer mixes 1×1 and 3×3 convolution kernels and can simultaneously extract features of different scales.

[0035] Furthermore, in the step S103, in the group convolution module, the input feature map and the convolution kernel are divided into g groups, and each group performs a convolution operation independently; in the channel rearrangement module, the grouped multiple channels are reshaped into a matrix of m rows and n columns, and then a transpose operation is performed to obtain a new matrix, which is then Flattened into a one-dimensional vector, which can promote information exchange and feature fusion between different channels and mix the information of different channels together.

[0036] Furthermore, the specific processing process of the N-EffS module is as follows:

[0037] S111: Three branches are used to form three paths. In the first branch, the standard convolution is decomposed into a 3×3 depthwise convolution and two 1×1 pointwise convolutions. The depthwise convolution processes the spatial information in the length and width directions, and the pointwise convolution processes the information in the cross-channel direction. The calculation process is as follows:

[0038] F3'1=PDP(I2)

[0039] Among them, PDP is the combined convolution of depthwise convolution and pointwise convolution, and F3'1 is the image feature obtained by the first branch;

[0040] In the second branch, 3×3 average pooling and 1×1 convolution are used for processing, and the expression is as follows:

[0041] F3'2=AG(I2)

[0042] Among them, AG is the combined convolution formed by 3×3 average pooling and 1×1 convolution, and F3'2 is the image feature obtained by the second branch;

[0043] In the third branch, 1×1 convolution is directly used for processing, and the expression is as follows:

[0044] F3'3=Conv(I2)

[0045] Among them, Conv represents 1×1 convolution, and F3'3 is the image feature obtained by the third branch;

[0046] S112: Perform ConCat fusion on the image features obtained from the three branches. The expression is as follows:

[0047] F3”=F3'1+F3'2+F3'3

[0048] Among them, F3" is the image feature obtained by fusion of three branches;

[0049] S113: The ConCat fused feature map is rearranged through the channel rearrangement module so that the channels in each group are shuffled with the channels of other groups, realizing cross-group information exchange. The expression is as follows:

[0050] I3'=CS(F3")

[0051] Among them, CS is the channel rearrangement operation, and I'3 is the image feature after passing through the N-EffS module.

[0052] Furthermore, the specific processing process of the HFC module is as follows:

[0053] S121: Input the features processed by layer n into the N-EffS module, where n is 2, 3 or 4;

[0054] S122: The output I'3 of the N-EffS module is used as a new branch to perform ConCat fusion with the output after global feature extraction, where the global feature extraction process expression is:

[0055]

[0056] Among them, Q, K and V represent query, key and value features, d k represents the dimension of the embedding;

[0057] The process expression of ConCat fusion of the output I'3 of the N-EffS module and the output after global feature extraction is:

[0058] I3"1=Conv(GR(I3'))+I3'

[0059] Among them, I3”1 represents the result of combining local features with global features, and GR represents the global features.

[0060] S123: In layer n, the input features are processed by the inverted residual structure composed of the PW layer, DW layer, and PW layer. The expression is as follows:

[0061] I3"2=PDW(I 3MV )

[0062] Among them, PDW represents the inverse residual structure composed of PW layer, DW layer, and PW layer, I 3MV is the feature after MV2 block in layer n;

[0063] S124: Perform ConCat fusion on I3”1 and I3”2 to obtain the output feature I3 of the next layer after layer n, which is expressed as follows:

[0064] I3=concat(I3"1,I3"2).

[0065] Furthermore, in the step S2, the trained Mobile Vit-mix model is converted from a Pytorch model into an ONNX format, and then the Mobile Vit-mix model is compiled into a model framework that can be deployed on a server.

[0066] Compared with the prior art, the present invention has the following advantages: the multi-category chest disease detection method based on the lightweight neural network model improves the Mobile Vit framework through the designed FSConv module, N-EffS module and HFC module to obtain the Mobile Vit-mix model, which can achieve deep feature extraction while keeping the network lightweight. By fusing local features with global features, MobileViT-Mix can better capture the information in the image to achieve lightweight and accurate detection. Through a simple operation process, the method of the present invention allows users to obtain test results in a short time, helps users to timely understand their own chest health status, prevent and discover potential disease risks in advance, and greatly reduces the detection cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is an overall framework diagram of the Mobile Vit-mix model in the first embodiment of the present invention;

[0068] FIG2( a ) is a schematic diagram of the structure of a group convolution module in Embodiment 1 of the present invention;

[0069] FIG2( b ) is a schematic diagram of the structure of a channel rearrangement module in Embodiment 1 of the present invention;

[0070] Figure 3 is a schematic diagram of the structure of the Fire module in the first embodiment of the present invention;

[0071] Figure 4 is a schematic diagram of the structure of the FSConv module in the first embodiment of the present invention;

[0072] Figure 5 Schematic diagram of the structure of the N-EffS module in the first embodiment of the present invention;

[0073] Figure 6 It is a framework diagram of the Vision Transformer in the first embodiment of the present invention;

[0074] Figure 7 It is an architecture diagram of a method for detecting multiple types of chest diseases based on a lightweight neural network model in Embodiment 1 of the present invention;

[0075] Figure 8 It is a schematic diagram of the overall process of the method for detecting multiple types of chest diseases based on a lightweight neural network model in the first embodiment of the present invention;

[0076] Fig. 9 Schematic diagram of the principle of a method for detecting multiple types of chest diseases based on a lightweight neural network model in Embodiment 1 of the present invention;

[0077] Fig.10a It is a mobile phone software user login interface corresponding to the chest multi-category disease detection method based on a lightweight neural network model in the second embodiment of the present invention;

[0078] Fig.10b It is a user information entry interface of the mobile phone software corresponding to the chest multi-category disease detection method based on a lightweight neural network model in the second embodiment of the present invention;

[0079] Fig.10c It is a user instruction interface of the mobile phone software corresponding to the method for detecting multiple types of chest diseases based on a lightweight neural network model in the second embodiment of the present invention;

[0080] Fig.10d It is the mobile phone software result viewing interface corresponding to the chest multi-category disease detection method based on the lightweight neural network model in the second embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0082] Embodiment 1

[0083] This embodiment provides a technical solution: a method for detecting multiple types of chest diseases based on a lightweight neural network model, comprising the following steps:

[0084] Step 1: Model building

[0085] Based on the basic architecture of the Mobile Vit framework, the Mobile Vit-mix model is constructed and trained by combining the designed FSConv module, N-EffS module, and HFC module;

[0086] In this step, the overall framework of the Mobile Vit-mix model is as follows Figure 1 As shown in Figure 1, in particular, it consists of three types of modules: FSConv (Squeezed Dilated Convolution and Channel Re-Following) module, N-EffS (New Efficient Channel Re-Following Convolution) module and HFC (Efficient Fusion Connection) module. Figure 1 As shown in the figure, Mobile Vit-mix adopts the basic structure of Mobile Vit framework. First, the three-layer repeated MV2 (Inverted Residual Block) structure in layer2 is replaced with the FSConv module designed by the present invention; secondly, the local feature extraction part in Mobile Vit Block in layers 3, 4, and 5 is replaced with the N-EffS module; finally, the Fusion part in Mobile Vit Block is replaced with the HFC module, which can effectively reduce the number of parameters.

[0087] As more specific, the FSConv module mainly serves as an input encoding layer, which divides the input image into a series of image blocks, maps each image block into a vector representation of a fixed dimension through linear transformation, and extracts features from the input image to obtain a low-level feature representation of each image block.

[0088] As more specific, the N-EffS module performs local feature extraction on the image, introducing more local feature information, enabling the model to better perceive the fine-grained features in the input image.

[0089] As more specific, the HFC module uses an improved efficient residual connection to fuse features, which helps optimize deeper levels in the architecture.

[0090] In the Mobile Vit-mix model, the number of output feature map channels of each layer is out1 to out5, where out1 = 16, out2 = 32, out3 = 48, out4 = 64, and out5 = 80. The design of Mobile Vit-mix reduces the number of repeated stacking of the second layer on the basis of Mobile Vit and replaces it with a module designed by us (FSConv module), which reduces the parameters and computational complexity. It integrates two lightweight core architectures with very good classification effects today; and adopts a more lightweight and effective local feature extraction module (N-EffS module) in the local feature extraction part; the fusion part (HFC module) adopts the improved efficient residual connection to fuse the original input, the features after local feature extraction, and the features after global feature extraction.

[0091] The design of Mobile Vit-mix is ​​different from other works in that: (1) it uses more skip connections, (2) it consists of a shallower network, and (3) it integrates the current excellent lightweight modules. The above design considerations reduce the parameters and computational complexity in the network. In order to maintain training accuracy and classification performance, while developing network modules, the following network performance enhancements are achieved: using more complex connection methods, adding more feature extraction modules, and enhancing the ability to extract image features; fusing input features, global features, and inputs from other locations to achieve multi-scale information fusion and enhance information interaction.

[0092] The FSConv module, N-EffS module, and HFC module are described in more detail below.

[0093] FSConv module

[0094] In this embodiment, the structures of the group convolution module and the channel rearrangement module (for performing the Channel Shuffle operation) are shown in Figures 2(a) and (b). First, group convolution can effectively reduce the capacity of the network and make the network lighter. The number of parameters and the amount of calculation of group convolution and ordinary convolution are as follows:

[0095] Ordinary convolution:

[0096]

[0097] Grouped convolution:

[0098]

[0099] From the above formula, we can see that in ordinary convolution, each convolution kernel is calculated at each position of the input feature map, so the number of parameters and the amount of calculation are proportional to the number and size of the convolution kernel. Let the number of input channels be M, each channel corresponds to 1 convolution, the size of each convolution kernel is K×K, the size of the feature map is H×W, and the number of output channels is N. In grouped convolution, the input feature map and convolution kernel are divided into g groups, and each group performs convolution operations independently. This means that the number of parameters and the amount of calculation for each group are only 1 / g of ordinary convolution. Therefore, the number of parameters and the amount of calculation for ordinary convolution are g times that of grouped convolution, that is, the number of parameters and the amount of calculation for ordinary convolution are multiples of the number of groups. Therefore, grouped convolution can reduce the number of parameters and the amount of calculation compared to ordinary convolution.

[0100] Secondly, the specific implementation process of Channel Shuffle can be seen as reshaping the grouped channels into a matrix of m rows and n columns, and then performing the transpose operation to obtain a new matrix, and then Flattening it into a one-dimensional vector. This operation can promote information exchange and feature fusion between different channels, mix the information of different channels together, thereby increasing the interaction between features and cross-channel information transmission, and helping to extract richer and more expressive features. In addition, by rearranging the channels, some redundant parameters and calculation operations can be merged together, reducing the number of parameters and calculations of the model, thereby improving the efficiency of the model.

[0101] like Figure 3 As shown in the figure, it is a schematic diagram of the structure of the Fire module. The main idea is to decompose a convolution layer into a squeeze layer and an expand layer. It uses a squeeze convolution layer (1×1 convolution kernel) to share parameters between channels and inputs into an expand layer. The expand layer mixes 1×1 and 3×3 convolution kernels to extract features of different scales at the same time. This multi-scale feature representation helps the model perceive objects and structures of different sizes, improving the model's expressiveness and generalization capabilities. The two layers are each accompanied by a Relu activation layer. The operation of these two layers greatly reduces the amount of calculation of the convolution layer, thereby achieving a very compact model structure. The kernels of the squeeze layer are all 1×1 convolution kernels, with a total of S. The expand layer contains 1×1 kernels and 3×3 convolution kernels, of which there are S1 1×1 convolution kernels and S2 3×3 convolution kernels, and it is required to satisfy S<(S1+S2).

[0102] By utilizing the advantages and characteristics of the above three modules, the FSConv module was obtained through innovation and improvement. Figure 4As shown in the figure, a 3×3 deep convolution (DW) layer is used at the beginning to connect it with the input feature map. It has a large receptive field and can capture a wider range of context information, thereby extracting richer and more expressive local features in the input data, as shown in the following formula (3):

[0103] F'=DW(I1) (3)

[0104] Among them, I1 represents the image features output by layer1, that is, the input feature map, and F' represents the image features after passing through the 3×3 deep convolution layer DW.

[0105] Previous excellent networks have demonstrated the importance of semantic context, and multi-scale information fusion is a suitable method. Therefore, we use a two-branch structure to jointly combine hierarchical information. In order to reduce the computational cost, we use the Squeeze layer and expand layer in the Fire module in the first branch to extract local information, as shown in the following formula (4):

[0106] F1'=Fire(F') (4)

[0107] Among them, F1' represents the feature obtained by the Fire module.

[0108] In the second branch, the group convolution module is used to divide the number of channels into groups, and the channel rearrangement module is used to shuffle the number of channels to obtain information exchange and feature fusion between different channels, as shown in the following formula (5):

[0109] F2'=GC(F') (5)

[0110] Among them, F2' represents the features obtained through the second branch, and GC represents the features obtained through the group convolution module and the channel rearrangement module.

[0111] We perform ConCat fusion on the two branches and then use 1×1 point-by-point convolution to restore the number of channels as follows (6):

[0112] F'=F1'+F2' (6)

[0113] Finally, we connect the input and the output after 1×1 point-by-point convolution by truncating the branches to promote feature fusion and multi-scale representation at different levels, and effectively increase the depth and complexity of the network without introducing too many additional parameters and computation, as shown in Equation 7:

[0114] I2=I1+F” (7)

[0115] Among them, I2 represents the image features after passing through layer2.

[0116] N-EffS Module

[0117] A large number of studies have shown that cascading can improve network performance and running speed to a certain extent. Through cascading, the depth and complexity of the network can be increased, thereby improving the network's representation ability. Deeper networks can learn more abstract and higher-level feature representations, which helps to improve the model's understanding and discrimination of complex data. In addition, by introducing mechanisms such as skip connections or residual connections, the flow of information and the transfer of gradients are promoted, the gradient vanishing problem is alleviated, and it helps to accelerate the training process and improve the convergence of the model. However, cascading may also increase the computational complexity and parameter volume of the network, making the model larger and more time-consuming. Therefore, convolutions with smaller parameters or lightweight convolutions are used in each path.

[0118] The structure of the N-EffS module proposed in the present invention is Figure 5 The N-EffS module is a three-pathway composed of lightweight convolutions, where the first branch uses an inverted residual structure based on depthwise separable convolutions to decompose the standard convolution into a 3×3 depthwise convolution and two 1×1 pointwise convolutions. The depthwise convolution processes spatial information in the length and width directions, and the pointwise convolution processes information in the cross-channel direction. According to formulas (8) and (9), the number of input channels is M and the size is D. F ×D F , the number of output channels is N, and the convolution kernel size is D k ×D k , its structure can reduce the amount of parameters and calculations in network transmission, and its representation process is formula 10.

[0119]

[0120] F3'1=PDP(I2) (10)

[0121] Among them, PDP is the combined convolution of depthwise convolution and point-by-point convolution, and F3'1 is the image feature obtained by the first branch.

[0122] The second branch uses 3×3 average pooling and 1×1 convolution to simplify the compression process and reduce the amount of calculation. The process is as follows:

[0123] F3'2=AG(I2) (11)

[0124] Among them, AG is a combined convolution formed by 3×3 average pooling and 1×1 convolution, and F3'2 is the image feature obtained by the second branch.

[0125] The third branch directly uses 1×1 convolution, and the process is:

[0126] F3'3=Conv(I2) (12)

[0127] Among them, F3'3 is the image feature obtained through the third branch.

[0128] Then make the three branches ConCat fusion:

[0129] F3”=F3'1+F3'2+F3'3 (13)

[0130] The N-EffS module is a series of modules integrated by lightweight convolution and cascade operations, which are used to extract low-level features and act on the local feature extraction position in the present invention. N-EffS runs in layer3 and every subsequent stage, and the above three branches together constitute the architecture of N-EffS. The configuration of N-EffS balances the channels of the input and output feature maps, avoids group convolution, and reduces the amount of calculation.

[0131] Finally, ConCat fusion is performed through the structure after the output of the three branches, and then the input feature map is rearranged through Channel Shuffle, so that the channels in each group are shuffled with the channels of other groups, thereby realizing cross-group information exchange and effectively reducing the increase in parameter quantity and computational complexity caused by multiple branches, and improving the performance of the model to a certain extent, as shown in Equation 14.

[0132] I3'=CS(F3") (14)

[0133] Among them, CS is the channel rearrangement operation, and I3' is the image feature after passing through the N-EffS module.

[0134] HFC Module

[0135] Since the basic network adopts the basic structure of Mobile Vit, a lightweight general visual transformer for mobile devices. Mobile Vit provides a different perspective for using Transformer for global information processing. The global feature extraction part uses VIT (Vision Transformer) and uses the Transformer network architecture to further extract the features of the image.

[0136] like Figure 6As shown in the figure, there are channel attention (ChannelAttention) and spatial attention (Spatial Attention) in the Vision Transformer (ViT) framework. Channel Attention generally uses global pooling operations (such as average pooling or maximum pooling) to summarize each channel to obtain a channel-level representation. Then, through one or more fully connected layers or convolutional layers, attention weights are calculated for each channel. These weights can adjust the representation of feature maps according to the importance of the channel; Spatial Attention generally uses a multi-head self-attention mechanism (Multi-head Self-Attentiono) to model the spatial dimension. Each attention head can learn different spatial relationships and generate corresponding attention weights. These two attention mechanisms complement each other and help the ViT model better understand the content in the image.

[0137] However, the biggest problem with Transformer is that it requires too much computation, because the amount of information in images is much larger than that in text in NLP. The self-attention operation will result in N-squared complexity, which is unacceptable for lightweight networks. Therefore, it is important to reduce the amount of computation.

[0138] Generally speaking, the width of the shallow layer of the network is smaller than that of the deep layer. In the deep layer of the network, more channel information can be collected, so the significance of each channel is more worthy of attention. Therefore, the local and global feature extraction and fusion parts in Mobile ViT use HFC modules for efficient fusion connection operations to achieve deep feature extraction. First, the feature input after layer2 is input to the local feature extraction part (N-EffS module), and the output I3' of the N-EffS module is used as a new branch to perform ConCat splicing with the output after global feature extraction. The global feature extraction process is:

[0139]

[0140] Among them, Q, K and V represent query, key and value features, d k Represents the dimensionality of the embedding.

[0141] The process of ConCat fusion of the output I3' of the N-EffS module and the output after global feature extraction is as follows:

[0142] I3"1=Conv(GR(I3'))+I3' (16)

[0143] Among them, I3”1 represents the result of combining local features with global features, and GR represents global features.

[0144] Then, the features after passing through the MV2 block in layer2 are input into the inverted residual structure formed by the PW, DW, and PW lightweight convolutional layers, as shown in the following formula (17):

[0145] I3"2=PDW(I 3MV ) (17)

[0146] Among them, I 3MV It is the feature after MV2 block in layer2.

[0147] After obtaining the result I3”2 after passing through the PW, DW, and PW lightweight convolution layers, I3”1 and I3”2 are fused to obtain the final output features. The process is shown in the following formula (18). Figure 1 As shown in the MobileViT-Mix block in .

[0148] I3=concat(I3”1,I3”2) (18)

[0149] According to the above design, deep feature extraction can be achieved while keeping the network lightweight. By fusing local features with global features, MobileViT-Mix can better capture the information in the image.

[0150] The calculation process of Layer4 and Layer5 is the same as that of Layer3, please refer to the above instructions.

[0151] Step 2: Compile the model framework

[0152] Convert the trained Mobile Vit-mix model from a Pytorch model to the ONNX format, and then compile the Mobile Vit-mix model into a model framework that can be deployed on the server;

[0153] In this step, after the Mobile Vit-mix model network is successfully trained through a large amount of data (hospital chest X-ray medical imaging data is collected in advance and a relevant database is established), the model is converted from a Pytorch model to the ONNX format (Open Neural Network Exchange), and then the ONNX format is converted to a model supported by ncnn, MACE, MNN, and TNN. The ONNX format is a standard for representing deep learning models that allows models to be transferred between different frameworks.

[0154] It should be noted that Pytorch is converted to ncnn, MACE, MNN or TNN through ONNX, and the library of the mobile inference framework is compiled according to iOS or Android. In order to further pursue the inference speed of the model, the inference framework can be used to fuse the convolution layer and activation layer of the model to reduce the IO overhead between network layers; float32 can also be quantized to int8, which can theoretically reduce the model size by 4 times and increase the inference speed by 2 to 3 times without reducing the accuracy of the model.

[0155] Step 3: Deploy on the server

[0156] Deploy the compiled Mobile Vit-mix model in step 2 to the server;

[0157] Step 4: Chest disease detection and classification

[0158] The patient uploads the chest X-ray medical image to the mobile terminal, and the mobile terminal transmits the chest X-ray medical image to the server through the network. The Mobile Vit-mix model in the server is used to detect and classify chest diseases in the chest X-ray medical image, and the results are sent back to the mobile terminal.

[0159] Embodiment 2

[0160] In this embodiment, a dedicated mobile phone software for the chest multi-class disease detection method in the first embodiment is designed for users. The software can be used to perform real-time detection and classification of chest X-ray medical images uploaded by patients, so that patients can obtain the results of their own chest health status more conveniently and quickly. The specific process is as follows:

[0161] Data Transfer

[0162] After professional chest X-ray collection in the hospital, the qualified collected image data will be transmitted to the server using the 5G network after judgment by the hospital collection staff and software detection, so as to facilitate subsequent server detection and classification.

[0163] Server Identification

[0164] More than 90% of the data uploaded by the hospital received by the server can be used to detect and classify multiple chest diseases through the lightweight neural network algorithm (Mobile Vit-mix model) deployed on the server, obtaining relatively fast and accurate classification results. The remaining unqualified parts need to be returned to senior doctors for re-reading to complete the accurate detection and classification of patient images.

[0165] Result Output

[0166] The chest image data taken by the hospital is detected and classified through a lightweight neural network algorithm on the server, and then the mobile network is used to return the accurate identification results to the patient's mobile terminal, so that the patient can view the chest X-ray image results in real time without having to go to the hospital, thereby achieving timely prevention and treatment of the disease.

[0167] In summary, the chest multi-category disease detection method based on the lightweight neural network model in the above-mentioned embodiment improves the Mobile Vit framework through the designed FSConv module, N-EffS module and HFC module, and obtains the MobileVit-mix model, which can realize deep feature extraction while keeping the network lightweight. By fusing local features and global features, MobileViT-Mix can better capture the information in the image to achieve lightweight and accurate detection. Through a simple operation process, the method of the present invention allows users to obtain test results in a short time, helps users to timely understand their own chest health status, prevent and discover potential disease risks in advance, and greatly reduces the detection cost.

[0168] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for detecting multiple types of chest diseases based on a lightweight neural network model, characterized in that: The following steps are involved: S1: Model construction Based on the basic architecture of the Mobile Vit framework, the Mobile Vit-mix model is constructed and trained by combining the designed FSConv module, N-EffS module, and HFC module; S2: Compile model framework Compile the trained Mobile Vit-mix model into a model framework that can be deployed on the server; S3: Deployed on the server Deploy the compiled Mobile Vit-mix model in step S2 to the server; S4: Detection and classification of chest diseases The patient uploads the chest X-ray medical image to the mobile terminal, and the mobile terminal transmits the chest X-ray medical image to the server through the network. The Mobile Vit-mix model in the server is used to detect and classify chest diseases in the chest X-ray medical image, and the results are sent back to the mobile terminal.

2. According to claim 1, a method for detecting multiple types of chest diseases based on a lightweight neural network model is characterized in that: In the step S1, the Mobile Vit framework includes layer1, layer2, layer3, layer 4 and layer5. In the Mobile Vit-mix model, the 3-layer repeated MV2 module in layer2 is replaced by the FSConv module, and the Mobile Vit Block in layer3, layer 4 and layer5 is replaced by the Mobile Vit Mix Block, wherein the local feature extraction part in the Mobile Vit Mix Block is the N-EffS module, and the Fusion part is the HFC module.

3. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 2, characterized in that: The FSConv module is used as an input coding layer to divide the input image into a series of image blocks, map each image block into a vector representation of a fixed dimension through linear transformation, and extract features from the input image; the N-EffS module is used to extract local features from the image; the HFC module uses an improved efficient residual connection to perform feature fusion operations on the features.

4. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 2, characterized in that: The specific processing process of the FSConv module is as follows: S101: The input feature map is processed by a 3×3 DW layer to obtain image features, which are connected to the input feature map. The expression is as follows: F'=DW(I1) Among them, I1 represents the image features output by layer1, that is, the input feature map, and F' represents the image features obtained by processing the 3×3 DW layer; S102: Use two branches to jointly combine hierarchical information. In the first branch, the Squeeze layer and the expand layer in the Fire module are used to extract local information. The expression is as follows: F1'=Fire(F') Among them, F1' represents the feature obtained by the Fire module; In the second branch, the group convolution module is used to divide the number of channels into groups, and the channel rearrangement module is used to shuffle the number of channels to obtain information exchange and feature fusion between different channels. The expression is as follows: F'2=GC(F') Among them, F'2 represents the features obtained through the second branch, and GC represents the features obtained through the group convolution module and the channel rearrangement module; S103: Perform ConCat fusion on the features obtained from the two branches. The expression is as follows: F'=F'1+F'2 Among them, F” is the feature obtained after fusion; S104: Then use 1×1 point-by-point convolution to restore the number of channels; S105: The input feature map and the output after 1×1 point-by-point convolution are connected by truncation branches to promote feature fusion and multi-scale representation at different levels. The expression is as follows: I2=I1+F” Among them, I2 represents the image features after passing through layer2.

5. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 4, characterized in that: In step S102, the Fire module includes a Squeeze layer and an expand layer. The Squeeze convolution layer with a 1×1 convolution kernel is used to realize parameter sharing between channels and is input into the expand layer. The expand layer mixes 1×1 and 3×3 convolution kernels and can simultaneously extract features of different scales.

6. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 5, characterized in that: In the step S103, in the group convolution module, the input feature map and the convolution kernel are divided into g groups, and each group performs a convolution operation independently; in the channel rearrangement module, the grouped multiple channels are reshaped into a matrix of m rows and n columns, and then a transpose operation is performed to obtain a new matrix, and then it is Flattened into a one-dimensional vector, which can promote information exchange and feature fusion between different channels and mix the information of different channels together.

7. A method for detecting multiple chest diseases based on a lightweight neural network model according to claim 4 or 6, characterized in that: The specific processing process of the N-EffS module is as follows: S111: Three branches are used to form three paths. In the first branch, the standard convolution is decomposed into a 3×3 depthwise convolution and two 1×1 pointwise convolutions. The depthwise convolution processes the spatial information in the length and width directions, and the pointwise convolution processes the information in the cross-channel direction. The calculation process is as follows: F’ 31 =PDP(I2) Among them, PDP is the combined convolution of depth convolution and point-by-point convolution, F' 31 is the image feature obtained by the first branch; In the second branch, 3×3 average pooling and 1×1 convolution are used for processing, and the expression is as follows: F’ 32 =AG(I2) Among them, AG is a combined convolution formed by 3×3 average pooling and 1×1 convolution, and F' 32 is the image feature obtained through the second branch; In the third branch, 1×1 convolution is directly used for processing, and the expression is as follows: F’ 33 =Conv(I2) Among them, Conv represents 1×1 convolution, F' 33 is the image feature obtained through the third branch; S112: Perform ConCat fusion on the image features obtained from the three branches. The expression is as follows: F”3=F’ 31 +F’ 32 +F’ 33 Among them, F”3 is the image feature obtained by fusion of three branches; S113: The ConCat fused feature map is rearranged through the channel rearrangement module so that the channels in each group are shuffled with the channels of other groups, realizing cross-group information exchange. The expression is as follows: I'3=CS(F"3) Among them, CS is the channel rearrangement operation, and I'3 is the image feature after passing through the N-EffS module.

8. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 7, characterized in that: The specific processing process of the HFC module is as follows: S121: Input the features processed by layer n into the N-EffS module, where n is 2, 3 or 4; S122: The output I'3 of the N-EffS module is used as a new branch to perform ConCat fusion with the output after global feature extraction, where the global feature extraction process expression is: Among them, Q, K and V represent query, key and value features, d k represents the dimension of the embedding; The process expression of ConCat fusion of the output I'3 of the N-EffS module and the output after global feature extraction is: I” 31 =Conv(GR(I’3))+I’3 Among them, I” 31 Represents the result of combining local features with global features, and GR represents global features S123: In layer n, the input features are processed by the inverted residual structure composed of the PW layer, DW layer, and PW layer. The expression is as follows: I” 32 =PDW(I 3MV ) Among them, PDW represents the inverse residual structure composed of PW layer, DW layer, and PW layer, I 3MV is the feature after MV2 block in layer n; S124: "To I" 31 and I" 32 After ConCat fusion, we get the output feature I3 of the next layer after layer n, which is expressed as follows: I3=concat(I” 31 ,I” 32 )。 9. The method for detecting multiple chest diseases based on a lightweight neural network model according to claim 1, characterized in that: In the step S2, the trained Mobile Vit-mix model is converted from a Pytorch model into an ONNX format, and then the Mobile Vit-mix model is compiled into a model framework that can be deployed on a server.

Images