Image classification method based on improved MobileViT model, electronic device and readable storage medium

Abstract

The application provides an image classification method based on an improved MobileViT model, comprising the following steps: S1. An image dataset is acquired, and the image dataset is preprocessed to obtain a sample dataset; S2. An improved MobileViT model is constructed; S3. The sample dataset is input into the improved MobileViT model for training; S4. It is judged whether the improved MobileViT model is trained or not, if yes, step S5 is entered, if not, parameters are updated, and the step S3 is returned; S5. A to-be-classified image is preprocessed, and the preprocessed image is input into the trained improved MobileViT model to obtain a classification result.

Description

Technical Field

[0001] This invention relates to the field of image classification, and more particularly to an image classification method, electronic device, and storage medium based on an improved MobileViT model. Background Technology

[0002] Lattitis is a common disease in otolaryngology. The conventional diagnostic method for lattitis is to use a laryngoscopy or endoscopy to collect images of the larynx, and then the doctor makes a diagnosis based on the images. This method relies on the doctor's subjective judgment, which can lead to large diagnostic errors due to differences in doctors' experience, and it is also time-consuming.

[0003] There are few existing technologies that use neural networks to predict inflammation, and there are no neural networks that can be directly used to predict laryngitis. The only method that uses bilinear involutional neural networks to predict fundus diseases is to perform a lot of bilinear confluence calculations when using this method to predict laryngitis, which results in a heavy computational burden.

[0004] Therefore, in order to solve the above-mentioned technical problems, it is urgent to propose a new technical approach. Summary of the Invention

[0005] In view of this, in order to improve the detection accuracy of laryngitis, reduce the computational complexity of the model, and save detection time, this invention proposes an image classification method, electronic device, and readable storage medium based on an improved MobileViT model.

[0006] This invention proposes an image classification method based on an improved MobileViT model, comprising the following steps:

[0007] S1. Obtain the image dataset and preprocess it to obtain the sample dataset;

[0008] S2. Construct an improved MobileViT model;

[0009] The improved MobileViT model includes: the MobileViT model, two SE attention mechanism modules, two PPMT modules, and one CBAM attention mechanism module;

[0010] The second and third MobileViT modules in the MobileViT model were replaced with PPMT modules, the two SE attention mechanism modules were inserted into the fifth and sixth MobileNetV2 modules in the MobileViT model, and the CBAM attention mechanism module was inserted after the 1×1 convolution module in the MobileViT model.

[0011] Specifically, the input of the SE attention mechanism module is connected to the output of the depthwise separable convolution module in the MobileNetV2 module, the output of the SE attention mechanism module is connected to the input of the 1×1 convolution module in the MobileNetV2 module, the input of the CBAM attention mechanism module is connected to the output of the 1×1 convolution module, and the output of the CBAM attention mechanism module is connected to the input of the global pooling module in the MobileViT model.

[0012] S3. Input the sample dataset into the improved MobileViT model for training;

[0013] S4. Determine whether the improved MobileViT model has been trained. If yes, proceed to step S5; otherwise, update the parameters and return to step S3.

[0014] S5. Preprocess the image to be classified, and input the preprocessed image into the trained and improved MobileViT model to obtain the classification result.

[0015] Furthermore, the preprocessing steps are as follows:

[0016] The image is sequentially subjected to the following operations: increasing noise, increasing brightness, decreasing brightness, and blurring.

[0017] The blurred image is further processed using the PMT formula to obtain the preprocessed image. The PMT formula is as follows:

[0018] F final =Σ S A s (×)F s

[0019] A s =σ(W scale *F s +b scale )

[0020] Among them, F final This represents the image processed by the PMT formula, where s represents the scale and A represents the image. s F represents the attention weights at each scale. s Represents feature maps at different scales, (×) indicates element-wise multiplication, σ represents the activation function, and W scale and b scale This represents the learnable parameters.

[0021] Furthermore, the improved MobileViT model includes: the MobileViT model, two SE attention mechanism modules, two PPMT modules, and one CBAM attention mechanism module;

[0022] The second and third MobileViT modules in the MobileViT model were replaced with PPMT modules, and the two SE attention mechanism modules were inserted into the fifth and sixth MobileNetV2 modules in the MobileViT model, respectively. The CBAM attention mechanism module was inserted after the 1×1 convolution module in the MobileViT model.

[0023] Specifically, the input of the SE attention mechanism module is connected to the output of the residual module in the MobileNetV2 module, and the output of the SE attention mechanism module is connected to the input of the inverse residual module in the MobileNetV2 module; the input of the CBAM attention mechanism module is connected to the output of the 1×1 convolution module, and the output of the CBAM attention mechanism module is connected to the input of the global pooling module in the MobileViT model.

[0024] Furthermore, the PPMT module first passes the input image through a local feature extractor and a global feature extractor in sequence, then passes the feature map output by the global feature extractor through a 1×1 convolution module; finally, the feature map output by the 1×1 standard convolution module is input into an n×n convolution module.

[0025] Furthermore, the local feature extractor consists of an n×n convolution module and a 1×1 convolution module. The n×n convolution module performs local processing on the feature map and inputs the locally processed feature map into the 1×1 convolution module to change the number of channels.

[0026] The global feature extractor consists of an unfolding module, a ShuffleNet lightweight neural network, and a folding module. The unfolding module converts the feature map output by the local feature extractor into image patches, inputs the image patches into the ShuffleNet lightweight neural network, and then inputs the image patches output by the ShuffleNet lightweight neural network into the folding module. The folding module combines the image patches into the size that was input into the unfolding module.

[0027] Furthermore, training is performed using the loss function Loss, which is as follows:

[0028]

[0029] Where n represents the sample index, N represents the number of samples, I represents the number of categories, i represents the category index, and w n,i y represents the dynamic feature weight of the nth sample with respect to the i-th class. n,i This represents the true label of the i-th class corresponding to the n-th sample, logp n,i This represents the predicted probability of the nth sample for the i-th category.

[0030] Furthermore, the training of the improved MobileViT model is complete when the improved MobileViT model is trained to a preset degree of fit.

[0031] Accordingly, the present invention also provides an electronic device, comprising:

[0032] The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the image classification method based on the improved MobileViT model described above.

[0033] Accordingly, the present invention also provides a readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described image classification method based on the improved MobileViT model.

[0034] The beneficial effects of this invention are as follows: By adding a PPMT modulus to the MobileViT model, the computational complexity of the model can be reduced, enabling the model to perform more efficient inference on mobile devices and quickly obtain classification results, thereby saving diagnostic time; by adding an SE attention mechanism module to the MobileViT model, the learning ability and robustness of the model can be improved; by adding a CBAM attention mechanism module to the MobileViT model, spatial location information can be deeply explored and focused on, the modeling ability of channel features can be strengthened, and the model's ability to focus on important features can be improved, enabling the model to achieve more accurate classification judgments; thus improving the accuracy of diagnosis. Attached Figure Description

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0036] Figure 1 This is a flowchart of the present invention;

[0037] Figure 2 This is a schematic diagram of the improved MobileViT model structure of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings:

[0039] This invention proposes an image classification method based on an improved MobileViT model, comprising the following steps:

[0040] S1. Obtain the image dataset and preprocess it to obtain the sample dataset;

[0041] S2. Construct an improved MobileViT model;

[0042] The improved MobileViT model includes: the MobileViT model, two SE attention mechanism modules, two PPMT modules, and one CBAM attention mechanism module;

[0043] The second and third MobileViT modules in the MobileViT model were replaced with PPMT modules, the two SE attention mechanism modules were inserted into the fifth and sixth MobileNetV2 modules in the MobileViT model, and the CBAM attention mechanism module was inserted after the 1×1 convolution module in the MobileViT model.

[0044] Specifically, the input of the SE attention mechanism module is connected to the output of the depthwise separable convolution module in the MobileNetV2 module, the output of the SE attention mechanism module is connected to the input of the 1×1 convolution module in the MobileNetV2 module, the input of the CBAM attention mechanism module is connected to the output of the 1×1 convolution module, and the output of the CBAM attention mechanism module is connected to the input of the global pooling module in the MobileViT model.

[0045] S3. Input the sample dataset into the improved MobileViT model for training;

[0046] S4. Determine whether the improved MobileViT model has been trained. If yes, proceed to step S5; otherwise, update the parameters and return to step S3.

[0047] S5. Preprocess the image to be classified, and then input the preprocessed image into the trained improved MobileViT model to obtain the classification result. This method improves classification accuracy and reduces classification time.

[0048] MobileNet has a wide range of applications, but in medicine, it is only used for the identification of eye diseases. This application improves the MobileNet model so that it can be used to predict all types of inflammation, preferably for the prediction of laryngeal images, and for the diagnosis of laryngitis.

[0049] In this embodiment, in step S1, an image dataset is obtained and preprocessed to obtain a sample dataset;

[0050] The preprocessing steps are as follows:

[0051] The image is then subjected to the following operations in sequence: increasing noise, increasing brightness, decreasing brightness, and blurring. Figure 1Data augmentation in this context refers to adding noise, increasing brightness, reducing brightness, and blurring operations; adding noise, increasing brightness, reducing brightness, and blurring operations are all existing technologies and will not be elaborated upon here.

[0052] The blurred image is further processed using the PMT formula to obtain the preprocessed image. The PMT formula is as follows:

[0053] F final =Σ s A s (×)F s

[0054] A s =σ(W scale *F s +b scale )

[0055] Among them, F final This represents the image processed by the PMT formula, where s represents the scale and A represents the image. s F represents the attention weights at each scale. s Represents feature maps at different scales, (×) indicates element-wise multiplication, σ represents the activation function, and W scale and b scale This represents the learnable parameters. Processing with the PMT formula not only improves the model's generalization ability, robustness, and efficiency, and reduces the risk of overfitting, but also allows the model to weight and integrate features at different scales during the preprocessing stage. This mechanism helps the model better handle information at different scales and improves its ability to model multi-scale features.

[0056] In this embodiment, in step S2, an improved MobileViT model is constructed, such as... Figure 2 As shown;

[0057] The improved MobileViT model includes: the MobileViT model, two SE attention mechanism modules, two PPMT (POP MAT) modules, and one CBAM attention mechanism module;

[0058] The second and third MobileViT modules in the MobileViT model were replaced with PPMT modules, the two SE attention mechanism modules were inserted into the fifth and sixth MobileNetV2 modules in the MobileViT model, and the CBAM attention mechanism module was inserted after the 1×1 convolution module in the MobileViT model.

[0059] Specifically, the input of the SE attention mechanism module is connected to the output of the depthwise separable convolution (Dwise) module in the MobileNetV2 module. There is also a ReLU6 activation function between the depthwise separable convolution module and the SE attention mechanism module. The output of the SE attention mechanism module is connected to the input of the 1×1 convolution module in the MobileNetV2 module. The input of the CBAM attention mechanism module is connected to the output of the 1×1 convolution module. The output of the CBAM attention mechanism module is connected to the input of the global pooling module in the MobileViT model.

[0060] Adding the SE attention mechanism module to the MobileNetV2 module not only improves the network's learning ability and robustness, but also helps the model better understand and utilize the feature information in the input data; adding the CBAM attention mechanism module after the 1×1 convolution module can deeply mine and focus on spatial location information, strengthen the modeling ability of channel features, not only help improve the network's ability to focus on important features, but also effectively improve the accuracy and generalization ability of recognition.

[0061] Among them, the SE attention mechanism module, the depthwise separable convolution module, the MobileNetV2 module, the CBAM attention mechanism module, and the MobileViT module are all existing modules and will not be described in detail here;

[0062] Furthermore, the PPMT module first processes the input image sequentially through local representations and global representations. Then, the feature map output from the global representations is passed through a 1×1 convolutional module. Finally, the feature map output from the 1×1 standard convolutional module is input into an n×n convolutional module. The PPMT module is based on the MobileViT module, replacing the transformer module with a ShuffleNet lightweight neural network. The ShuffleNet lightweight neural network is an existing structure and will not be described in detail here.

[0063] Furthermore, the local feature extractor consists of an n×n convolution module and a 1×1 convolution module. The n×n convolution module performs local processing on the feature map and inputs the locally processed feature map into the 1×1 convolution module to change the number of channels.

[0064] The global feature extractor consists of an unfolding module, a ShuffleNet lightweight neural network, and a folding module. The unfolding module transforms the feature map output from the local feature extractor into image patches, which are then fed into the ShuffleNet lightweight neural network. The image patches output from the ShuffleNet lightweight neural network are then fed into the folding module, which combines the image patches to the size they would have been when input into the unfolding module. The ShuffleNet lightweight neural network employs strategies such as channel rearrangement to reduce computational complexity, making it suitable for efficient inference on mobile devices.

[0065] The Unfold module performs an unfolding operation to convert the feature map into image blocks; converting feature maps into image blocks is existing technology and will not be described in detail here. The Fold module performs a folding operation to combine image blocks into feature maps; combining image blocks into feature maps is existing technology and will not be described in detail here.

[0066] In this embodiment, in step S3, the sample dataset is input into the improved MobileViT model for training; the Adam optimizer is used for training, the batch size parameter is set to 32, the learning rate parameter is set to 0.001, and the loss function Loss is used for training, as follows:

[0067]

[0068] Where n represents the sample index, N represents the number of samples, I represents the number of categories, i represents the category index, and w n,i y represents the dynamic feature weight of the nth sample with respect to the i-th class. n,i This represents the true label of the i-th class corresponding to the n-th sample, logp n,i This represents the predicted probability of the nth sample for the i-th category. By using the loss function described above, the model can focus more on the features it deems important during the calculation process, thus better guiding the model's learning and prediction.

[0069] In this embodiment, in step S4, it is determined whether the improved MobileViT model has been trained. If yes, proceed to step S5; otherwise, update the parameters and return to step S3.

[0070] The improved MobileViT model is considered trained when it reaches a preset level of fit. This method enables more accurate model predictions.

[0071] In this embodiment, in step S6, the image to be classified is preprocessed, and the preprocessed image is input into the trained and improved MobileViT model to obtain the classification result.

[0072] Accordingly, the present invention also provides an electronic device, comprising:

[0073] The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the image classification method based on the improved MobileViT model described above.

[0074] Accordingly, the present invention also provides a readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described image classification method based on the improved MobileViT model.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An image classification method based on an improved MobileViT model, characterized in that: Includes the following steps: S1. Obtain the image dataset and preprocess it to obtain the sample dataset; S2. Construct an improved MobileViT model; The improved MobileViT model includes: the MobileViT model, two SE attention mechanism modules, two PPMT modules, and one CBAM attention mechanism module; The second and third MobileViT modules in the MobileViT model were replaced with PPMT modules, the two SE attention mechanism modules were inserted into the fifth and sixth MobileNetV2 modules in the MobileViT model, and the CBAM attention mechanism module was inserted after the 1×1 convolution module in the MobileViT model. Specifically, the input of the SE attention mechanism module is connected to the output of the depthwise separable convolution module in the MobileNetV2 module, the output of the SE attention mechanism module is connected to the input of the 1×1 convolution module in the MobileNetV2 module, the input of the CBAM attention mechanism module is connected to the output of the 1×1 convolution module, and the output of the CBAM attention mechanism module is connected to the input of the global pooling module in the MobileViT model. The PPMT module first passes the input image through a local feature extractor and a global feature extractor in sequence. Then, the feature map output by the global feature extractor is passed through a 1×1 convolutional module. Finally, the feature map output by the 1×1 standard convolutional module is input into an n×n convolutional module. The local feature extractor consists of an n×n convolutional module and a 1×1 convolutional module. The n×n convolutional module performs local processing on the feature map and inputs the locally processed feature map into the 1×1 convolutional module to change the number of channels. The global feature extractor consists of an unfolding module, a ShuffleNet lightweight neural network, and a folding module. The unfolding module converts the feature map output by the local feature extractor into image patches, inputs the image patches into the ShuffleNet lightweight neural network, and then inputs the image patches output by the ShuffleNet lightweight neural network into the folding module. The folding module combines the image patches into the size that was input into the unfolding module. S3. Input the sample dataset into the improved MobileViT model for training; S4. Determine whether the improved MobileViT model has been trained. If yes, proceed to step S5; otherwise, update the parameters and return to step S3. S5. Preprocess the image to be classified, and input the preprocessed image into the trained and improved MobileViT model to obtain the classification result.

2. The image classification method based on the improved MobileViT model according to claim 1, characterized in that: The preprocessing steps are as follows: The image is sequentially subjected to the following operations: increasing noise, increasing brightness, decreasing brightness, and blurring. The blurred image is further processed using the PMT formula to obtain the preprocessed image. The PMT formula is as follows: in, This represents the image after processing with the PMT formula, where 's' represents the scale. This represents the attention weights at each scale. Feature maps representing different scales. This indicates element-wise multiplication. This represents the activation function. and This represents the learnable parameters.

3. The image classification method based on the improved MobileViT model according to claim 1, characterized in that: Training is performed using the loss function Loss, which is as follows: Where n represents the sample index, N represents the number of samples, I represents the number of categories, and i represents the category index. This represents the dynamic feature weight of the nth sample with respect to the i-th class. This indicates that the nth sample corresponds to the true label of the i-th category. This represents the predicted probability of the nth sample for the i-th category.

4. The image classification method based on the improved MobileViT model according to claim 3, characterized in that: The training of the improved MobileViT model is complete when it reaches a preset level of fit.

5. An electronic device, characterized in that: include: A memory and a processor, the memory being used to store a computer program that, when executed by the processor, implements the image classification method based on the improved MobileViT model as described in any one of claims 1-4.

6. A readable storage medium, characterized in that: The readable storage medium stores computer instructions that, when executed by a processor, implement the image classification method based on the improved MobileViT model as described in any one of claims 1-4.

Images