Multi-scale feature fusion image classification method based on MobileNetV2

Abstract

The invention discloses an image classification method based on multi-scale feature fusion of MobileNetV2, and belongs to the technical field of image processing and classification, and the method comprises the steps: collecting a template image, inputting the template image into the MobileNetV2, and extracting multi-scale features; respectively calculating an average value of the multi-scale features of the template image to form a corresponding category feature template; inputting the multi-scale features of the to-be-classified image into the MobileNetV2, performing similarity comparison on the multi-scale features and the feature templates in the template library, and obtaining the category to which the to-be-classified image belongs; and judging the confidence coefficient of the MobileNetV2, and storing the to-be-classified image as a new template image based on the confidence coefficient. According to the method, the accuracy of the model and the template generation efficiency are improved, and the stability and the distinction degree of the features are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and classification, and specifically relates to an image classification method based on multi-scale feature fusion of MobileNetV2. Background Art

[0002] With the development of computers and networks, and the popularization of large-capacity storage, more and more digital images have appeared. How to quickly and accurately retrieve the desired digital images is a hot research topic today.

[0003] Traditional image classification methods rely on a single feature (such as color, texture or shape) or a deep learning model. The image classification method that relies on a single feature is sensitive to noise and has insufficient discrimination. The classification method based on multi-scale feature extraction overcomes the problem of insufficient discrimination in the image classification method that relies on a single feature. For example, the invention patent with application publication number CN112561976A discloses a method for extracting the main color features of an image, which extracts the main color of the image to be retrieved to obtain the main color features of the image to be retrieved; extracts the texture features and shape features of the image to be retrieved to obtain the texture features and shape features of the image to be retrieved; and determines the matching degree of each candidate image in the image library based on the similarity between the main color features, texture features and shape features of the image to be retrieved and the image features of each candidate image in the image library.

[0004] However, these multi-scale feature extraction methods still have the following problems: deep learning models are prone to misjudgment in complex scenes or when data is insufficient; feature extraction and template generation are inefficient, making it difficult to meet real-time requirements. Therefore, there is an urgent need to design an image classification method that integrates multiple features, improves robustness, and maintains efficiency. Summary of the Invention

[0005] The purpose of the present invention is to provide an image classification method based on multi-scale feature fusion of MobileNetV2 to solve the problems of low efficiency of feature extraction and template generation in multi-scale image feature classification, and easy misjudgment in complex scenes or when data is insufficient.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: The present invention relates to an image classification method based on multi-scale feature fusion of MobileNetV2, which comprises the following steps: S1. Collect a template image and input it into MobileNetV2 to extract image features at different levels, obtaining multi-scale features including color, texture, and shape. S2. Calculate the average value of the multi-scale features of the template image to form the corresponding category feature template; S3. Input the multi-scale features of the image to be classified into MobileNetV2, compare the similarity with the feature templates in the template library, and obtain the category to which the image to be classified belongs; S4. Determine the confidence level of MobileNetV2 and save the image to be classified as a new template image based on the confidence level.

[0007] Preferably, the method for extracting multi-scale features in S1 is: converting the template image from the BGR color space to the HSV color space, calculating the HS histogram, and normalizing the HS histogram to extract color features; converting the template image into a grayscale image, calculating the texture histogram using the local binary pattern algorithm, normalizing the texture histogram, and extracting texture features; detecting the image edge through the Canny algorithm, and calculating the improved Hu moment to extract shape features, and then extracting multi-scale features of the template image.

[0008] Preferably, the formula for normalizing the HS histogram in S1 is: , in, C is the output normalized color feature vector, H and S They are hue channel and saturation channel respectively. I is the input template image, To extract the hue channel and saturation channel of the template image converted from BGR space to HSV space, Hist To calculate the two-dimensional histogram Hist function, n and m are the number of bin configurations for the hue channel and the saturation channel, Normalize is the normalization function, D is the dimension of the output color feature vector.

[0009] Preferably, the formula for normalizing the texture histogram in S1 is: , in, T is the output normalized texture feature vector, Q for LBP The dimensions of the histogram, I is the input template image, Hist To calculate LBP Histogram Hist function, Gray ( I ) is the template image I The corresponding grayscale image, Indicates the use ofuniform The mode performs local binarization processing on grayscale images. P and R The number of neighbors and radius configured for the Local Binary Patterns algorithm respectively.

[0010] Preferably, the calculation of the improved Hu moment in S1 refers to performing logarithmic transformation enhancement on the original Hu moment, and the expression is: , in, is the value after logarithmic enhancement of Hu moment, is the original Hu moment value, is the sign of the original Hu moment value, The value range is 10 -8 ~10 -4 Positive numbers between is a logarithmic function.

[0011] Preferably, when extracting the multi-scale features of the template image in S1, color features, texture features and shape features are extracted in parallel for all template images in each category directory; the average value of the multi-scale features is calculated in S2 using a parallel processing method, and the resulting category feature template is stored as a structured JSON file.

[0012] Preferably, the specific step of obtaining the category to which the image to be classified belongs in S3 includes: S3.1. Calculate color similarity and texture similarity based on chi-square distance, and calculate shape similarity based on normalized Euclidean distance; S3.2. Calculate a composite similarity score based on color similarity, texture similarity, and shape similarity; S3.3. Select the result with the highest similarity as the final classification result.

[0013] Preferably, the calculation formula for calculating color similarity and texture similarity based on chi-square distance in S3.1 is: , , in, is the color similarity, is the chi-square distance of the color feature, For the image to be classified in i The values ​​on the color channels, For the template image in i The values ​​on the color channels, is the texture similarity, is the chi-square distance of texture features, For the image to be classified inj Texture feature values, For the template image in j Texture feature values; The calculation formula for calculating shape similarity based on normalized Euclidean distance in S3.1 is: , in, is the shape similarity, is the normalized Euclidean distance, is the shape feature vector of the image to be classified, is the shape feature vector of the template image; The calculation formula for the comprehensive similarity score in S3.2 is: , in, is the comprehensive similarity score, 、 and are the weights of color similarity, texture similarity, and shape similarity, respectively, and .

[0014] Preferably, the specific steps of S4 judging the confidence of MobileNetV2 and saving the image to be classified as a new template image based on the confidence are as follows: judging the confidence of MobileNetV2, if the confidence is not less than the set threshold, saving the classification result output by S3.3, and then adding a template image; if the confidence is less than the set threshold, weighting and fusing the model results to obtain the final classification result and saving it, and then adding a template image; the formula for weighted fusion of the model results is: , in, is the final classification result, is the fusion weight, is the output category probability of MobileNetV2.

[0015] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: 1. The image classification method based on multi-scale feature fusion of MobileNetV2 involved in the present invention determines the confidence of MobileNetV2 after each image classification, and saves the image to be classified as a new template image based on the confidence, so as to gradually improve the confidence of MobileNetV2 and increase the capacity of the template, thereby improving the accuracy of the model and the efficiency of template generation.

[0016] 2. The image classification method based on multi-scale feature fusion of MobileNetV2 involved in the present invention detects image edges for shape features through the Canny algorithm and calculates the improved Hu moment, that is, enhances the stability and discrimination of features through logarithmic transformation and sign preservation.

[0017] 3. The image classification method based on MobileNetV2 multi-scale feature fusion involved in the present invention uses a parallel extraction method to extract color features, texture features and shape features when generating templates, and uses a parallel calculation method to calculate the average value of multi-scale features, thereby further improving the efficiency of template generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the image classification method based on multi-scale feature fusion of MobileNetV2. DETAILED DESCRIPTION

[0019] In order to further understand the content of the present invention, the present invention is described in detail with reference to the examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0020] Refer to the attached Figure 1 As shown, the present invention relates to an image classification method based on multi-scale feature fusion of MobileNetV2, which includes the following steps: S1. Collect template images and input them into MobileNetV2. For all template images in each category, color features, texture features, and shape features are extracted in parallel to form multi-scale features. Parallel processing refers to creating a process pool equal to the number of CPU cores, using pool.imap to achieve inter-class parallel processing, and monitoring the processing progress through the tqdm progress bar. The method for extracting multi-scale features is as follows: Convert the template image from BGR color space to HSV color space, calculate the two-dimensional histogram of H channel and S channel to form HS histogram, and normalize the HS histogram to extract color features; the formula for normalizing the HS histogram is: , in, C is the output normalized color feature vector, H and S They are hue channel and saturation channel respectively. I is the input template image, To extract the hue channel and saturation channel of the template image converted from BGR space to HSV space, Hist To calculate the two-dimensional histogram Hist function, n andm are the bin configuration numbers for the hue channel and the saturation channel, respectively. In this embodiment, the H channel is divided into 90 bins and the S channel is divided into 128 bins. Normalize is the normalization function, D is the dimension of the output color feature vector.

[0021] The template image is converted into a grayscale image, and the texture histogram is calculated using the local binary pattern (LBP) algorithm. A 59-dimensional normalized texture histogram is generated using the uniform mode configuration of P=8 and R=1 to extract texture features. The formula for normalizing the texture histogram is: , in, T is the output normalized texture feature vector, Q for LBP The dimensions of the histogram, I is the input template image, Hist To calculate LBP Histogram Hist function, Gray ( I ) is the template image I The corresponding grayscale image, Indicates the use of uniform The mode performs local binarization processing on grayscale images. P and R The number of neighbors and radius configured for the Local Binary Patterns algorithm respectively.

[0022] The image is converted into a grayscale image, and edge detection is performed at three scales [50, 100, 150] using the Canny algorithm. The Hu moment is calculated for the edge image at each scale, and the original Hu moment is enhanced by logarithmic transformation to improve the Hu moment and extract shape features. Calculating the improved Hu moment means performing logarithmic transformation enhancement on the original Hu moment, and the expression is: , in, is the value after logarithmic enhancement of Hu moment, is the original Hu moment value, is the sign of the original Hu moment value, The value range is 10 -8 ~10 -4 Positive numbers between is a logarithmic function.

[0023] S2. Calculate the average values ​​of the multi-scale features of the template image using parallel processing to form a corresponding category feature template; store the generated category feature template as a structured JSON file.

[0024] S3. Input the multi-scale features of the image to be classified into MobileNetV2, compare the similarity with the feature templates in the template library, and obtain the category to which the image to be classified belongs. The specific steps for obtaining the category to which the image to be classified belongs include: S3.1. Calculate color similarity and texture similarity based on chi-square distance, and calculate shape similarity based on normalized Euclidean distance.

[0025] The calculation formula for calculating color similarity and texture similarity based on chi-square distance is: , , in, is the color similarity, is the chi-square distance of the color feature, For the image to be classified in i The values ​​on the color channels, For the template image in i The values ​​on the color channels, is the texture similarity, is the chi-square distance of texture features, For the image to be classified in j Texture feature values, For the template image in j Texture feature values; The calculation formula for calculating shape similarity based on normalized Euclidean distance is: , in, is the shape similarity, is the normalized Euclidean distance, is the shape feature vector of the image to be classified, is the shape feature vector of the template image.

[0026] S3.2. Calculate a comprehensive similarity score based on color similarity, texture similarity, and shape similarity; the calculation formula is: , in, is the comprehensive similarity score, 、 and are the weights of color similarity, texture similarity, and shape similarity, respectively, and .

[0027] S3.3. Select the result with the highest similarity as the final classification result.

[0028] S4. Determine the confidence of MobileNetV2 and save the image to be classified as a new template image based on the confidence. The specific steps are: Determine the confidence of MobileNetV2. The confidence calculation formula is: , Among them, conf represents confidence, is the class probability vector, k is the total number of categories, and max() means taking the maximum value of the category probability vector; If the confidence level is not less than the set threshold, the classification result output by S3.3 is saved, and a template image is added; if the confidence level is less than the set threshold, the model results are weighted and fused to obtain the final classification result, which is saved and a template image is added; the formula for weighted fusion of the model results is: , in, is the final classification result, is the fusion weight, is the output category probability of MobileNetV2.

[0029] This method is used in practical applications for rapid image comparison and classification. For example, in scenarios such as face recognition, object retrieval, and industrial defect detection, it calculates the similarity between the multimodal features extracted from the image to be tested and the features of each category in the template library to achieve efficient and explainable category determination and anomaly recognition.

[0030] The present invention has been described in detail above with reference to the embodiments. However, the contents described are only preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. An image classification method based on multi-scale feature fusion of MobileNetV2, characterized by: It includes the following steps: S1. Collect a template image and input it into MobileNetV2 to extract image features at different levels, obtaining multi-scale features including color, texture, and shape. S2. Calculate the average value of the multi-scale features of the template image to form the corresponding category feature template; S3. Input the multi-scale features of the image to be classified into MobileNetV2, compare the similarity with the feature templates in the template library, and obtain the category to which the image to be classified belongs; S4. Determine the confidence level of the MobileNetV2 output and save the image to be classified as a new template image based on the confidence level.

2. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 1 is characterized in that: The specific method of extracting multi-scale features in S1 is as follows: converting the template image from the BGR color space to the HSV color space, calculating the HS histogram, and normalizing the HS histogram to extract color features; converting the template image to a grayscale image, calculating the texture histogram using the local binary pattern algorithm, normalizing the texture histogram, and extracting texture features; The image edges are detected by the Canny algorithm, and the improved Hu moment is calculated to extract the shape features, and then the multi-scale features of the template image are extracted.

3. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 2 is characterized in that: The formula for normalizing the HS histogram in S1 is: ， in, C is the output normalized color feature vector, H and S They are hue channel and saturation channel respectively. I is the input template image, To extract the hue channel and saturation channel of the template image converted from BGR space to HSV space, Hist To calculate the two-dimensional histogram Hist function, n and m are the number of bin configurations for the hue channel and the saturation channel, Normalize is the normalization function, D is the dimension of the output color feature vector.

4. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 2, characterized in that: The formula for normalizing the texture histogram in S1 is: ， in, T is the output normalized texture feature vector, Q for LBP The dimensions of the histogram, I is the input template image, Hist To calculate LBP Histogram Hist function, Gray ( I ) is the template image I The corresponding grayscale image, Indicates the use of uniform The mode performs local binarization processing on grayscale images. P and R The number of neighbors and radius configured for the Local Binary Patterns algorithm respectively.

5. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 2, characterized in that: The calculation of the improved Hu moment in S1 refers to performing logarithmic transformation enhancement on the original Hu moment, and the expression is: ， in, is the value after logarithmic enhancement of Hu moment, is the original Hu moment value, is the sign of the original Hu moment value, The value range is 10 -8 ~10 -4 Positive numbers between is a logarithmic function.

6. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 2, characterized in that: When extracting the multi-scale features of the template image in S1, color features, texture features, and shape features are extracted in parallel for all template images in each category directory; the average value of the multi-scale features in S2 is calculated using a parallel processing method, and the resulting category feature template is stored as a structured JSON file.

7. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 2, characterized in that: The specific steps of obtaining the category to which the image to be classified belongs in S3 include: S3.

1. Calculate color similarity and texture similarity based on chi-square distance, and calculate shape similarity based on normalized Euclidean distance; S3.

2. Calculate a composite similarity score based on color similarity, texture similarity, and shape similarity; S3.

3. Select the result with the highest similarity as the final classification result.

8. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 7, characterized in that: The calculation formula for calculating color similarity and texture similarity based on chi-square distance in S3.1 is: ， ， in, is the color similarity, is the chi-square distance of the color feature, For the image to be classified in i The values ​​on the color channels, For the template image in i The values ​​on the color channels, is the texture similarity, is the chi-square distance of texture features, For the image to be classified in j Texture feature values, For the template image in j Texture feature values; The calculation formula for calculating shape similarity based on normalized Euclidean distance in S3.1 is: ， in, is the shape similarity, is the normalized Euclidean distance, is the shape feature vector of the image to be classified, is the shape feature vector of the template image; The calculation formula for the comprehensive similarity score in S3.2 is: ， in, is the comprehensive similarity score, 、 and are the weights of color similarity, texture similarity, and shape similarity, respectively, and .

9. The image classification method based on multi-scale feature fusion of MobileNetV2 according to claim 8, characterized in that: The specific steps of S4 judging the confidence of MobileNetV2 and saving the image to be classified as a new template image based on the confidence are as follows: judging the confidence of MobileNetV2, if the confidence is not less than the set threshold, saving the classification result output by S3.3, and then adding a template image; if the confidence is less than the set threshold, weighting and fusing the model results to obtain the final classification result and saving it, and then adding a template image; the formula for weighted fusion of model results is: ， in, is the final classification result, is the fusion weight, is the output category probability of MobileNetV2.

Images