An image classification method based on MobileNetV2 multi-scale feature fusion

Abstract

The application discloses a kind of image classification methods based on MobileNetV2's multi-scale feature fusion, belong to image processing and classification technical field, the method includes: acquisition template image, the multi-scale feature of template image is extracted by inputting into MobileNetV2;The average value of the multi-scale feature of template image is calculated respectively, and the corresponding category feature template is formed;The multi-scale feature of the image to be classified is input into MobileNetV2, and the feature template in the template library is compared with similarity, and the category to which the image to be classified belongs is obtained;The confidence of MobileNetV2 is judged, and the image to be classified is saved as new template image based on confidence.The method improves the accuracy of model and the efficiency of template generation, enhances the stability and distinguish degree of feature.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and classification, and particularly relates to an image classification method based on multi-scale feature fusion of MobileNetV2. BACKGROUND

[0002] With the development of computers and networks, and the popularity of large-capacity storage, more and more digital images appear, and how to quickly and accurately retrieve the desired digital image is a hot research topic today.

[0003] Traditional image classification methods rely on single features (such as color, texture or shape) or deep learning models, and the way of image classification relying on single features is sensitive to noise and has insufficient discrimination; the classification method based on multi-scale feature extraction overcomes the problem of insufficient discrimination existing in the way of image classification relying on single features, such as the image main color feature extraction method disclosed in the invention with the application publication number CN112561976A, which extracts the main color of the image to be retrieved to obtain the main color feature of the image to be retrieved; extracts the texture feature and shape feature of the image to be retrieved to obtain the texture feature and shape feature of the image to be retrieved; according to the similarity between the main color feature, texture feature and shape feature of the image to be retrieved and the image features of each candidate image in the image library, the matching degree of each candidate image in the image library is determined.

[0004] However, the above multi-scale feature extraction method still has the following problems: the deep learning model is prone to misjudgment in complex scenes or insufficient data; the feature extraction and template generation efficiency is low, and it is difficult to meet the real-time demand. Therefore, it is urgent to design an image classification method that integrates multiple features, improves robustness and takes into account efficiency. SUMMARY

[0005] The purpose of the present application is to provide an image classification method based on multi-scale feature fusion of MobileNetV2 to solve the problems of low efficiency of multi-scale image feature classification feature extraction and template generation, and misjudgment in complex scenes or insufficient data.

[0006] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0007] The present application relates to an image classification method based on multi-scale feature fusion of MobileNetV2, which comprises the following steps:

[0008] S1. Collecting template images, inputting the template images into MobileNetV2, extracting image features of different levels respectively, and obtaining multi-scale features containing color, texture and shape;

[0009] S2. Calculate the average value of the multi-scale features of the template image respectively to form the corresponding category feature template;

[0010] S3. Input the multi-scale features of the image to be classified into MobileNetV2, compare the features with the feature templates in the template library, and obtain the category to which the image to be classified belongs;

[0011] S4. Determine the confidence of MobileNetV2, and save the image to be classified as a new template image based on the confidence.

[0012] Preferably, the extraction method of the multi-scale features in S1 is: converting the template image from BGR color space to HSV color space, calculating the HS histogram, and normalizing the HS histogram to extract color features; converting the template image into a grayscale image, calculating the texture histogram using the local binary pattern algorithm, and normalizing the texture histogram to extract texture features; detecting the image edge through the Canny algorithm and calculating the improved Hu moment to extract the shape feature, and then extracting the multi-scale features of the template image.

[0013] Preferably, the formula for normalizing the HS histogram in S1 is:

[0014] ,

[0015] wherein, C is the output normalized color feature vector, H and S are the hue channel and the saturation channel respectively, I is the input template image, is the hue channel and the saturation channel of the image extracted from the BGR space to the HSV space of the template image, Hist is the calculation of the two-dimensional histogram Hist function, n and m are the binning configuration numbers of the hue channel and the saturation channel respectively, Normalize is the normalization function, D is the dimension of the output color feature vector.

[0016] Preferably, the formula for normalizing the texture histogram in S1 is:

[0017] ,

[0018] wherein, T is the output normalized texture feature vector, Q is LBP the dimension of the histogram, I is the input template image,Hist For calculating LBP the histogram Hist function, Gray ( I ) is the template image I corresponding gray image, denotes the local binary processing of the gray image using uniform pattern, P and R are the number of adjacent points and the radius configured for the local binary pattern algorithm, respectively.

[0019] Preferably, the calculation in S1 to improve the Hu moment refers to logarithmic transformation enhancement of the original Hu moment, and the expression is:

[0020] ,

[0021] wherein, is the value of the Hu moment after logarithmic enhancement, is the original Hu moment value, is the sign of the original Hu moment value, is a positive number with a value range of 10 -8 ~10 -4 , and is the logarithmic function.

[0022] Preferably, in S1, when extracting the multi-scale features of the template image, the color feature, the texture feature and the shape feature are extracted in parallel for all template images in each category directory; and in S2, the average value of the multi-scale features is calculated in parallel to form a category feature template stored as a structured JSON file.

[0023] Preferably, the specific steps of obtaining the category to which the image to be classified belongs in S3 include:

[0024] S3.1. Calculate the color similarity and the texture similarity based on the chi-square distance, and calculate the shape similarity based on the normalized Euclidean distance;

[0025] S3.2. Calculate the comprehensive similarity score based on the color similarity, the texture similarity and the shape similarity;

[0026] S3.3. Select the result with the highest similarity as the final classification result.

[0027] Preferably, the calculation formula for calculating the color similarity and the texture similarity based on the chi-square distance in S3.1 is:

[0028] ,

[0029] ,

[0030] wherein, is a color similarity, is a chi-square distance of color feature, is a value of the image to be classified in the i-th color channel, i is a value of the template image in the i-th color channel, is a texture similarity, i is a chi-square distance of texture feature, is a value of the image to be classified in the i-th texture feature value, is a value of the template image in the i-th texture feature value; j The calculation formula of shape similarity based on normalized Euclidean distance in S3.1 is: j

[0031] wherein,

[0032] is a shape similarity,

[0033] wherein, is a shape similarity, is a normalized Euclidean distance, is a shape feature vector of the image to be classified, is a shape feature vector of the template image;

[0034] The calculation formula of the comprehensive similarity score in S3.2 is:

[0035]

[0036] wherein, is a comprehensive similarity score, and are weights of color similarity, texture similarity and shape similarity respectively, and .

[0037] Preferably, the S4 judges the confidence of MobileNetV2, and the specific steps of saving the image to be classified as a new template image based on the confidence are: judging the confidence of MobileNetV2, if the confidence is not less than a set threshold, saving the classification result output by S3.3, and then adding a template image; if the confidence is less than the set threshold, weighting and fusing the model results to obtain the final classification result for saving, and then adding a template image; the formula of weighting and fusing the model results is:

[0038]

[0039] wherein,​​​​​​ as a final classification result, as a fusion weight, as an output category probability of MobileNetV2.

[0040] Compared with the prior art, the technical scheme provided by the present application has the following beneficial effects:

[0041] 1. The image classification method based on MobileNetV2 multi-scale feature fusion provided by the present application judges the confidence of MobileNetV2 after each image classification, and saves the image to be classified as a new template image based on the confidence, so as to gradually improve the confidence of MobileNetV2, increase the capacity of the template, and improve the accuracy of the model and the efficiency of template generation.

[0042] 2. The image classification method based on MobileNetV2 multi-scale feature fusion provided by the present application detects the image edge through the Canny algorithm, and calculates the improved Hu moment, that is, through logarithmic transformation and symbol retention, the stability and discrimination of the feature are enhanced.

[0043] 3. The image classification method based on MobileNetV2 multi-scale feature fusion provided by the present application extracts color features, texture features and shape features in a parallel extraction manner when generating a template, and calculates the average value of multi-scale features in a parallel calculation manner, further improving the efficiency of template generation. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the image classification method based on MobileNetV2 multi-scale feature fusion. DETAILED DESCRIPTION

[0045] In order to further understand the content of the present application, the present application will be described in detail in conjunction with the embodiments, and the following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0046] Referring to the accompanying Figure 1 The present application relates to an image classification method based on MobileNetV2 multi-scale feature fusion, which comprises the following steps:

[0047] S1. Collecting template images, inputting the template images into MobileNetV2, extracting color features, texture features and shape features in parallel for all template images in each category directory to form multi-scale features, wherein the parallel processing refers to creating a process pool equal to the number of CPU cores, using pool.imap to realize inter-class parallel processing, monitoring the processing progress through a tqdm progress bar, and the multi-scale feature extraction manner is:

[0048] The template image is converted from the BGR color space to the HSV color space. Two-dimensional histograms of the H and S channels are calculated to form the HS histogram. The HS histogram is then normalized to extract color features. The formula for normalizing the HS histogram is as follows:

[0049] ,

[0050] in, C This is the output normalized color feature vector. H and S These are the hue channel and the saturation channel, respectively. I The input template image, To extract the tone and saturation channels of the template image after converting it from BGR space to HSV space, Hist To calculate a two-dimensional histogram Hist function, n and m These represent the number of bins for the hue channel and the saturation channel, respectively. In this embodiment, the H channel is divided into 90 bins, and the S channel is divided into 128 bins. Normalize This is the normalization function. D This specifies the dimension of the output color feature vector.

[0051] The template image is converted to grayscale, and a texture histogram is calculated using the Local Binary Pattern (LBP) algorithm. A 59-dimensional normalized texture histogram is generated using a uniform mode configuration of P=8 and R=1, and texture features are extracted. The formula for normalizing the texture histogram is as follows:

[0052] ,

[0053] in, T The output is a normalized texture feature vector. Q for LBP The dimensions of a histogram I The input template image, Hist For calculation LBP histogram Hist function, Gray ( I () is the template image I The corresponding grayscale image, Indicates adoption uniform The mode performs local binarization processing on grayscale images. P and R These are the number of neighboring points and the radius configured for the local binary mode algorithm, respectively.

[0054] The image is converted to grayscale, and edge detection is performed at three scales [50, 100, 150] using the Canny algorithm. Hu moments are calculated for the edge images at each scale. The original Hu moments are then enhanced by a logarithmic transformation, thus improving the Hu moments and extracting shape features. Calculating the improved Hu moments involves enhancing the original Hu moments with a logarithmic transformation, expressed as:

[0055] ,

[0056] in, The value of Hu's moments after logarithmic augmentation. The original Hu moment value, The sign of the original Hu moment value. For values ​​in the range of 10 -8 ~10 -4 Positive numbers between It is a logarithmic function.

[0057] S2. Calculate the average value of the multi-scale features of the template image in parallel to form the corresponding category feature template; store the formed category feature template as a structured JSON file.

[0058] S3. Input the multi-scale features of the image to be classified into MobileNetV2, compare them with the feature templates in the template library, and obtain the category to which the image belongs. The specific steps for obtaining the category to which the image belongs include:

[0059] S3.1. Calculate color similarity and texture similarity based on chi-square distance, and calculate shape similarity based on normalized Euclidean distance.

[0060] The formulas for calculating color similarity and texture similarity based on chi-square distance are as follows:

[0061] ,

[0062] ,

[0063] in, For color similarity, The chi-square distance is the color feature. For the image to be classified in the first... i The values ​​on each color channel For the template image in the first i The values ​​on each color channel For texture similarity, The chi-square distance of the texture features. For the image to be classified in the first... j Each texture feature value, The template image is in the first j texture feature value;

[0064] The calculation formula of the shape similarity based on the normalized Euclidean distance is:

[0065] ,

[0066] wherein, is the shape similarity, is the normalized Euclidean distance, is the shape feature vector of the image to be classified, is the shape feature vector of the template image.

[0067] S3.2. Calculate the comprehensive similarity score based on the color similarity, texture similarity and shape similarity; the calculation formula is:

[0068] ,

[0069] wherein, is the comprehensive similarity score, , and are the weights of the color similarity, texture similarity and shape similarity respectively, and .

[0070] S3.3. Select the result with the highest similarity as the final classification result.

[0071] S4. Judge the confidence of MobileNetV2, and save the image to be classified as a new template image based on the confidence; the specific steps are: judging the confidence of MobileNetV2, and the calculation formula of the confidence is:

[0072] ,

[0073] wherein, conf represents the confidence, is the category probability vector, k is the total number of categories, and max() represents taking the maximum value of the category probability vector;

[0074] If the confidence is not less than the set threshold, save the classification result output by S3.3, and then increase a template image; if the confidence is less than the set threshold, weight and fuse the model results to obtain the final classification result for saving, and then increase a template image; the formula of the weight and fusion of the model results is:

[0075] ,

[0076] wherein, is the final classification result, is a fusion weight, is an output class probability of MobileNetV2.

[0077] The method is used for image rapid comparison and classification in practical application, for example, in the scenes of face recognition, article retrieval, industrial defect detection and the like, similarity calculation is performed on the multi-modal features extracted from the to-be-tested image and the features of various classes in the template library, and efficient, interpretable class determination and abnormality identification are realized.

[0078] The above has carried out the detailed description to the application combining the embodiment, but the content described is only the preferred embodiment of the application, cannot be considered for limiting the implementation scope of the application. All equivalent changes and improvements made according to the application scope should still belong to the patent coverage range of the application.

Claims

1. An image classification method based on MobileNetV2 multi-scale feature fusion, characterized in that: It comprises the following steps: S1. Collecting a template image, inputting the template image into MobileNetV2, extracting image features of different levels respectively to obtain multi-scale features containing color, texture and shape; S2. Calculating the average value of the multi-scale features of the template image respectively to form corresponding category feature templates; S3. Inputting the multi-scale features of the image to be classified into MobileNetV2, comparing with the feature templates in the template library to obtain the category to which the image to be classified belongs, the specific steps comprising: S3.

1. Calculating color similarity and texture similarity based on chi-square distance, and calculating shape similarity based on normalized Euclidean distance; The calculation formula for calculating color similarity and texture similarity based on chi-square distance is: ， ， in, For color similarity, The chi-square distance is the color feature. For the image to be classified in the first... i The values ​​on each color channel For the template image in the first i The values ​​on each color channel For texture similarity, The chi-square distance of the texture features. For the image to be classified in the first... j Each texture feature value, For the template image in the first j Each texture feature value; The calculation formula for calculating shape similarity based on normalized Euclidean distance is: ， wherein, is a shape similarity, is a normalized Euclidean distance, is a shape feature vector of the image to be classified, is a shape feature vector of the template image; S3.

2. Calculating comprehensive similarity score based on color similarity, texture similarity and shape similarity; The calculation formula for calculating comprehensive similarity score is: ， wherein, is a comprehensive similarity score, , and are weights for color similarity, texture similarity, and shape similarity, respectively, and ; S3.

3. Selecting the result with the highest similarity as the final classification result; S4. Judging the confidence of MobileNetV2 output and saving the image to be classified as a new template image based on the confidence.

2. The MobileNetV2 based multi-scale feature fusion image classification method according to claim 1, characterized in that: The specific way of extracting multi-scale features in S1 is: converting the template image from BGR color space to HSV color space, calculating HS histogram, and normalizing the HS histogram to extract color features; converting the template image into a grayscale image, calculating the texture histogram using the local binary pattern algorithm, and normalizing the texture histogram to extract texture features; The multi-scale features of the template image are extracted by detecting the image edge through Canny algorithm and calculating the improved Hu moment.

3. The MobileNetV2-based multi-scale feature fusion image classification method of claim 2, characterized in that: The formula for normalizing the HS histogram in S1 is: ， wherein, C is the normalized color feature vector of the output, H and S are the hue channel and the saturation channel, respectively, I is the input template image, is the hue channel and the saturation channel of the image extracted from the template image converted from BGR space to HSV space, Hist is the function for computing the two-dimensional histogram, Hist function, n and m are the binning configuration numbers of the hue channel and the saturation channel, respectively, Normalize is the normalization function, D is the dimension of the output color feature vector.

4. The MobileNetV2-based multi-scale feature fusion image classification method of claim 2, characterized in that: The formula for normalizing the texture histogram in S1 is: ， in, T The output is a normalized texture feature vector. Q for LBP The dimensions of a histogram I The input template image, Hist For calculation LBP histogram Hist function, Gray ( I () is the template image I The corresponding grayscale image, Indicates adoption uniform The mode performs local binarization processing on grayscale images. P and R These are the number of neighboring points and the radius configured for the local binary mode algorithm, respectively.

5. The MobileNetV2-based multi-scale feature fusion image classification method according to claim 2, characterized in that: The calculation of the improved Hu moment in S1 refers to the logarithmic transformation enhancement of the original Hu moment, and the expression is: ， wherein is the value of the log-enhanced Hu moment, is the original Hu moment value, is the sign of the original Hu moment value, is a positive number with a value range of 10 -8 ~ 10 -4 , and is a log function.

6. The MobileNetV2-based multi-scale feature fusion image classification method according to claim 2, characterized in that: When extracting the multi-scale features of the template image in S1, the color features, texture features and shape features of all template images in each category directory are extracted in parallel; the average value of the multi-scale features is calculated in parallel in S2, and the category feature templates formed are stored as structured JSON files.

7. The MobileNetV2-based multi-scale feature fusion image classification method of claim 1, wherein: The specific steps of judging the confidence of MobileNetV2 in S4 and saving the image to be classified as a new template image based on the confidence are: judging the confidence of MobileNetV2, if the confidence is not less than the set threshold, saving the classification result output by S3.3 to further increase a template image; if the confidence is less than the set threshold, weighting and fusing the model results to obtain the final classification result for saving, and further increasing a template image; the formula for weighting and fusing the model results is: ， wherein, is the final classification result, is the fusion weight, is the output class probability of MobileNetV2.

Images