Image classification identification method and system based on feature fusion

Abstract

The invention belongs to the field of image classification and recognition, and discloses an image classification and recognition method and system based on feature fusion, the system comprises an image classification and recognition module, the image classification and recognition method adopts a fusion algorithm, and an improved PANet-Swin Transformer algorithm and a DenseNet algorithm are fused to construct a final image recognition model; the background service is divided into a data management module, a model training module, a model updating and maintaining module and a background monitoring and management module and is used for providing support for model training and application; and the mobile phone APP is used for checking the identification result and providing image classification suggestions for workers. The method can assist a user in identifying images, can be widely applied to the fields of goods sorting, quality detection, medical image analysis and the like, and improves the article classification and identification quality; meanwhile, the system is low in layout cost and easy to popularize and land, and economic benefits can be practically improved.

Description

Technical Field

[0001] The present invention relates to the field of image classification and recognition, and in particular to an image classification and recognition method and system based on feature fusion. Background Art

[0002] Image classification and recognition is an important research direction in the field of image processing technology. With the rapid development of computer technology and artificial intelligence, image classification and recognition technology has been widely used in various fields, such as cargo sorting, quality inspection, and medical image analysis.

[0003] With the rise of deep learning technologies such as convolutional neural networks, image classification and recognition technology has made significant progress. By simulating the connection mode of neurons in the human brain, convolutional neural network models can automatically extract feature information from images and perform efficient classification and recognition. However, the classification accuracy and efficiency of neural network models still need to be improved. Overfitting may occur during model training, and a single network model may not be able to capture all features in the data when faced with complex data sets or problems, thus failing to achieve satisfactory classification results. Therefore, how to improve the accuracy and recognition efficiency of image classification and improve the classification accuracy of the model are technical problems that technicians in this field urgently need to solve. Summary of the Invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides an image classification and recognition method and system based on feature fusion.

[0005] To achieve the above object, according to one aspect of the present invention, a method for image classification and recognition based on feature fusion is provided, comprising the following steps:

[0006] S1 divides the data set: the initial images and their corresponding label values are divided into the initial training set, validation set and test set in a ratio of 6:3:1;

[0007] S2: Preprocessing the images in the initial training set obtained in step S1 to obtain target images, and forming a target training set based on the target images;

[0008] S3: Use the mixup linear interpolation method to expand the target training set obtained in step S2, mix the pixel values and labels of several groups of images in proportion, and generate the final training set;

[0009] S4: Constructing a fusion model: Taking the final training set generated in step S3 as input, the final image recognition model is constructed by fusing the improved PANet-SwinTransformer algorithm with the DenseNet algorithm.

[0010] S5: Input the new image collected and use the final image recognition model built in step S4 to predict the image classification result.

[0011] Preferably, the preprocessing method in step S2 includes geometric transformation data enhancement and image color transformation data enhancement;

[0012] The geometric transformation data augmentation method is as follows: first, the image is enlarged or reduced by changing its size, so that the model can adapt to input images of different sizes; then, a portion of the image is cut off to remove information irrelevant to the image boundary or generate inputs of different sizes, so that the model has better generalization ability; finally, the image is flipped horizontally or vertically, so that the model can better handle the situation where the object is flipped left and right;

[0013] The image color transformation data enhancement method is as follows: first, by increasing or decreasing the brightness of the image, images under different lighting conditions are simulated; then, the contrast of the image is adjusted to affect the degree of difference between adjacent pixel values in the image. Increasing the contrast can make the color of the image more saturated, while decreasing the contrast can make the color softer; finally, the image saturation is adjusted to make the color of the image more vivid or more gray.

[0014] Preferably, step S3 includes the following steps:

[0015] S31: For each pair of target training samples, an interpolation weight λ is randomly generated from the Beta distribution, which ranges from [0, 1].

[0016] S32: For the two target training sample tensors x1 and x2, and their corresponding label values y1 and y2, linear interpolation is performed according to the following formula:

[0017]

[0018] in, is the new sample generated, is the label of the new sample.

[0019] Preferably, step S4 includes the following steps:

[0020] S41 uses the improved PANet-Swin Transformer algorithm to train several snapshot models, and uses the prediction results output by the snapshot models as new features 1;

[0021] S42 uses the DenseNet algorithm to train several snapshot models, and uses the prediction results output by the snapshot models as new features 2;

[0022] S43 uses the bagging method to fuse new features 1 and 2, and uses the LightGBM algorithm to train to obtain the final image recognition model.

[0023] To achieve the above object, according to one aspect of the present invention, there is provided an image classification and recognition system based on feature fusion, comprising:

[0024] The image classification and recognition module uses the above-mentioned image classification and recognition method to fuse the improved PANet-SwinTransformer algorithm with the DenseNet algorithm to construct the final image recognition model;

[0025] Backend services to support model training and application;

[0026] Mobile app, used to view recognition results and provide image classification suggestions to workers.

[0027] Preferably, the improved PANet-Swin Transformer algorithm architecture consists of four parts: Swin Transformer backbone network, branch bypass, PANet network, and optimization solver.

[0028] Preferably, the improved PANet-Swin Transformer algorithm and the DenseNet algorithm are fused using a bagging method and trained using a LightGBM algorithm to obtain a final image recognition model.

[0029] Preferably, the LightGBM algorithm is a classifier of a classification system, which receives the fused new feature vector and predicts the type of the image.

[0030] Preferably, the background service includes a data management module, a model training module, a model update and maintenance module, and a background monitoring and management module. The data management module is responsible for the full-process management of image data to ensure high-quality data supply; the model training module constructs a training framework based on image classification and recognition methods, and realizes efficient model training through dynamic resource scheduling and hyperparameter automatic tuning technology; the model update and maintenance module adopts a version control mechanism and a grayscale release strategy to support model fine-tuning and realize model iterative upgrades; the background monitoring and management module ensures service stability and maintainability by collecting system operation indicators in real time.

[0031] Preferably, the mobile phone APP includes an image acquisition module, an image transmission module and a result display module to realize the application of image classification function. The image acquisition module supports multi-source image input, quickly obtains the image data to be collected, and performs preliminary preprocessing, cropping, scaling and format conversion to adapt to subsequent processing procedures; the image transmission module is responsible for uploading the processed image to the background server safely and efficiently, supports stable transmission under various network environments, and encrypts the data to ensure user privacy and data security; the result display module presents the classification and recognition results to the user in an intuitive and clear manner, supports multiple display forms, and helps users quickly apply the recognition results.

[0032] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0033] (1) By integrating different neural network models and making full use of the advantages of different models, the image classification and recognition model achieves a good balance between computational efficiency and accuracy, significantly improving the classification accuracy and robustness of the model. It can be widely used in cargo sorting, quality inspection, medical image analysis and other fields to improve the quality of item classification and recognition. At the same time, the system has low deployment cost and is easy to promote and implement, which helps to effectively improve economic benefits.

[0034] (2) Through the combination of "Internet of Things + Cloud Platform", the collected data is managed online, and an image classification and recognition model is constructed through the improved image classification and recognition method to identify image classification.

[0035] (3) Both the backend service and the mobile app integrate a series of image recognition data management functions, so that the image recognition results can be further applied to actual work and provide support for workers to classify items. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram provided by an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the image classification and recognition model construction process provided by an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the improved PANet-SwinTransformer algorithm provided by an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the DenseNet algorithm provided by an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the overall design of the image classification and recognition algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0042] like Figure 1 As shown, the present invention provides an image classification and recognition system based on feature fusion, including: an image classification and recognition module, which adopts an image classification and recognition method to fuse the improved PANet-Swin Transformer algorithm with the DenseNet algorithm to construct a final image recognition model;

[0043] Backend services to support model training and application;

[0044] Mobile app, used to view recognition results and provide image classification suggestions to workers.

[0045] In an embodiment of the present invention, the background service includes a data management module, a model training module, a model update and maintenance module, and a background monitoring and management module. The data management module is responsible for the full-process management of image data, supports multi-source data acquisition, and HDFS distributed storage intelligent labeling functions to ensure high-quality data supply; the model training module builds a parallel training framework, and realizes efficient model training through dynamic resource scheduling and hyperparameter automatic tuning technology; the model update and maintenance module adopts a version control mechanism and a grayscale release strategy, supports online incremental learning and model hot replacement, and realizes imperceptible model iteration and upgrade; the background monitoring and management module collects system operation indicators in real time, including CPU / GPU load, inference delay time, memory usage, etc., to build a multi-dimensional visual monitoring panel, and integrates anomaly detection algorithms and automatic alarm systems to ensure service stability and maintainability. Each module works together through standardized interfaces to form a closed-loop management link, significantly improving the system's automation level and engineering implementation capabilities.

[0046] In an embodiment of the present invention, the mobile phone APP includes an image acquisition module, an image transmission module and a result display module to realize the application of image classification function. The image acquisition module supports multi-source image input, including camera shooting, album import, and cloud synchronization, quickly obtains the image data that needs to be collected, and performs preliminary preprocessing, including cropping, scaling and format conversion to adapt to subsequent processing procedures; the image transmission module is responsible for uploading the processed image to the background server safely and efficiently, supports stable transmission under various network environments, and encrypts the data to ensure user privacy and data security; the result display module presents the classification and recognition results to the user in an intuitive and clear manner, supports multiple display forms, including text descriptions, label classifications and related image examples, etc., to help users quickly apply the recognition results.

[0047] In the embodiment of the present invention, the improved PANet-SwinTransformer algorithm architecture consists of four parts: SwinTransformer backbone network, branch bypass, PANet network, and optimization solver. Figure 3 shown.

[0048] In an embodiment of the present invention, the SwinTransformer backbone network is divided into four stages, wherein the PatchEmbedding module divides the input image into fixed-size patches, thereby converting the image into a serialized format suitable for processing. The PatchMerging module downsamples the feature map at the beginning of each stage, reducing the resolution while expanding the receptive field. The SwinTransformerBlock is used to extract features and perform self-attention calculations.

[0049] In the embodiment of the present invention, the branch bypass is a branch network introduced on the basis of the backbone network to make up for the shortcomings of the backbone network PatchMerging module. The branch passes through the PixelShuffle module with a sampling rate of 2 to improve the resolution, and then passes through the DepthwiseConv module. The output of the DepthwiseConv module in the previous stage is used as the input of the DepthwiseConv module in the next stage, and the PatchEmbedding module is used to reshape the dimension, and then the branch passes through the linear scaling module L i Then it is integrated into the backbone network.

[0050] In the embodiment of the present invention, the output of the first three stages of the SwinTransformer backbone network is fed into the PANet network after passing through the custom valves. Each valve is represented by a valve variable k. i (i=1,2,3) control, the valve variable controls the weight when the feature map is imported.

[0051] In an embodiment of the present invention, the PANet network is composed of FPN and PAN, and receives feature maps of different levels output by the backbone network after four stages of forward propagation as input. FPN samples from top to bottom and horizontally connects the feature maps generated from bottom to top by PAN for fusion. Each fusion result is then convolved to eliminate the aliasing effect in the sampling. Finally, after aggregation, the output is obtained through the fully connected layer.

[0052] In the embodiment of the present invention, the PatchEmbedding module transforms the image x∈R h*w*t (height h, width w, number of channels t) into small feature maps of size p×p, and these feature map blocks are linearly mapped into two-dimensional feature blocks for subsequent Transformer processing, where x'∈R n*c , The PatchMerging module concatenates each 2×2 patch in the channel dimension, quadrupling the number of channels to form a new feature map. The PixelShuffle module then decomposes the number of channels C of the input feature map into C' = C / r 2 , and move the r×r small blocks to the spatial dimensions respectively to form a feature map of size (C', rH, rW), where r is 2, thereby converting the low-resolution map into a high-resolution map.

[0053] In the embodiment of the present invention, the optimization solver modifies the hyperparameter k i The value of (i=1,2,3) adjusts the network to achieve the optimal value. In this example, the gradient descent method is used to iteratively solve the hyperparameters. First, the objective function L is the mean square error of the neural network; the random initial value k is set. i (0) , set the learning rate α; in each iteration, calculate the network loss function L with respect to the hyperparameter k i The gradient of Formula iteration, where is the parameter value of the tth iteration; repeat the above process until the update step size is less than the threshold Or the loss function is less than the threshold Where ∈ is a small positive number, the optimal solution is obtained and optimal networks.

[0054] The DenseNet algorithm, such as Figure 4As shown in the figure, the DenseNet network structure consists of several key parts. First, the input image passes through a 7×7 convolutional layer with a stride of 2 to extract the initial features, and then is downsampled through a 3×3 maximum pooling layer. Next, the network consists of four dense blocks, each of which contains multiple convolutional layers, which are densely connected. The four dense blocks contain 6, 12, 24, and 16 bottles, respectively. The output feature map of each bottleneck is concatenated with the output feature maps of all previous bottleneck layers as the input of the current layer. Between dense blocks, a transition layer consisting of a 1×1 convolutional layer and a 2×2 average pooling layer is introduced. After that, the network uses a global average pooling layer to average pool each channel of the feature map to obtain a fixed-size feature vector. Finally, a fully connected layer maps the feature vector to the category label to complete the classification task.

[0055] DenseNet's dense connection method ensures that the input of each layer contains not only the output of the previous layer, but also the output of all previous layers. This connection method improves the reusability of features. DenseNet121 has high parameter efficiency. Since features are reused in the network, this parameter efficiency not only reduces the storage overhead of the model, but also reduces computational complexity. DenseNet121 performs well in suppressing overfitting.

[0056] The improved PANet-Swin Transformer algorithm and the DenseNet algorithm are fused using the bagging method and trained using the LightGBM algorithm to obtain the final image recognition model. The LightGBM algorithm is a classifier of the classification system, which receives the fused new feature vector and predicts the image type.

[0057] First, the gradient of all data points is calculated, and the samples with the largest absolute gradient values are retained. At the same time, samples with low gradients are randomly sampled from the remaining samples. The weight coefficient is used to compensate for the low gradient samples after sampling to ensure that the data distribution is unbiased.

[0058]

[0059] in, are the first-order and second-order gradients of the loss function.

[0060] Then, the features are grouped by the graph coloring algorithm, and each group of features is bound into a new feature. The formula is expressed as:

[0061]

[0062] Where offsetf It is an artificial offset to ensure that the original feature values have no conflicts after bundling.

[0063] Traverse all features and bucket cutoff points, and select the optimal split point based on the gain formula.

[0064] The gain is:

[0065]

[0066] Where L / R represents the left and right child nodes after the split, and λ is the regularization coefficient. Expanding only the split node with the largest gain, rather than expanding layer by layer, improves efficiency and reduces memory consumption.

[0067] The final model is an additive combination of T decision trees, and the prediction result is the sum of the outputs of each tree:

[0068]

[0069] In an embodiment of the present invention, the improved image classification and recognition method is performed according to the following steps: Figure 2 As shown:

[0070] S1 divides the data set: the initial images and their corresponding label values are divided into the initial training set, validation set and test set in a ratio of 6:3:1;

[0071] Training set: used for training in each epoch of the training process to make the model more capable of recognizing pictures.

[0072] Validation set: used in each epoch of the training process to monitor the effect of this epoch and to determine whether to stop training or whether overfitting is occurring.

[0073] The test set is a part of the reserved data set that does not participate in training. It is mainly used to evaluate the effect of the entire model and to show that your own model is excellent or better than other models in terms of accuracy. However, the test set is not used to evaluate the model effect in this application.

[0074] The above are all common knowledge in the field of neural networks and should not be explained in detail.

[0075] S2: Preprocessing the images in the initial training set obtained in step S1 to obtain target images, and forming a target training set based on the target images;

[0076] The preprocessing method in step S2 includes geometric transformation data enhancement and image color transformation data enhancement;

[0077] The geometric transformation data augmentation method is as follows: first, the image is enlarged or reduced by changing its size, so that the model can adapt to input images of different sizes; then, a portion of the image is cut off to remove information irrelevant to the image boundary or generate inputs of different sizes, so that the model has better generalization ability; finally, the image is flipped horizontally or vertically, so that the model can better handle the situation where the object is flipped left and right;

[0078] The image color transformation data enhancement method is as follows: first, by increasing or decreasing the brightness of the image, images under different lighting conditions are simulated; then, the contrast of the image is adjusted to affect the degree of difference between adjacent pixel values in the image. Increasing the contrast can make the color of the image more saturated, while decreasing the contrast can make the color softer; finally, the image saturation is adjusted to make the color of the image more vivid or more gray.

[0079] S3: Use the mixup linear interpolation method to expand the target training set obtained in step S2, mix the pixel values and labels of several groups of images in proportion, and generate the final training set;

[0080] S31: For each pair of target training samples, an interpolation weight λ is randomly generated from the Beta distribution, which ranges from [0, 1].

[0081] S32: For the two target training sample tensors x1 and x2, and their corresponding label values y1 and y2, linear interpolation is performed according to the following formula:

[0082]

[0083] in, is the new sample generated, is the label of the new sample.

[0084] S4: Constructing a fusion model: Taking the final training set generated in step S3 as input, the final image recognition model is constructed by fusing the improved PANet-SwinTransformer algorithm with the DenseNet algorithm.

[0085] S41 uses the improved PANet-Swin Transformer algorithm for training. During the training process, it saves model snapshots at regular intervals without waiting for the optimization solver iteration to end, and obtains several snapshot models. The prediction results obtained by the snapshot model output are used as new features 1, such as Figure 3 ;

[0086] S42 uses the DenseNet algorithm for training. During the training process, it saves model snapshots regularly without waiting for the optimization solver iteration to complete, and obtains several snapshot models. The prediction results output by the snapshot models are used as new features 2.

[0087] S43 uses the bagging method to fuse new features 1 and 2, and uses the LightGBM algorithm to train the final image recognition model, as shown in Figure 5 .

[0088] S5: Input the new image collected and use the final image recognition model obtained in step S4 to predict the image classification result.

[0089] By integrating different neural network models and fully utilizing the advantages of different models, the image classification and recognition model achieves a good balance between computational efficiency and accuracy, significantly improving the classification accuracy and robustness of the model. It can be widely used in cargo sorting, quality inspection, medical image analysis and other fields to improve the quality of item classification and recognition. At the same time, the system has low deployment costs and is easy to promote and implement, which helps to effectively improve economic benefits.

[0090] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. An image classification and recognition method based on feature fusion, characterized in that: The following steps are involved: S1 divides the data set: the initial images and their corresponding label values are divided into the initial training set, validation set and test set in a ratio of 6:3:1; S2: Preprocessing the images in the initial training set obtained in step S1 to obtain target images, and forming a target training set based on the target images; S3: Use the mixup linear interpolation method to expand the target training set obtained in step S2, mix the pixel values and labels of several groups of images in proportion, and generate the final training set; S4: Constructing a fusion model: Taking the final training set generated in step S3 as input, the final image recognition model is constructed by fusing the improved PANet-SwinTransformer algorithm with the DenseNet algorithm. S5: Input the new image collected and use the final image recognition model built in step S4 to predict the image classification result.

2. The image classification and recognition method based on feature fusion according to claim 1, characterized in that: The preprocessing method in step S2 includes geometric transformation data enhancement and image color transformation data enhancement.

3. The image classification and recognition method based on feature fusion according to claim 2, characterized in that: The geometric transformation data augmentation method is as follows: first, the image is enlarged or reduced by changing its size, so that the model can adapt to input images of different sizes; then, a portion of the image is cut off to remove information irrelevant to the image boundary or generate inputs of different sizes, so that the model has better generalization ability; finally, the image is flipped horizontally or vertically, so that the model can better handle the situation where the object is flipped left and right; The image color transformation data enhancement method is as follows: first, by increasing or decreasing the brightness of the image, images under different lighting conditions are simulated; then, the contrast of the image is adjusted to affect the degree of difference between adjacent pixel values in the image. Increasing the contrast can make the color of the image more saturated, while decreasing the contrast can make the color softer; finally, the image saturation is adjusted to make the color of the image more vivid or more gray.

4. The image classification and recognition method based on feature fusion according to claim 1, characterized in that: Step S3 includes the following steps: S31: For each pair of target training samples, an interpolation weight λ is randomly generated from the Beta distribution, which ranges from [0, 1]. S32: For the two target training sample tensors x1 and x2, and their corresponding label values y1 and y2, linear interpolation is performed according to the following formula: in, is the new sample generated, is the label of the new sample.

5. An image classification and recognition method based on feature fusion according to any one of claims 1 to 4, characterized in that: Step S4 includes the following steps: S41 uses the improved PANet-Swin Transformer algorithm to train several snapshot models, and uses the prediction results output by the snapshot models as new features 1; S42 uses the DenseNet algorithm to train several snapshot models, and uses the prediction results output by the snapshot models as new features 2; S43 uses the bagging method to fuse new features 1 and 2, and uses the LightGBM algorithm to train to obtain the final image recognition model.

6. An image classification and recognition system based on feature fusion, characterized in that: include: An image classification and recognition module, which adopts the image classification and recognition method described in claim 5 and fuses the improved PANet-SwinTransformer algorithm with the DenseNet algorithm to construct a final image recognition model; Backend services to support model training and application; Mobile app, used to view recognition results and provide image classification suggestions to workers.

7. The image classification and recognition system based on feature fusion according to claim 6, characterized in that: The improved PANet-Swin Transformer algorithm architecture consists of four parts: the Swin Transformer backbone network, branch bypass, PANet network, and optimization solver.

8. The image classification and recognition method based on feature fusion according to claim 7, characterized in that: The improved PANet-Swin Transformer algorithm and the DenseNet algorithm are fused using the bagging method and trained using the LightGBM algorithm to obtain the final image recognition model. The LightGBM algorithm is a classifier of the classification system, which receives the fused new feature vector and predicts the image type.

9. The image classification and recognition method based on feature fusion according to claim 6, characterized in that: The background service includes a data management module, a model training module, a model update and maintenance module, and a background monitoring and management module. The data management module is responsible for the full-process management of image data to ensure high-quality data supply; the model training module builds a training framework based on image classification and recognition methods, and realizes efficient model training through dynamic resource scheduling and hyperparameter automatic tuning technology; the model update and maintenance module adopts a version control mechanism and a grayscale release strategy to support model fine-tuning and realize model iterative upgrades; the background monitoring and management module ensures service stability and maintainability by collecting system operation indicators in real time.

10. The image classification and recognition method based on feature fusion according to claim 6, characterized in that: The mobile phone APP includes an image acquisition module, an image transmission module and a result display module to realize the application of image classification function. The image acquisition module supports multi-source image input, quickly obtains the image data to be collected, and performs preliminary preprocessing, cropping, scaling and format conversion to adapt to the subsequent processing flow; The image transmission module is responsible for uploading the processed images securely and efficiently to the backend server, supports stable transmission in a variety of network environments, and encrypts the data to ensure user privacy and data security; the result display module presents the classification and recognition results to the user in an intuitive and clear manner, supports multiple display formats, and helps users quickly apply the recognition results.

Images