An image classification method, apparatus, electronic device, and storage medium

By enhancing image features through a pre-trained image feature extraction network and a foreground enhancement network, the background interference problem of image classification models under few-shot learning conditions is solved, thereby improving the accuracy of image classification.

CN117011568BActive Publication Date: 2026-11-10TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211157002.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-21
Publication Date
2026-11-10
Estimated Expiration
2042-09-21

AI Technical Summary

Technical Problem

In existing image classification models, under few-sample learning conditions, image feature extraction is easily affected by background content, resulting in poor generalization performance and insufficient classification accuracy.

Method used

By employing a pre-trained target image feature extraction network, a foreground enhancement network, and a classification network, the foreground features of the initial image are enhanced to highlight the features of the target object in the image, reduce background interference, and improve feature expressiveness.

Benefits of technology

It improves the accuracy of image classification, effectively characterizes the features of the target object to be classified in the image under small sample conditions, and ensures the accuracy of the classification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117011568B_ABST
    Figure CN117011568B_ABST
Patent Text Reader

Abstract

This application relates to the field of data processing technology and can be applied to in-vehicle scenarios. In particular, it relates to an image classification method, apparatus, electronic device, and storage medium. The method involves: inputting the acquired image to be classified and various candidate images into a target image feature extraction network to obtain corresponding initial features of the image to be classified and initial features of various candidate images; employing a target foreground enhancement network, based on the category attribute features learned during pre-training for sample images of different categories, to obtain corresponding target image features to be classified and various target candidate image features; and then determining the classification result corresponding to the image to be classified based on the feature similarity between the target image features to be classified and each target candidate image feature. This enhances the features corresponding to the foreground content in the extracted image features, enabling the processed image features to highlight the target object region in the foreground of the image, thus improving the accuracy of image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image classification method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, in order to achieve image classification tasks when sufficient training samples are unavailable, an image classification model can be constructed based on few-shot learning, and then the target image classification model trained by few-shot learning can be used to achieve the image classification task.

[0003] In the process of image classification, based on the image feature extraction network in the target image classification model, features are extracted from various reference images and the image to be classified. Then, based on the similarity between the extracted image features, the image content category that matches the image to be classified in various reference images is determined, so as to determine the classification result corresponding to the image to be classified.

[0004] However, the target image classification models under related technologies can only comprehensively extract background and foreground features from the image. As a result, the extracted image features are greatly affected by the background content, making it impossible to effectively determine the classification result based on the extracted image features, thus reducing the image classification effect. In addition, the generalization performance of the image feature extraction network in the target image classification model is poor. In the application process, it can only classify the image content categories involved in training. When faced with images of new content categories, the expressiveness of the extracted image features is insufficient, and the accuracy of image classification cannot be guaranteed. Summary of the Invention

[0005] This application provides an image classification method, apparatus, electronic device, and storage medium to address the problems in the prior art where the extracted image features are greatly affected by background content and lack sufficient expressiveness, thus failing to guarantee the accuracy of image classification.

[0006] Firstly, an image classification method is proposed, including:

[0007] Obtain the image to be classified, and obtain various candidate images pre-constructed corresponding to various candidate sub-classification categories;

[0008] The image to be classified and the candidate images of each category are input into a pre-trained target image feature extraction network to obtain the corresponding initial features of the image to be classified and the features of each initial candidate image.

[0009] A pre-trained target foreground enhancement network is used. Based on the attribute features of each category learned for sample images of different categories during the pre-training process, foreground feature enhancement processing is performed on the initial image features to be classified and the initial candidate image features of each category to be obtained, so as to obtain the corresponding target image features to be classified and the target candidate image features of each category.

[0010] A pre-trained target classification network is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image.

[0011] Secondly, an image classification device is proposed, comprising:

[0012] The first acquisition unit is used to acquire the image to be classified.

[0013] The second acquisition unit is used to acquire various candidate images that are pre-constructed corresponding to various candidate sub-classification categories;

[0014] The input unit is used to input the image to be classified into the pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and to input the candidate images of each type into the target image feature extraction network to obtain the corresponding initial candidate image features.

[0015] The processing unit is used to employ a pre-trained target foreground enhancement network to perform foreground feature enhancement processing on the initial image features to be classified and the initial candidate image features of each category based on the category attribute features learned during the pre-training process for sample images of different categories, thereby obtaining the corresponding target image features to be classified and target candidate image features of each category.

[0016] The classification unit is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image, using a pre-trained target classification network.

[0017] Optionally, when performing foreground feature enhancement processing on the initial image features to be classified based on the category attribute features learned during pre-training for sample images of different categories, the processing unit is used to:

[0018] Based on the category attribute features learned for sample images of different categories during pre-training, the similarity relationship between the initial image features to be classified and the category attribute features is determined respectively.

[0019] Based on the similarity relationship, the matching weights between the initial image features to be classified and the features of each category attribute are determined;

[0020] Based on the matching weights, the enhancement level of each category attribute feature for each position feature value in the initial image to be classified is determined, and based on each enhancement level value, foreground enhancement processing of the initial image to be classified is achieved.

[0021] Optionally, when determining the similarity relationship between the initial image features to be classified and each category attribute feature based on the category attribute features learned during pre-training for sample images of different categories, the processing unit is used to:

[0022] The system acquires the category attribute features learned from sample images of different categories during pre-training, and performs form adjustment on the category attribute features to obtain the attribute feature matrix corresponding to each category attribute feature. The system also performs form adjustment on the initial image features to be classified to obtain the corresponding image feature matrix.

[0023] Calculate the matrix product between the attribute feature matrix and the image feature matrix to obtain a similarity matrix that characterizes the similarity relationship between the initial image features to be classified and the attribute features of each category.

[0024] Optionally, when determining the matching weights between the initial image features to be classified and each category attribute feature based on the similarity relationship, and determining the enhancement level of each category attribute feature for each positional feature value in the initial image features to be classified based on the matching weights, the processing unit is configured to:

[0025] The similarity values ​​at each position in the similarity matrix are normalized, and a weight parameter matrix representing the matching weight between the initial image features to be classified and the features of each category attribute is obtained based on the normalized similarity values ​​at each position.

[0026] Based on the matrix product between the feature matrix composed of each category attribute feature and the weight parameter matrix, the enhancement degree of each category attribute feature for the feature value at each position in the initial image to be classified is determined.

[0027] Optionally, when performing foreground enhancement processing on the initial image features to be classified based on various enhancement levels, the processing unit is used to:

[0028] The enhancement value at each position is superimposed with the feature value at the corresponding position in the initial image features to be classified, to obtain the feature value superposition result corresponding to each position.

[0029] The feature values ​​corresponding to each position are superimposed and adaptively processed to enhance the foreground features in the initial image to be classified.

[0030] Optionally, when acquiring various candidate images pre-constructed corresponding to various candidate sub-classification categories, the acquisition unit is used for:

[0031] In response to a selection instruction for a relevant object, the various candidate subcategories to which the selection instruction is directed are determined, and various candidate images pre-constructed for each of the various candidate subcategories are acquired.

[0032] Optionally, when pre-training an image classification model that includes an image feature extraction network, a foreground enhancement network, and a classification network, the apparatus further includes a training unit, which is used for:

[0033] Based on the open-source target dataset, the image classification model is iteratively pre-trained for multiple training cycles until a preset training cycle is reached. Within one training cycle, the following operations are performed:

[0034] Based on a set of sample images extracted from a subset of training data, the image classification model to be trained is iteratively trained for a set number of rounds to obtain a test image classification model, and based on test images extracted from a subset of test data, the classification accuracy of the test image classification model is determined, wherein the training data subset and the test data subset are contained in the target dataset.

[0035] Based on the classification accuracy determined for the image classification models under test in different training periods, target image classification models that meet the preset screening conditions are selected from each image classification model under test.

[0036] Optionally, when extracting sample images from a subset of the training data, the training unit is used to:

[0037] In the training data subset, for each of the pre-selected N sub-classification categories, K sample images are extracted as one class of reference samples, and M non-overlapping sample images are extracted as samples to be classified.

[0038] Based on the extracted reference samples and samples to be classified, N*M sample image groups are constructed. Each sample image group includes a sample to be classified corresponding to a sub-classification category, where N, K, and M are positive integers.

[0039] Optionally, when the image classification model to be trained is iteratively trained for a set number of rounds based on a sample image group extracted from a subset of the training data to obtain the image classification model to be tested, the training unit is used for:

[0040] Using the constructed sample image set, the image classification model to be trained is iteratively trained for a set number of rounds, and the model parameters of the image classification model are adjusted based on the loss values ​​calculated in each iteration. Specifically, the following operations are performed during one round of iterative training:

[0041] A training sample set is input into the image classification model to be trained to obtain the predicted classification result of the sample to be classified in the training sample set, and the loss value is calculated based on the classification difference between the predicted classification result and the corresponding classification label.

[0042] Optionally, when inputting a set of acquired training samples into the image classification model to be trained to obtain the predicted classification result for the sample to be classified in the training sample set, the training unit is used for:

[0043] The image feature extraction network to be trained is input into the image feature extraction network of the training sample group, which includes the sample to be classified and the reference samples of each class.

[0044] A foreground enhancement network to be trained is used. Based on the category attribute features learned for each category, foreground feature enhancement processing is performed on the features of the sample image to be classified and the features of each reference sample image to obtain the features of the target sample image to be classified and the features of each target reference sample image.

[0045] A classification network to be trained is used to determine the classification result corresponding to the target sample image based on the feature similarity between the features of the target sample image to be classified and the features of each target reference sample image.

[0046] Thirdly, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image classification method described in any one of the first aspects above.

[0047] Fourthly, a computer-readable storage medium is proposed, on which a computer program is stored, which, when executed by a processor, implements the image classification method described in any one of the first aspects above.

[0048] Fifthly, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the image classification method described in any one of the first aspects above.

[0049] The beneficial effects of this application are as follows:

[0050] This application proposes an image classification method, apparatus, electronic device, and storage medium. After acquiring an image to be classified, various candidate images corresponding to different candidate sub-classification categories are acquired. The image to be classified is then input into a pre-trained target image feature extraction network to obtain corresponding initial image features, and the various candidate images are input into the target image feature extraction network to obtain corresponding initial candidate image features. Next, a pre-trained target foreground enhancement network is used to perform foreground feature enhancement processing on the initial image features and various initial candidate image features based on the category attribute features learned during pre-training for sample images of different categories, obtaining corresponding target image features and various target candidate image features. Finally, a pre-trained target classification network is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the target image features and each target candidate image feature.

[0051] In this way, the target image feature extraction network, target foreground enhancement network, and target classification network, which are pre-trained based on few-shot learning, can determine the classification result corresponding to the image to be classified among various candidate sub-classification categories. Moreover, during the feature extraction process, the target foreground enhancement network, which learns various category attribute features during pre-training, can enhance the features corresponding to the foreground content in the extracted image features. This makes the processed image features highlight the target object region in the foreground of the image, increases the expressiveness of the target object in the image, and reduces the interference of background content on the image features. Therefore, it can effectively characterize the features of the target object to be classified in the image to be classified, providing a guarantee for the accuracy of the subsequent classification results based on image features and helping to improve the accuracy of image classification. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating possible application scenarios in the embodiments of this application;

[0053] Figure 2A This is a schematic diagram of the image classification model framework constructed in the embodiments of this application;

[0054] Figure 2B This is a schematic diagram of the network structure of the foreground enhancement network in the embodiments of this application;

[0055] Figure 2C This is a schematic diagram illustrating the process of pre-training the target image classification model in an embodiment of this application;

[0056] Figure 2D This is a schematic diagram illustrating the process of generating reference sample image features in an embodiment of this application;

[0057] Figure 3AThis is a schematic diagram illustrating the process of image classification based on a target image classification model in an embodiment of this application;

[0058] Figure 3B This is a schematic diagram of the target foreground enhancement network processing in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram illustrating the processing tasks that can be implemented in the embodiments of this application;

[0060] Figure 5 This is a schematic diagram illustrating the process of intelligent item sorting based on image classification in the application embodiment;

[0061] Figure 6 This is a schematic diagram of the logical structure of the image classification device in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application;

[0063] Figure 8 This is a schematic diagram of the structure of a computing device in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0066] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0068] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0069] Subcategories: These are the subcategories that are further refined and defined under a classification category. For example, for the classification category "animals", specific animals such as "dogs" and "cats" can be considered as the corresponding subcategories; similarly, for the classification category "plants", specific plants such as "orchids" and "mimosa" can be considered as the corresponding subcategories.

[0070] Few-shot learning primarily enables fast and efficient model learning when the number of labeled samples is small. It is an application of meta-learning in the field of supervised learning.

[0071] N-way K-shot is a common training setup for few-shot learning in classification. In this embodiment, during the pre-training phase, N sub-classes are extracted from the training sample subset, with K sample images for each sub-class, totaling N*K sample images constituting a meta-task. These serve as various reference samples (collectively referred to as support sets) during model pre-training. Then, a batch of sample images is extracted from the remaining sample images of the N sub-classes as the model's unclassified samples (also called prediction objects, denoted as query sets). This task is known as the N-way K-shot problem.

[0072] Foreground: This is a concept corresponding to the background. For an image, the image consists of foreground content and background content. In the embodiments of this application, the foreground corresponds to the target object region in the image that determines the corresponding sub-category.

[0073] The design concept of the embodiments of this application is briefly introduced below:

[0074] Due to factors such as data privacy, security, and the high cost of labeling training data, many real-world applications (e.g., industry, medicine, and finance) may not be able to obtain sufficient labeled samples. In such cases, training a deep learning system to efficiently learn and generalize its cognitive abilities from a very small number of labeled samples is of significant importance and practical value. Compared to conventional deep learning methods that require a large number of labeled training samples, few-shot learning methods can learn high-performance models with only a small number of labeled samples.

[0075] In the process of image classification based on few-shot learning, metric-based few-shot classification methods can be used, such as those based on MatchingNet, ProtoNet, or RelationNet. The process of few-shot classification based on metric learning is as follows: First, an image feature extraction network is used to extract deep features from a small set of labeled training samples and the samples to be classified, obtaining corresponding features. The training sample set includes various types of reference image samples. Then, a similarity metric method is used to calculate the similarity between features, identifying the target reference image that is most similar to the sample to be classified among the various types of reference image samples. Finally, the classification result corresponding to the target reference image is determined based on the category of the target reference image.

[0076] However, current target image classification models have poor ability to distinguish between target objects and background content in images during feature extraction. They can only extract background and foreground features, resulting in significant interference from background content. Furthermore, the generalization ability of image feature extraction networks in image classification models is poor, and the extracted features lack representativeness. In application, they can only classify the subcategories involved in training. When faced with images of new subcategories, the extracted image features lack expressiveness, making it difficult to effectively distinguish between images of different subcategories and failing to guarantee the accuracy of image classification.

[0077] In view of this, this application proposes an image classification method, apparatus, electronic device, and storage medium. After acquiring the image to be classified, various candidate images corresponding to different candidate sub-classification categories are acquired. The image to be classified is then input into a pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and various candidate images are input into the target image feature extraction network to obtain the corresponding initial candidate image features. Subsequently, a pre-trained target foreground enhancement network is used to perform foreground feature enhancement processing on the initial image features to be classified and the various initial candidate image features based on the category attribute features learned during pre-training for sample images of different categories, obtaining the corresponding target image features to be classified and the various target candidate image features. Finally, a pre-trained target classification network is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the target image features to be classified and each target candidate image feature.

[0078] In this way, the target image feature extraction network, target foreground enhancement network, and target classification network, which are pre-trained based on few-shot learning, can determine the classification result corresponding to the image to be classified among various candidate sub-classification categories. Moreover, during the feature extraction process, the target foreground enhancement network, which learns various category attribute features during pre-training, can enhance the features corresponding to the foreground content in the extracted image features. This makes the processed image features highlight the target object region in the foreground of the image, increases the expressiveness of the target object in the image, and reduces the interference of background content on the image features. Therefore, it can effectively characterize the features of the target object to be classified in the image to be classified, providing a guarantee for the accuracy of the subsequent classification results based on image features and helping to improve the accuracy of image classification.

[0079] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0080] See Figure 1 The diagram shown is a possible application scenario illustration in the embodiments of this application. The application scenario illustration includes a terminal device 110 (specifically, it may be a terminal device 1101, a terminal device 1102, etc.) and a server device 120.

[0081] It should be noted that the terminal device 110 and the server device 120 can communicate via wired or wireless networks.

[0082] In this embodiment of the application, when the terminal device 110 has an image acquisition function, it can directly acquire two-dimensional images. When the terminal device 110 does not have an image function, it can acquire images acquired by an image acquisition device. The image acquisition device can be various types of cameras or other devices capable of acquiring two-dimensional images.

[0083] Terminal device 110 can be a computer device with certain computing capabilities, such as a personal computer, mobile phone, tablet computer, laptop, e-book reader, smart home device, or vehicle terminal; or it can be a video resource acquisition device such as a camera or webcam.

[0084] The server device 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0085] The technical solution proposed in this application can obtain corresponding classification results based on the obtained image to be classified in various application scenarios.

[0086] Specifically, it can be applied to scenarios involving animal and plant identification, classifying and recognizing photographs of animals or plants; or to scenarios involving fruit and vegetable classification, identifying photographs of vegetables or fruits to achieve automatic classification; or to scenarios involving building identification, recognizing photographed building images to identify specific buildings; or to scenarios involving comprehensive content recognition, classifying and recognizing photographed images.

[0087] It should be noted that, in the embodiments of this application, based on actual processing needs, the target image classification model can achieve classification tasks of multiple categories. For example, after pre-training based on open-source sample images including animals, plants, and buildings, a target image classification model can be used to achieve classification and recognition of animal images, plant images, or building images.

[0088] In this embodiment of the application, either or both of the terminal device 110 and the server device 120 may be configured with a pre-trained target image classification model, wherein the target image classification model includes a pre-trained target image feature extraction network, a target foreground enhancement network, and a target classification network; so that the terminal device 110 or the server device 120 can use the pre-trained target image classification model to classify the image to be identified.

[0089] It should be noted that, in the embodiments of this application, the target image classification model used on the device can be trained by itself, or it can be trained by other devices.

[0090] In some possible embodiments, when a target image classification model is configured on the terminal device 110, after the terminal device 110 acquires or collects the image to be classified, it can obtain various candidate images corresponding to various candidate sub-classification categories pre-constructed from the server device 120, and then use the target image classification model to determine the classification result corresponding to the image to be classified based on the image to be classified and various candidate images, and present it to the relevant user.

[0091] In some other possible embodiments, when the server device 120 is configured with a target image classification model, the server device 120 can acquire the image to be classified sent by the terminal device 110, or it can acquire the image to be classified sent by the image acquisition device; then, using the target image classification model, based on the image to be classified and various candidate images corresponding to various pre-constructed candidate sub-classification categories, it obtains the classification result corresponding to the image to be classified, and sends the classification result to the designated terminal device 110, so that the classification result can be presented to the relevant user on the terminal device 110.

[0092] It should be noted that, in the embodiments of this application, the pre-trained target image classification model running on the device can be trained by the device itself, or it can be obtained from other devices after training by other devices.

[0093] For example, if a target image classification model is running on the terminal device 110, the target classification model may be trained by the server device 120 or other devices and then provided to the terminal device 110 by the server device 120.

[0094] In the following description of this application, the processing process will be illustrated from the perspective of the processing device, taking the implementation of model training and classification processing by the processing device as an example. Depending on the different application scenarios of this application, the processing device may specifically correspond to… Figure 1 The terminal device 110 shown in the diagram, or any one of the server device 120, is not specifically limited in this application.

[0095] The training process of the target image classification model in the embodiments of this application will be described in detail below with reference to the accompanying drawings:

[0096] See Figure 2A As shown, it is a schematic diagram of the image classification model framework constructed in the embodiments of this application. The following is in conjunction with the attached diagram. Figure 2A The model framework of the image classification model is illustrated below:

[0097] according to Figure 2A As shown in the diagram, the constructed image classification model includes an image feature extraction network, a foreground enhancement network, and a classification network.

[0098] 1) Image feature extraction network: This can be regarded as the backbone network in the image classification model. It is used to extract features from the input image to obtain image features. The network structure corresponding to the image feature extraction network can be any structure or combination of network structures that can achieve image feature extraction, such as Residual Network (ResNet-12), Wide ResNet, Squeeze-and-Excitation Networks (SENet), and High-Resolution Network (HRNet).

[0099] 2) Foreground Enhancement Network: Used to highlight the features of target objects in an image. The target object is the object in the image that determines the classification result. For example, in a 2D image with the subclass "dog," the target object is the "dog" in the image. The foreground enhancement network receives various image features extracted by the image feature extraction network and performs foreground feature enhancement processing on each image feature to highlight the target object features and reduce background feature interference, resulting in image features with enhanced foreground characteristics.

[0100] For details, please refer to Figure 2B As shown, this is a schematic diagram of the network structure of the foreground enhancement network in an embodiment of this application. The following is a description of the network structure in conjunction with the attached diagram. Figure 2B The network structure of the foreground enhancement network is explained below:

[0101] Figure 2B The network structure shown is a foreground enhancement network (denoted as tSF network), used to highlight and enhance the features of the target object in image features. Figure 2B In terms of the network structure, its input is feature f, which is the image feature extracted by the image feature extraction network; then, a foreground enhancement network is used to process feature f and output the enhanced feature f', denoted as f' = g. θ (f).

[0102] Combination Figure 2BAs can be seen, in the foreground enhancement network, learnable parameters θ, called semantic vectors, are constructed for various types of images. During pre-training, these vectors can learn the category attributes of different image categories. For example, from the various animal images included in the category "animals," the attribute features of the category "animals" can be learned. The attention part is used to calculate the enhancement level value for each position in the image features based on the input image features and the constructed semantic vectors. Then, the enhancement level value corresponding to each position is superimposed with the feature value of the corresponding position to obtain the image features after foreground enhancement. The processed image features are then adaptively processed by a feed-forward network (FFN) to generate image features with stable transitions at each position. The FFN can be a feed-forward network in the standard Transformer structure, and the FFN is generally a multilayer perceptron (MLP) structure.

[0103] It should be noted that the specific process of using a foreground enhancement network to process the input image features will be explained in detail in the following steps.

[0104] 3) Classification network: used to obtain classification results by calculating the similarity between features. In the embodiments of this application, the classification network can be constructed using the structure of a metric classifier, or it can use other network structures that can achieve the same or similar functions as the metric classifier. This application does not impose specific restrictions on this.

[0105] During the model training phase, the classification network receives the target sample image features processed by the foreground enhancement network and the features of each target reference sample image, and calculates the feature similarity between the target sample image features and each reference sample image features, and finally obtains the classification result corresponding to the sample to be classified. The target sample image features are features obtained based on the processing of the sample to be classified.

[0106] In the model application phase, the target classification network receives the target image features to be classified and the features of each target candidate image after processing by the pre-trained target foreground enhancement network, and calculates the feature similarity between the target image features to be classified and the features of each target candidate image, and finally obtains the classification result corresponding to the image to be classified. The target image features to be classified are features obtained based on the processing of the image to be classified.

[0107] It should be noted that when calculating feature similarity between features, measures such as cosine distance, learnable metric, Euclidean distance, Manhattan distance, and Minkowski distance can be used to calculate feature similarity.

[0108] See Figure 2C As shown, this is a schematic diagram illustrating the process of pre-training the target image classification model in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 2C The process of pre-training the target image classification model in the embodiments of this application is described below:

[0109] It should be noted that the constructed image classification model can be understood as a few-sample classification model, which includes an image feature extraction network, a foreground enhancement network, and a classification network.

[0110] In this embodiment of the application, when the processing device pre-trains the image classification model, it uses an open-source target dataset to perform iterative pre-training of the image classification model for multiple training cycles until a preset training cycle is reached. The preset number of training cycles is set according to the actual processing needs, and this application does not specifically limit it.

[0111] It should be noted that, in the embodiments of this application, when pre-training the image classification model, an open-source target dataset can be directly selected. The open-source target dataset can be any one or a combination of the miniImageNet dataset, the tieredImageNet dataset, and the ImageNet dataset.

[0112] In this embodiment of the application, for the publicly disclosed dataset, the dataset includes sample images of various subcategories, wherein each sample image has a corresponding classification label; subcategories and categories are relative concepts, which can be understood as categories corresponding to a large range of classifications, and subcategories have a class to which they belong, such as subcategories "cat" and "dog" belonging to the class "animal".

[0113] In this embodiment of the application, after the processing device obtains the open-source target dataset, it can divide the target dataset into a training data subset and a test data subset according to the actual processing needs.

[0114] When specifically dividing the training data subset and the test data subset, they can be allocated proportionally according to the total number of subcategories included in the target dataset. The proportional value used for allocation can be set according to the actual processing needs, and this application does not impose specific limitations on it.

[0115] For example, assuming the target dataset includes sample images of 100 subcategories, you can randomly select 80 sample images of subcategories as the training data subset and use the remaining 20 sample images of subcategories as the test data subset.

[0116] It should be noted that in the embodiments of this application, the subcategories included in the training data subset and the test data subset may not overlap. Moreover, in order to ensure that stable category attribute information can be learned during the pre-training stage, the subcategories included in the training data subset and the test data subset are usually not adjusted during the pre-training process.

[0117] In addition, considering that open-source datasets typically include a large number of image subcategories, this application assumes that each subcategory belonging to the same category can be assigned to the training data subset and the test data subset during the allocation process. That is, neither the training data subset nor the test data subset includes images of unique categories. For example, sample images of each subcategory (i.e., specific animal species) under the category "animals" will not be uniformly assigned to either the training data subset or the test data subset.

[0118] The following example illustrates the training process performed within a single training cycle:

[0119] Step 201: The processing device iteratively trains the image classification model to be trained for a set number of rounds based on the sample image group extracted from the training data subset, to obtain the image classification model to be tested, wherein the training data subset is contained in the target dataset.

[0120] In this embodiment of the application, when the processing device extracts sample image groups from the training data subset, the processing device extracts K sample images as reference samples for each of the pre-selected N sub-classification categories in the training data subset, and extracts M non-overlapping sample images as samples to be classified; then, based on the extracted reference samples and samples to be classified, N*M sample image groups are constructed, wherein each sample image group includes a sample to be classified corresponding to a sub-classification category, and N, K, and M are positive integers.

[0121] Specifically, the processing device can construct an N-way K-shot task, randomly selecting N sub-classes from the training data subset as pre-selected sub-classes, and then extracting K sample images for each pre-selected sub-class as a reference sample, thus obtaining N*K sample images forming each type of reference sample (denoted as SupportSet); at the same time, extracting M sample images from the remaining sample images of each pre-selected sub-class as samples to be classified, thus obtaining N*M samples to be classified (denoted as query set).

[0122] Furthermore, for each sample to be classified, a sample image group is constructed by combining the sample to be classified with the extracted reference samples of each category, so that N*M sample image groups can be obtained. The values ​​of M, M, and K are set according to the actual processing needs, and all are positive integers. The number of sample images corresponding to each sub-category in the open-source target dataset is at least greater than M+N.

[0123] In this way, the sample image set used in a training cycle can be determined from the training data subset, providing a training basis for the subsequent pre-training process.

[0124] Furthermore, after the processing device extracts a sample image group from the training data subset, it uses the constructed sample image group to perform iterative training on the image classification model to be trained for a set number of rounds. Based on the loss value calculated in each round of iteration, the model parameters of the image classification model are adjusted. The set number of rounds is set according to the actual processing needs, and this application does not impose specific restrictions on it.

[0125] In this way, within a training cycle, the model parameters in the image classification model can be adjusted in reverse based on the loss values ​​calculated during each round of iterative training within that training cycle, thus enabling effective training of the image classification model.

[0126] In this embodiment of the application, during one round of iterative training, the processing device performs the following operations: inputs a training sample group into the image classification model to be trained, obtains the predicted classification result of the sample to be classified in the corresponding training sample group, and calculates the loss value based on the classification difference between the predicted classification result and the corresponding classification label.

[0127] Specifically, when a training sample set is input into the image classification model to be trained, and the predicted classification result of the sample to be classified in the corresponding training sample set is obtained, the processing device inputs the sample to be classified and various reference samples included in the training sample set into the image feature extraction network to be trained, to obtain the image features of the sample to be classified and the image features of each reference sample; then, the foreground enhancement network to be trained performs foreground feature enhancement processing on the image features of the sample to be classified and the image features of each reference sample based on the category attribute features learned for each classification category, to obtain the image features of the target sample to be classified and the image features of each target reference sample; finally, the classification network to be trained determines the classification result corresponding to the sample image to be classified based on the feature similarity between the image features of the target sample to be classified and the image features of each target reference sample.

[0128] It should be noted that, in the embodiments of this application, when a training sample group includes N*K reference samples and 1 sample to be classified, the processing device inputs each training sample group into the image feature extraction network to be trained, so that the image feature extraction network can perform feature extraction of the sample to be classified and various types of reference samples in parallel.

[0129] Furthermore, considering that each type of reference sample includes K sample images, when performing feature extraction, for each type of reference sample, K sample image features can be obtained. Then, the feature values ​​at the same position in the K sample image features can be weighted and averaged, and reference sample image features can be generated based on the feature values ​​obtained at each corresponding position. The weights involved are set according to the actual processing needs, and this application does not make specific limitations on them. The extracted image features can specifically exist in the form of feature maps.

[0130] For example, suppose the features of a sample image are in the form of c×h×w, where the features of the sample image are represented in the form of a feature map, c is the number of channels of the feature map, h represents the height of the feature map, and w represents the width of the feature map; when generating the features of the reference sample image, the weighted average result of each feature value with the same relative position in the K features of the sample image is calculated to obtain the target feature value at the corresponding position, and then the reference sample image features are generated based on the target feature values ​​at each position.

[0131] For example, see Figure 2D As shown, it is a schematic diagram of the process of generating reference sample image features in an embodiment of this application. Figure 2DAs shown, based on K sample images included in a corresponding class of reference samples, K sample image features are extracted respectively. Each sample image feature is in the form of c×h×w. Then, when generating reference sample image features, assuming that the weight parameters are all set to 1, when generating reference sample image features, taking the feature position defined by the thick black line as an example, the feature values ​​with the same relative position in different sample image features are weighted and averaged to obtain the target feature value at the corresponding position. Then, based on the target feature value at each position, the corresponding class of reference sample image features are obtained.

[0132] It should be noted that, in the embodiments of this application, when using a foreground enhancement network for foreground enhancement processing, the processing device can use the following formula to obtain the various features after foreground enhancement:

[0133] f = f θ (x);

[0134] f′=g θ (f θ (x))=FFN{f+softmax(fθ T )θ}

[0135] In this model, the FFN representation uses a feedforward network in the standard Transformer structure to achieve adaptive processing; f represents the image features extracted by the image feature extraction network; x represents the sample image; f θ (x) represents the image feature extraction operation performed on the sample image; θ represents the learnable parameters in the foreground enhancement network, denoted as semantic vector, which is used to represent the attribute features of each category. The categories represented by θ correspond to a wide range of categories, such as "animals" and "plants".

[0136] In this embodiment, during one round of iterative training, the processing device uses an image feature extraction network to perform deep feature extraction on various types of input reference samples and samples to be classified, obtaining corresponding reference image features and sample image features. The various reference image features are assumed to be [f...]. S_1 ,f S_2 ,…,f S_N The image features of the sample to be classified are assumed to be f. Q Then, a foreground enhancement network is used to enhance all features [f] S_1 ,f S_2 ,…,f S_N ] and f Q Feature enhancement is performed separately to highlight the target object's features and reduce background feature interference, resulting in the enhanced features [P1, P2, ..., P]. N Finally, a classification network is used to calculate feature pairs [P] and Q; kThe feature similarity between [Q], k = 1, 2, ..., N is determined, and the Pi most similar to Q is identified. Then, the sub-classification category corresponding to Pi is determined as the classification result of the sample to be classified.

[0137] In this way, the classification results obtained from the image classification model during the pre-training process provide direction for adjusting the image classification model.

[0138] Based on this, the processing device calculates the loss value using the cross-entropy loss function, based on the classification difference between the classification result of the sample to be classified and the classification label corresponding to the sample to be classified.

[0139] Similarly, within a training cycle, by using the extracted sample image groups to achieve multiple rounds of iterative training of the image classification model, each sample image group can be used to obtain a loss value. The processing device can then weight and sum the loss values ​​obtained within a training cycle to obtain a weighted sum of the corresponding loss values. Then, based on the weighted sum of the loss values, gradient descent training is performed to adjust the model parameters in each network of the image classification model. The weight parameters set for the loss values ​​obtained in each round of iterative training can be set according to the actual processing needs, and this application does not impose specific restrictions on them.

[0140] Furthermore, after adjusting the model parameters in the image classification model, a test image classification model can be obtained for the current training cycle. Similarly, the processing device can perform iterative training in each training cycle, and in each subsequent training cycle, the test image classification model obtained in the previous cycle is used as the image classification model to be trained in the current training cycle, and the test image classification model is obtained for the current training cycle in the same way.

[0141] Step 202: The processing device determines the classification accuracy of the image classification model under test based on the test images extracted from the test data subset, wherein the test data subset is contained in the target dataset.

[0142] Specifically, similar to the training process of an image classification model, when testing the image classification model to be tested, the processing device extracts D sample images from the test data subset for each of the pre-selected B-class sub-classification categories as test samples, and extracts E non-overlapping sample images as samples to be matched. Then, each sample to be matched and the B-class test samples are combined to construct a test sample group, resulting in E*B test sample groups. Afterwards, each test sample group and the sample to be matched are input into the image classification model to be tested, obtaining the predicted classification result corresponding to the sample to be matched. This allows the accuracy of the image classification model to be tested to be determined based on the matching between the predicted classification result and the classification label corresponding to the sample to be matched. The values ​​of B, D, and E are set according to actual processing needs and are not specifically limited in this application. Furthermore, the value of B is not affected by the value of N during training, the value of D is not affected by the value of K during training, and the value of E is not affected by the value of M during training.

[0143] Furthermore, the processing device can obtain the predicted classification results of the image classification model under test for each test sample group, and can obtain the judgment result of whether the classification is accurate for each test sample group; then, for each test sample group currently constructed, the processing device counts the total number of accurate classifications, and calculates the classification accuracy of the image classification model under test in the current training period based on the total number of accurate classifications and the total number of test sample groups.

[0144] For example, assuming that the image classification model trained for the current training cycle has constructed M*N test sample groups, and after the test is completed, the total number of times the classification is accurate is Z, then the calculated classification accuracy is: Z / (M*N)*100%.

[0145] In this way, by calculating the classification accuracy, we can provide a reference for evaluating the performance of the image classification model trained in the current training cycle, and make it possible to quantitatively measure the training effect of the image classification model.

[0146] Furthermore, the processing device can similarly test the image classification models obtained in each training cycle, and based on the classification accuracy determined for the image classification models in different training cycles, select target image classification models that meet preset screening conditions from each image classification model. The preset screening conditions can be that the corresponding classification accuracy is the highest and not lower than a first set value, or that the comprehensive classification performance is optimal and the comprehensive score is higher than a second set value after comprehensively measuring the classification accuracy and related technical indicators. The first and second set values ​​are set according to the actual processing needs and are not specifically limited here.

[0147] Specifically, in this embodiment, when the classification accuracy of each image classification model to be tested is lower than a first preset value, or when the comprehensive score of each image classification model to be tested is not higher than a second preset value, the processing device may repeatedly perform the following operations until a target image classification model that meets the requirements is obtained: select an open-source dataset as the target dataset according to actual processing needs; then select a subset of training data and a subset of test data from the target dataset, and perform multi-cycle iterative training based on the image classification model to be tested obtained from the last training; perform similar testing using the above testing method, and similarly screen the image classification models to be tested.

[0148] In this way, by reasoning and verifying the image classification model, and through multiple rounds of iterative training in each training cycle, a target image classification model that meets the expected requirements can be trained.

[0149] See Figure 3A As shown, it is a schematic diagram of the process of image classification based on the target image classification model in the embodiments of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 3A The process of image classification using a target classification model is explained in detail below:

[0150] It should be noted that in this embodiment of the application, the processing device uses an open-source target dataset to pre-train the image classification model. After obtaining the target image classification model, the image classification model can be directly applied to perform specific image classification tasks. The target image classification model includes a pre-trained target feature extraction network, a target foreground enhancement network, and a target classification network.

[0151] Furthermore, considering that currently available public datasets (or open-source datasets) typically include images of various subcategories, covering a wide range of classification categories in common scenarios, pre-training an image classification model using public datasets can assist the model in learning the category attribute features for each category. In this embodiment, it is assumed that the classification category of the image identified during actual image classification processing is included in the dataset used for training. That is, assuming the actual application scenario aims to classify vegetables, the open-source dataset used should include specific sample images belonging to the "vegetable" category.

[0152] Step 301: The processing device acquires the image to be classified.

[0153] In this embodiment of the application, depending on the actual business scenario, if the processing device itself has an image capturing function, the processing device can directly use the captured image as the image to be classified; or, if the processing device itself does not have an image capturing function, the processing device can use the indirectly acquired image as the image to be classified.

[0154] Step 302: The processing device acquires various candidate images pre-constructed corresponding to various candidate sub-classification categories.

[0155] It should be noted that when the processing device performs image classification processing, it can pre-construct candidate images for each type of candidate sub-classification, so that the sub-classification that matches the image to be classified can be determined from the candidate images in subsequent processing. The total number of candidate sub-classifications and the number of candidate images under each candidate sub-classification can be set according to actual processing needs. For example, they can be set in the same way as in the pre-training process, such as N candidate sub-classifications and K images selected under each candidate sub-classification. Alternatively, the total number of candidate sub-classifications and the number of images under each candidate sub-classification can be determined according to actual processing needs, and the specific values ​​are not affected by the values ​​of N and K in the pre-training process.

[0156] Optionally, in order to meet the image classification needs of various business scenarios, the processing device can construct corresponding candidate images for different business scenarios.

[0157] In this context, the processing device can identify the various subcategories belonging to the corresponding classification categories when performing image classification in various business scenarios, and use them as candidate subcategories. It can then acquire images belonging to each candidate subcategory and construct corresponding candidate images. Furthermore, the processing device can provide a business scenario selection function to determine the classification category range for which relevant objects need image classification, and then acquire candidate images constructed for this classification category range. The specific images and candidate subcategories included in each candidate image can be updated according to actual processing needs; this application does not impose specific limitations on this.

[0158] Furthermore, in response to the selection instruction of the relevant object, the processing device can determine the various candidate subclasses to which the selection instruction is directed, and acquire various candidate images pre-constructed for each of the various candidate subclasses.

[0159] In this way, by using the selection instructions triggered by the relevant objects, various candidate images can be selected in a targeted manner according to the usage needs of the relevant objects, so that the various candidate images used in the classification process can better match the actual classification scenario.

[0160] Specifically, in this embodiment of the application, when the processing device performs image classification only for a fixed business scenario, the processing device can maintain various candidate images only for that business scenario. For example, when the processing device is only used for vegetable classification and recognition, it can generate various candidate images in advance based on images of various vegetables.

[0161] In this way, by acquiring various candidate images, the range of results for the classification of the image to be classified is determined, providing a basis for subsequent classification processing of the image to be classified.

[0162] Step 303: The processing device inputs the image to be classified into the pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and inputs various candidate images into the target image feature extraction network to obtain the corresponding initial candidate image features.

[0163] Specifically, similar to the image feature extraction process for the input image during training, the processing device inputs a target image feature extraction network and various candidate images into the pre-trained target image feature extraction network. This allows the target image feature extraction network to extract image features in parallel for the target image and various candidate images, obtaining the initial target image features extracted by the target image feature extraction network for the target image and the initial candidate image features extracted by the target image feature extraction network for each candidate image class. Each candidate image class includes at least one candidate image.

[0164] It should be noted that since a candidate image in a candidate subclass may include multiple candidate images (let's say K images), after the processing device obtains the K original candidate image features extracted by the target image feature extraction network for the K candidate images, it can use the same processing method as in the training process to perform weighted averaging of the feature values ​​at the same position in the K original candidate image features, and generate an initial candidate image feature based on the feature values ​​obtained at each corresponding position. The weights involved are set according to the actual processing needs, and this application does not make specific limitations on them.

[0165] Based on this, the processing device can process various candidate images to obtain the features of each initial candidate image.

[0166] In this way, with the help of the target image feature extraction network, it is possible to extract the corresponding initial candidate image features for candidate images of various candidate sub-classification categories, and to extract the initial unclassified image features for the image to be classified. This is equivalent to processing the image into a feature form that the model can further process and match.

[0167] Step 304: The processing device uses a pre-trained target foreground enhancement network. Based on the attribute features of each category learned during the pre-training process for sample images of different categories, it performs foreground feature enhancement processing on the initial image features to be classified and the initial candidate image features to obtain the corresponding target image features to be classified and the target candidate image features of each category.

[0168] In this embodiment of the application, the processing device uses a target foreground enhancement network, which performs foreground feature enhancement processing on the initial image features to be classified and the features of each initial candidate image based on the category attribute features learned during the pre-training process for sample images of different categories.

[0169] It should be noted that since the foreground feature enhancement logic is the same when performing foreground feature enhancement processing on the initial features to be classified and each initial candidate image feature, the following description will only use the foreground feature enhancement processing on the initial features to be classified as an example for illustrative purposes. In addition, in the embodiments of this application, the processing logic implemented by the foreground enhancement network during the pre-training process is consistent with the processing logic implemented by the target foreground enhancement network when implementing image classification. The specific processing logic executed internally by the target foreground enhancement network will be illustrated below:

[0170] In the target foreground enhancement network, based on the category attribute features learned during the pre-training process for sample images of different categories, the similarity relationship between the initial image features to be classified and each category attribute feature is determined. Then, based on the obtained similarity relationship, the matching weight between the initial image features to be classified and each category attribute feature is determined. Furthermore, based on the matching weight, the enhancement degree of each category attribute feature for each position feature value in the initial image features to be classified is determined, and based on each enhancement degree value, the foreground enhancement processing of the initial image features to be classified is realized.

[0171] Specifically, when determining the similarity relationship between the initial image features to be classified and the various category attribute features based on the category attribute features learned from sample images of different categories during pre-training, the processing device acquires the category attribute features learned from sample images of different categories during pre-training, and performs form adjustment on each category attribute feature to obtain the attribute feature matrix corresponding to each category attribute feature. It also performs form adjustment on the initial image features to be classified to obtain the corresponding image feature matrix. Then, it calculates the matrix product between the attribute feature matrix and the image feature matrix to obtain the similarity matrix representing the similarity relationship between the initial image features to be classified and the various category attribute features.

[0172] See Figure 3B As shown, this is a schematic diagram of the target foreground enhancement network in an embodiment of this application. The following is a description of the process in conjunction with the attached diagram. Figure 3B The processing procedure implemented using a target foreground enhancement network is explained. For the target foreground enhancement network, during the pre-training process, semantic vectors θ representing the attribute features of each category are learned. The semantic vector θ is of the form n×c, where c is the number of channels of the semantic vector, and n is the preset total number of each category, which is set before pre-training, for example, a value of 5.

[0173] Combination Figure 3B As can be seen, when generating the similarity matrix within the target foreground enhancement network, the processing device inputs the initial unclassified image features f extracted by the target feature extraction network based on the unclassified image x into the target foreground enhancement network. Here, the initial unclassified image is of the form c×h×w, where c is the number of channels in the feature map, h represents the height of the feature map, and w represents the width of the feature map. Then, within the target foreground enhancement network, the initial unclassified image features f are reshaped to obtain an image feature matrix Q of the form hw×c. Simultaneously, the semantic vector θ is reshaped to obtain an attribute feature matrix K of the form c×n. Afterwards, the MatMul function is used to calculate the matrix product between the image feature matrix Q and the attribute feature matrix K, obtaining a similarity matrix representing the similarity relationship between the initial unclassified image features and the attribute features of each category. This similarity matrix is ​​of the form hw×n.

[0174] In this way, by calculating the matrix product, it is equivalent to determining the matching between the initial image features to be classified and the features of each category attribute, thereby identifying the category attribute features similar to the initial image features to be classified and characterizing the similarity relationship.

[0175] Within the target foreground enhancement network, based on the obtained similarity relationships, the matching weights between the initial image features to be classified and the features of each category attribute are determined. Then, based on these matching weights, the enhancement degree of each category attribute feature for each position of the initial image features to be classified is determined. The similarity values ​​at each position in the similarity matrix are normalized, and based on the normalized similarity values ​​at each position, a weight parameter matrix representing the matching weights between the initial image features to be classified and the features of each category attribute is obtained. Finally, based on the matrix product between the feature matrix composed of each category attribute feature and the weight parameter matrix, the enhancement degree of each category attribute feature for each position of the initial image features to be classified is determined.

[0176] Continue to combine Figure 3BTo illustrate, after obtaining a similarity matrix of the form hw×n to represent similarity relationships, a softmax function is used to normalize the similarity values ​​at each position in the similarity matrix. Based on the processed similarity values ​​at each position, a weight parameter matrix is ​​constructed. This weight parameter matrix, also of the form hw×n, represents the matching weight between the initial image features to be classified and the features of each category attribute. Then, the MatMul function is used to calculate the matrix product between the feature matrix composed of each category attribute feature and the weight parameter matrix. This matrix product represents the degree to which each category attribute feature enhances the feature values ​​at each position in the initial image features to be classified. The resulting matrix product is of the form hw×c.

[0177] In this way, after obtaining the normalized weight parameter matrix based on feature similarity, the enhancement level of each feature value in the initial image to be classified can be determined by performing matrix multiplication on the weight parameter matrix and the feature matrix corresponding to each category attribute information, thus providing a basis for subsequent feature enhancement processing.

[0178] Furthermore, within the target foreground enhancement network, based on each enhancement level value, when performing foreground enhancement processing on the features of the initial image to be classified, the enhancement level value at each position is superimposed with the feature value at the corresponding position in the features of the initial image to be classified, to obtain the feature value superposition result corresponding to each position; the feature value superposition result corresponding to each position is adaptively processed to enhance the foreground features in the features of the initial image to be classified.

[0179] Continue to combine with the appendix Figure 3B To illustrate, after obtaining the matrix product of the form hw×c, which represents the enhancement level at each position, the feature values ​​at the same position of the matrix product can be calculated and superimposed with the initial image features to be classified after form adjustment. Then, FFN is used to adaptively process the feature value superposition results at each position, and finally the initial image features to be classified after foreground enhancement are obtained.

[0180] In addition, combined Figure 3B As illustrated in the diagram of the foreground enhancement network's enhancement effect, the image to be classified, x, is input into the target feature extraction network f. θ The initial features f of the image to be classified are obtained, and then the initial features f are input into the target foreground enhancement network (denoted as tSF:g). θ This allows us to obtain the features f' of the target image to be classified. For example... Figure 3BAs shown, in the initial image features f, the highlighted feature is the area next to the target object, indicated by the black solid line box. After processing by the target foreground enhancement network, in the target image features f', the highlighted feature is the area where the target object is located, indicated by the black solid line box. That is, the features of the target object area in the image x to be classified are highlighted, reducing the interference of background content.

[0181] In this way, by using the semantic vectors learned during the pre-training process, feature enhancement can be performed on image features, which can highlight the features of the foreground target area in the initial image features and reduce the interference of the background area on the image features.

[0182] Step 305: The processing device uses a pre-trained target classification network to determine the classification result corresponding to the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image.

[0183] The processing device uses a target foreground enhancement network to process and obtain the target image features to be classified and the features of each target candidate image. Then, it uses a pre-trained target classification network to calculate the feature similarity between the target image features to be classified and each target candidate image feature. Then, it determines the target candidate image feature corresponding to the maximum similarity from each feature similarity and determines the sub-classification category corresponding to the determined target candidate image feature as the classification result corresponding to the image to be classified.

[0184] In this way, by calculating feature similarity, it is possible to determine the sub-classification category that matches the image to be classified among the sub-classification categories corresponding to various candidate images, and thus obtain the classification result corresponding to the image to be classified.

[0185] It should be noted that, in the embodiments of this application, other task-implementing networks can also be added based on the feature extraction network and the target foreground enhancement network. For example, network structures can be added to implement object detection and object segmentation tasks. After enhancing the content of the target object in the image, the detection or segmentation of the target object in the image can be achieved.

[0186] For example, see Figure 4 As shown, this is a schematic diagram of the processing task that can be implemented in the embodiment of this application. The input image x is processed by an image feature extraction network (backbone) and a foreground enhancement network (tSF) for feature extraction and enhancement, resulting in feature f', where f' = g. θ (f θ (x)); and based on f', it is possible to achieve the corresponding tasks in other tasks such as object detection and object segmentation.

[0187] See Figure 5As shown, this is a schematic diagram of the process of intelligent item sorting based on image classification in the application embodiment. The following is a detailed explanation in conjunction with the attached diagram. Figure 5 Taking a specific application scenario as an example, the image classification process during smart item sorting will be explained:

[0188] It should be noted that the technical solution proposed in this application can be understood as constructing an image classification model, including an image feature extraction network, a foreground enhancement network, and a classification network, based on a few-sample semantic filtering approach. Moreover, the foreground enhancement network in the image classification model can achieve semantic filtering and highlight the features of the target object in the image. When performing image classification, this application models the distance distribution between samples based on the idea of ​​metric learning, so that samples belonging to the same category are close together and samples belonging to different categories are far apart. In addition, a transformer-based foreground enhancement network tSF is proposed to improve the transferability of the image feature extraction network, so that the image feature extraction network focuses more on the target object region when extracting features and reduces the interference of background content.

[0189] When training the image classification model, this application uses a publicly available dataset for pre-training. By leveraging few-shot learning, the pre-trained target image classification model achieves a high classification accuracy. Furthermore, during the pre-training process, gradient descent training is performed on the image classification model based on the cross-entropy loss calculated from the predicted classification results and the corresponding classification labels (true values), thereby updating the model parameters for the entire image classification model.

[0190] according to Figure 5 As shown, when this application is applied to an intelligent item sorting system, each item to be sorted is sequentially fed onto a sorting conveyor belt, where an image acquisition device captures images of the items to be sorted, resulting in an image containing the items to be sorted. After acquiring the image to be sorted, the processing device acquires pre-constructed candidate images of various types, and then uses a target image classification model to determine the item types included in the image to be sorted based on the captured image to be sorted and the pre-constructed candidate images. The item types are then output, enabling the intelligent sorting system to sort the items to be sorted based on the determined item types.

[0191] Based on the same inventive concept, see [reference] Figure 6 As shown, this is a schematic diagram of the logical structure of an image classification device in an embodiment of this application. The image classification device 600 includes a first acquisition unit 601, a second acquisition unit 602, an input unit 603, a processing unit 604, and a classification unit 605.

[0192] The first acquisition unit 601 is used to acquire the image to be classified;

[0193] The second acquisition unit 602 is used to acquire various candidate images pre-constructed corresponding to various candidate sub-classification categories;

[0194] The input unit 603 is used to input the image to be classified into the pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and to input various candidate images into the target image feature extraction network to obtain the corresponding initial candidate image features.

[0195] The processing unit 604 is used to employ a pre-trained target foreground enhancement network to perform foreground feature enhancement processing on the initial image features to be classified and the initial candidate image features of various categories based on the category attribute features learned during the pre-training process for sample images of different categories, thereby obtaining the corresponding target image features to be classified and target candidate image features of various categories.

[0196] The classification unit 605 is used to determine the classification result of the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image by using a pre-trained target classification network.

[0197] Optionally, when performing foreground feature enhancement processing on the initial image to be classified based on the category attribute features learned during pre-training for sample images of different categories, the processing unit 604 is used for:

[0198] Based on the category attribute features learned for sample images of different categories during pre-training, the similarity relationship between the initial image features to be classified and the category attribute features is determined respectively.

[0199] Based on similarity relationships, the matching weights between the initial features of the image to be classified and the features of each category attribute are determined.

[0200] Based on the matching weights, the enhancement level of each category attribute feature for each position feature value in the initial image to be classified is determined, and based on each enhancement level value, foreground enhancement processing of the initial image to be classified is achieved.

[0201] Optionally, when determining the similarity relationship between the initial image features to be classified and the various category attribute features based on the category attribute features learned during pre-training for sample images of different categories, the processing unit 604 is used to:

[0202] The system acquires the category attribute features learned from sample images of different categories during pre-training, and adjusts the form of each category attribute feature to obtain the attribute feature matrix corresponding to each category attribute feature. It also adjusts the form of the initial image features to be classified to obtain the corresponding image feature matrix.

[0203] The matrix product between the attribute feature matrix and the image feature matrix is ​​calculated to obtain the similarity matrix, which represents the similarity relationship between the initial image features to be classified and the attribute features of each category.

[0204] Optionally, when determining the matching weights between the initial image features to be classified and each category attribute feature based on similarity, and when determining the enhancement level of each category attribute feature for each position feature value in the initial image features to be classified based on the matching weights, the processing unit 604 is used to:

[0205] The similarity values ​​at each position in the similarity matrix are normalized, and a weight parameter matrix representing the matching weight between the initial image features to be classified and the features of each category attribute is obtained based on the normalized similarity values ​​at each position.

[0206] Based on the matrix product between the feature matrix composed of each category attribute feature and the weight parameter matrix, the enhancement degree of each category attribute feature for the feature value at each position in the initial image to be classified is determined.

[0207] Optionally, when performing foreground enhancement processing on the features of the initial image to be classified based on various enhancement levels, the processing unit 604 is used for:

[0208] The enhancement values ​​at each location are superimposed with the feature values ​​at the corresponding locations in the initial image to be classified to obtain the superimposed feature values ​​for each location.

[0209] The feature values ​​corresponding to each position are superimposed and adaptively processed to enhance the foreground features in the initial image to be classified.

[0210] Optionally, when acquiring various candidate images pre-constructed corresponding to various candidate sub-classification categories, the second acquisition unit 602 is used for:

[0211] In response to the selection instruction for the relevant object, determine the various candidate subcategories to which the selection instruction is directed, and obtain various candidate images pre-constructed for each of the various candidate subcategories.

[0212] Optionally, when pre-training an image classification model that includes an image feature extraction network, a foreground enhancement network, and a classification network, the device further includes a training unit 606, which is used for:

[0213] Based on the open-source target dataset, the image classification model is iteratively pre-trained for multiple training cycles until the preset training cycle is reached. Within one training cycle, the following operations are performed:

[0214] Based on a set of sample images extracted from a subset of training data, the image classification model to be trained is iteratively trained for a set number of rounds to obtain a test image classification model. Based on test images extracted from a subset of test data, the classification accuracy of the test image classification model is determined. The training data subset and the test data subset are contained in the target dataset.

[0215] Based on the classification accuracy determined for the image classification models under test in different training periods, target image classification models that meet the preset screening conditions are selected from each image classification model under test.

[0216] Optionally, when extracting sample images from a subset of the training data, training unit 606 is used for:

[0217] In the training data subset, for each of the pre-selected N sub-classification categories, K sample images are extracted as reference samples for one class, and M non-overlapping sample images are extracted as samples to be classified.

[0218] Based on the extracted reference samples and samples to be classified, N*M sample image groups are constructed. Each sample image group includes a sample to be classified corresponding to a sub-classification category, where N, K, and M are positive integers.

[0219] Optionally, when training the image classification model to be trained for a set number of rounds based on a sample image group extracted from the training data subset to obtain the image classification model to be tested, the training unit 606 is used for:

[0220] Using the constructed sample image set, the image classification model to be trained is iteratively trained for a set number of rounds, and the model parameters of the image classification model are adjusted based on the loss values ​​calculated in each round of iteration. Specifically, the following operations are performed during each round of iterative training:

[0221] The acquired training sample set is input into the image classification model to be trained, and the predicted classification result of the sample to be classified in the corresponding training sample set is obtained. The loss value is calculated based on the classification difference between the predicted classification result and the corresponding classification label.

[0222] Optionally, when inputting a set of acquired training samples into the image classification model to be trained, and obtaining the predicted classification result of the samples to be classified in the corresponding training sample set, the training unit 605 is used for:

[0223] The image feature extraction network to be trained is input into the image feature extraction network of the training sample group, which includes the sample to be classified and the reference samples of each class.

[0224] A foreground enhancement network to be trained is used. Based on the category attribute features learned for each category, foreground feature enhancement processing is performed on the features of the sample image to be classified and the features of each reference sample image to obtain the features of the target sample image to be classified and the features of each target reference sample image.

[0225] A classification network to be trained is used to determine the classification result of the sample image to be classified based on the feature similarity between the features of the target sample image to be classified and the features of each target reference sample image.

[0226] Having introduced the image classification method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0227] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0228] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device, see reference. Figure 7 As shown, this is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 700 may include at least a processor 701 and a memory 702. The memory 702 stores program code, which, when executed by the processor 701, causes the processor 701 to perform any of the above-mentioned image classification steps.

[0229] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the image classification steps described above according to the various exemplary embodiments of this application. For example, the processor may perform actions such as... Figure 3A The steps are shown in the figure.

[0230] The following reference Figure 8 To describe a computing device 800 according to this embodiment of the present application. For example... Figure 8 As shown, the computing device 800 is presented in the form of a general-purpose computing device. The components of the computing device 800 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including storage unit 802 and processing unit 801).

[0231] Bus 803 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus structures.

[0232] Storage unit 802 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include read-only memory (ROM) 8023.

[0233] Storage unit 802 may also include a program / utility 8025 having a set (at least one) program module 8024, such program module 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0234] The computing device 800 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.), one or more devices that enable objects to interact with the computing device 800, and / or any device that enables the computing device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. As shown, network adapter 806 communicates with other modules for the computing device 800 via bus 803. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0235] Based on the same inventive concept as the above-described method embodiments, various aspects of the image classification provided in this application can also be implemented as a program product, which includes program code. When the program product is run on an electronic device, the program code causes the electronic device to perform the steps of the image classification method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 3A The steps are shown in the figure.

[0236] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0237] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0238] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image classification method, characterized in that, include: Obtain the image to be classified; Obtain various candidate images pre-constructed corresponding to various candidate sub-classification categories; The image to be classified is input into the pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and the candidate images of each category are input into the target image feature extraction network to obtain the corresponding initial candidate image features. A pre-trained target foreground enhancement network is used. Based on the attribute features of each category learned for sample images of different categories during the pre-training process, foreground feature enhancement processing is performed on the initial image features to be classified and the initial candidate image features of each category to be obtained, so as to obtain the corresponding target image features to be classified and the target candidate image features of each category. A pre-trained target classification network is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image.

2. The method as described in claim 1, characterized in that, Based on the category attribute features learned during pre-training for sample images of different categories, foreground feature enhancement processing is performed on the initial image features to be classified, including: Based on the category attribute features learned for sample images of different categories during pre-training, the similarity relationship between the initial image features to be classified and the category attribute features is determined respectively. Based on the similarity relationship, the matching weights between the initial image features to be classified and the features of each category attribute are determined; Based on the matching weights, the enhancement level of each category attribute feature for each position feature value in the initial image to be classified is determined, and based on each enhancement level value, foreground enhancement processing of the initial image to be classified is achieved.

3. The method as described in claim 2, characterized in that, The step of determining the similarity relationship between the initial image features to be classified and each category attribute feature, based on the category attribute features learned during pre-training for sample images of different categories, includes: The pre-training process acquires the category attribute features learned from sample images of different categories. The form of each category attribute feature, represented as n×c, is adjusted to obtain a c×n attribute feature matrix corresponding to each category attribute feature. Similarly, the form of the initial image features to be classified, represented as c×h×w, is adjusted to obtain a corresponding image feature matrix represented as hw×c. The matrix product between the attribute feature matrix and the image feature matrix is ​​calculated to obtain a similarity matrix characterizing the similarity relationship between the initial image features to be classified and each category attribute feature. Here, n is the total number of categories during pre-training, c is the number of channels, h is the height of the feature map, and w is the width of the feature map.

4. The method as described in claim 3, characterized in that, The step of determining the matching weights between the initial image features to be classified and each category attribute feature based on the similarity relationship, and determining the enhancement level of each category attribute feature for each positional feature value in the initial image features to be classified based on the matching weights, includes: The similarity values ​​at each position in the similarity matrix are normalized, and a weight parameter matrix representing the matching weight between the initial image features to be classified and the features of each category attribute is obtained based on the normalized similarity values ​​at each position. Based on the matrix product between the attribute feature matrix composed of each category attribute feature and the weight parameter matrix, the enhancement degree of each category attribute feature for the feature value at each position in the initial image to be classified is determined.

5. The method as described in claim 2, characterized in that, The foreground enhancement processing of the initial image features to be classified based on various enhancement levels includes: The enhancement value at each position is superimposed with the feature value at the corresponding position in the initial image features to be classified, to obtain the feature value superposition result corresponding to each position. The feature values ​​corresponding to each position are superimposed and adaptively processed to enhance the foreground features in the initial image to be classified.

6. The method as described in claim 1, characterized in that, The step of obtaining various candidate images pre-constructed corresponding to various candidate sub-classification categories includes: In response to a selection instruction for a relevant object, the various candidate subcategories to which the selection instruction is directed are determined, and various candidate images pre-constructed for each of the various candidate subcategories are acquired.

7. The method according to any one of claims 1-6, characterized in that, Pre-training is performed on an image classification model that includes an image feature extraction network, a foreground enhancement network, and a classification network, including: Based on the open-source target dataset, the image classification model is iteratively pre-trained for multiple training cycles until a preset training cycle is reached. Within one training cycle, the following operations are performed: Based on a set of sample images extracted from a subset of training data, the image classification model to be trained is iteratively trained for a set number of rounds to obtain a test image classification model, and based on test images extracted from a subset of test data, the classification accuracy of the test image classification model is determined, wherein the training data subset and the test data subset are contained in the target dataset. Based on the classification accuracy determined for the image classification models under test in different training periods, target image classification models that meet the preset screening conditions are selected from each image classification model under test.

8. The method as described in claim 7, characterized in that, Sample image groups are extracted from a subset of the training data, including: In the training data subset, for each of the pre-selected N sub-classification categories, K sample images are extracted as one class of reference samples, and M non-overlapping sample images are extracted as samples to be classified. Based on the extracted reference samples and unclassified samples, construct N There are M sample image groups, where each sample image group includes one sample to be classified corresponding to a subclassification category, and N, K, and M are positive integers.

9. The method as described in claim 8, characterized in that, The step of iteratively training the image classification model to be trained on a predetermined number of rounds based on a sample image group extracted from a subset of the training data to obtain the image classification model to be tested includes: Using the constructed sample image set, the image classification model to be trained is iteratively trained for a set number of rounds, and the model parameters of the image classification model are adjusted based on the loss values ​​calculated in each iteration. Specifically, the following operations are performed during one round of iterative training: A sample image group is input into the image classification model to be trained to obtain the predicted classification result of the sample to be classified in the sample image group, and the loss value is calculated based on the classification difference between the predicted classification result and the corresponding classification label.

10. The method as described in claim 9, characterized in that, The step of inputting a set of acquired sample images into an image classification model to be trained, and obtaining the predicted classification result for the samples to be classified in the set of sample images, includes: The image feature extraction network to be trained is input into the image feature extraction network of a sample image group, which includes the sample to be classified and the reference samples of each class, to obtain the image features of the sample to be classified and the image features of each reference sample. A foreground enhancement network to be trained is used. Based on the category attribute features learned for each category, foreground feature enhancement processing is performed on the features of the sample image to be classified and the features of each reference sample image to obtain the features of the target sample image to be classified and the features of each target reference sample image. A classification network to be trained is used to determine the classification result corresponding to the target sample image based on the feature similarity between the features of the target sample image to be classified and the features of each target reference sample image.

11. An image classification device, characterized in that, include: The first acquisition unit is used to acquire the image to be classified. The second acquisition unit is used to acquire various candidate images that are pre-constructed corresponding to various candidate sub-classification categories; The input unit is used to input the image to be classified into the pre-trained target image feature extraction network to obtain the corresponding initial image features to be classified, and to input the candidate images of each type into the target image feature extraction network to obtain the corresponding initial candidate image features. The processing unit is used to employ a pre-trained target foreground enhancement network to perform foreground feature enhancement processing on the initial image features to be classified and the initial candidate image features of each category based on the category attribute features learned during the pre-training process for sample images of different categories, thereby obtaining the corresponding target image features to be classified and target candidate image features of each category. The classification unit is used to determine the classification result corresponding to the image to be classified based on the feature similarity between the features of the target image to be classified and the features of each target candidate image, using a pre-trained target classification network.

12. The apparatus as claimed in claim 11, characterized in that, Based on the category attribute features learned during pre-training for sample images of different categories, when performing foreground feature enhancement processing on the initial image features to be classified, the processing unit is used to: Based on the category attribute features learned for sample images of different categories during pre-training, the similarity relationship between the initial image features to be classified and the category attribute features is determined respectively. Based on the similarity relationship, the matching weights between the initial image features to be classified and the features of each category attribute are determined; Based on the matching weights, the enhancement level of each category attribute feature for each position feature value in the initial image to be classified is determined, and based on each enhancement level value, foreground enhancement processing of the initial image to be classified is achieved.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image classification method as described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the image classification method as described in any one of claims 1-10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image classification method as described in any one of claims 1-10.