Model training method, image classification method and device

Abstract

An embodiment of the present disclosure provides a model training method, comprising: in response to obtaining a sample image, label information corresponding to the sample image, and a query image, constructing an image pair corresponding to the sample image and the query image, and constructing a few-shot learning network comprising a local feature extractor, a semantic feature extractor, and a classification discriminator; inputting the image pair into the local feature extractor to extract local features, to obtain first local features corresponding to the sample image and second local features corresponding to the query image; inputting the first local features and the second local features into the semantic feature extractor to extract semantic features, to obtain first semantic features corresponding to the first local features and second semantic features corresponding to the second local features; and inputting the first local features and the first semantic features, and the second local features and the second semantic features into the classification discriminator, respectively, training the few-shot learning network based on the label information corresponding to the sample image, and obtaining an image classification model.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology and internet technology, specifically to the fields of artificial intelligence technology and image processing technology, and particularly to model training methods, image classification methods and apparatus. Background Technology

[0002] With the continuous development and widespread application of deep neural networks, deep learning models have made rapid progress in the field of computer vision. However, this progress largely depends on massive amounts of labeled image samples from relevant categories. In real-world scenarios such as defect detection in industrial manufacturing, due to high costs, it is not always possible to obtain such a large number of image samples for new categories of interest. With the continuous development and advancement of neural networks, few-shot learning methods for rapid learning from a small number of labeled images have been proposed, currently mainly including: metric learning techniques, semantic alignment techniques, and feature aggregation techniques.

[0003] However, metric learning techniques typically aggregate features from all regions of an image without considering whether these regions are semantically relevant, thus reducing the discriminative power of the acquired features. Alternatively, semantic alignment techniques focus on exploring effective measures of local feature similarity between paired images, which are limited by background clutter, intra-class variations, or high computational costs. In few-shot learning, the scarcity of labeled images makes accurate semantic alignment even more difficult. Furthermore, feature aggregation techniques lack awareness of image semantics, focusing more attention on image regions irrelevant to the task. Summary of the Invention

[0004] Embodiments of this disclosure provide a model training method, an image classification method, an apparatus, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a model training method, which includes: in response to acquiring a sample image, annotation information corresponding to the sample image, and a query image, constructing an image pair corresponding to the sample image and the query image; constructing a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator; inputting the image pair into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image; inputting the first local feature and the second local feature into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature; inputting the first local feature, the first semantic feature, the second local feature, and the second semantic feature into the classification discriminator, and training the few-shot learning network based on the annotation information corresponding to the sample image to obtain an image classification model.

[0006] In some embodiments, the first local feature and the second local feature are respectively input into a semantic feature extractor for semantic feature extraction to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature. This includes: performing feature compression on the first local feature and the second local feature respectively through the semantic feature extractor to obtain the compressed first local feature and the compressed second local feature; and performing dimensional expansion and semantic feature extraction on the compressed first local feature and the compressed second local feature respectively through the semantic feature extractor to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

[0007] In some embodiments, the first local feature, the first semantic feature, the second local feature, and the second semantic feature are respectively input into a classification discriminator, and the few-shot learning network is trained based on the annotation information corresponding to the sample image to obtain an image classification model. This includes: inputting the first local feature, the first semantic feature, the second local feature, and the second semantic feature into a classification discriminator for feature aggregation to obtain a first aggregated feature corresponding to the first local feature and a second aggregated feature corresponding to the second local feature; and training the few-shot learning network based on the first aggregated feature, the second aggregated feature, and the annotation information corresponding to the sample image to obtain an image classification model.

[0008] In some embodiments, the first local feature and the first semantic feature, the second local feature and the second semantic feature are respectively input into a classification discriminator for feature aggregation to obtain a first aggregated feature corresponding to the first local feature and a second aggregated feature corresponding to the second local feature. This includes: calculating the weights of the first local feature and the first semantic feature using the classification discriminator to obtain a first weight of the first local feature; determining the first aggregated feature corresponding to the first local feature based on the first weight and the first local feature; calculating the weights of the second local feature and the second semantic feature using the classification discriminator to obtain a second weight of the second local feature; and determining the second aggregated feature corresponding to the second local feature based on the second weight and the second local feature.

[0009] In some embodiments, a few-shot learning network is trained based on a first aggregated feature, a second aggregated feature, and the annotation information corresponding to the sample images to obtain an image classification model, including: determining a cross-entropy loss function based on the first aggregated feature and the second aggregated feature; and jointly training the few-shot learning network based on the cross-entropy loss function and the annotation information corresponding to the sample images to obtain an image classification model.

[0010] Secondly, embodiments of this disclosure provide an image classification method, the method comprising: acquiring an image to be classified; inputting the image to be classified into an image classification model to obtain a classification result of the image to be classified, wherein the image classification model is acquired based on the method of the first aspect described above.

[0011] Thirdly, embodiments of this disclosure provide a model training apparatus, comprising: a construction module configured to, in response to acquiring a sample image, annotation information corresponding to the sample image, and a query image, construct an image pair corresponding to the sample image and the query image; construct a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator; a local feature extraction module configured to input the image pair into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image; a semantic feature extraction module configured to input the first local feature and the second local feature into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature; and a training module configured to input the first local feature and the first semantic feature, the second local feature, and the second semantic feature into the classification discriminator, and train the few-shot learning network based on the annotation information corresponding to the sample image to obtain an image classification model.

[0012] In some embodiments, the semantic feature extraction module is further configured to: compress the first local feature and the second local feature respectively using a semantic feature extractor to obtain the compressed first local feature and the compressed second local feature; and expand the dimension and extract semantic features from the compressed first local feature and the compressed second local feature respectively using a semantic feature extractor to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

[0013] In some embodiments, the training module includes: a feature aggregation unit configured to input a first local feature and a first semantic feature, a second local feature and a second semantic feature into a classification discriminator for feature aggregation, to obtain a first aggregated feature corresponding to the first local feature and a second aggregated feature corresponding to the second local feature; and a training unit configured to train a few-shot learning network based on the first aggregated feature, the second aggregated feature and the annotation information corresponding to the sample image, to obtain an image classification model.

[0014] In some embodiments, the feature aggregation unit is further configured to: calculate the weights of the first local feature and the first semantic feature using a classification discriminator to obtain a first weight of the first local feature; determine a first aggregated feature corresponding to the first local feature based on the first weight and the first local feature; calculate the weights of the second local feature and the second semantic feature using a classification discriminator to obtain a second weight of the second local feature; and determine a second aggregated feature corresponding to the second local feature based on the second weight and the second local feature.

[0015] In some embodiments, the training unit is further configured to: determine a cross-entropy loss function based on a first aggregated feature and a second aggregated feature; and jointly train a few-shot learning network based on the cross-entropy loss function and the annotation information corresponding to the sample images to obtain an image classification model.

[0016] Fourthly, embodiments of this disclosure provide an image classification apparatus, the apparatus comprising: an acquisition module configured to acquire an image to be classified; and a classification module configured to input the image to be classified into an image classification model to obtain a classification result of the image to be classified, wherein the image classification model is acquired based on the method described in the first aspect above.

[0017] Fifthly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any embodiment of the first or second aspect.

[0018] In a sixth aspect, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the embodiments of the first or second aspect.

[0019] The model training method provided in the embodiments of this disclosure involves the execution entity first responding to the acquisition of a sample image, corresponding annotation information of the sample image, and a query image, constructing an image pair corresponding to the sample image and the query image, and constructing a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator. Then, the image pair is input into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image. Subsequently, the first and second local features are input into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature. Finally... The first local feature, the first semantic feature, the second local feature, and the second semantic feature are respectively input into the classifier. Based on the annotation information corresponding to the sample images, the few-shot learning network is trained to obtain an image classification model. This model can extract corresponding semantic features based on local features, highlight image regions related to the current classification task, and suppress image regions unrelated to the current classification task. It can focus more on image regions related to the current classification task without considering other regions, thereby improving the accuracy of feature aggregation based on semantic features. This overcomes the shortcomings of metric learning techniques and avoids the difficulties of semantic alignment techniques, further improving the accuracy of the image classification model. Attached Figure Description

[0020] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0021] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0022] Figure 2 This is a flowchart of an embodiment of the model training method according to the present disclosure;

[0023] Figure 3 This is a flowchart of an embodiment of semantic feature extraction according to the present disclosure;

[0024] Figure 4 This is a flowchart of an embodiment of training a few-shot learning network according to the present disclosure;

[0025] Figure 5 This is a flowchart of one embodiment of feature aggregation according to the present disclosure;

[0026] Figure 6 This is a flowchart of an embodiment of obtaining an image classification model according to the present disclosure;

[0027] Figure 7 This is a flowchart of an embodiment of the image classification method according to the present disclosure;

[0028] Figure 8 This is a schematic diagram of the structure of an embodiment of the model training apparatus according to the present disclosure;

[0029] Figure 9 This is a schematic diagram of the structure of an embodiment of the image classification apparatus according to the present disclosure;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the disclosure are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1An exemplary system architecture 100 is shown that can be applied to model training methods, image classification methods, and apparatus according to embodiments of the present disclosure.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 104, 105, and 106, network 107, and servers 101, 102, and 103. Network 107 serves as the medium for providing communication links between terminal devices 104, 105, and 106 and servers 101, 102, and 103. Network 107 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0035] Users can interact with servers 101, 102, and 103 belonging to the same server cluster via network 107 through terminal devices 104, 105, and 106 to receive or send information. Various applications can be installed on terminal devices 104, 105, and 106, such as product display applications, data analysis applications, and search applications.

[0036] Terminal devices 104, 105, and 106 can be either hardware or software. When the terminal device is hardware, it can be any electronic device with a display screen that supports communication with the server, including but not limited to smartphones, tablets, laptops, and desktop computers. When the terminal device is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0037] Servers 101, 102, and 103 can be servers that provide various services, such as backend servers that receive requests sent by terminal devices with which they have established communication connections. The backend servers can receive and analyze the requests sent by the terminal devices and generate processing results.

[0038] Servers 101, 102, and 103 can acquire sample images, their corresponding annotation information, and query images. They construct image pairs corresponding to the sample and query images, and then build a few-shot learning network comprising a local feature extractor, a semantic feature extractor, and a classification discriminator. Servers 101, 102, and 103 can input the image pairs into the local feature extractor for local feature extraction, obtaining the first local feature corresponding to the sample image and the second local feature corresponding to the query image. The first and second local features are then input into the semantic feature extractor for semantic feature extraction, obtaining the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature. Finally, the first local feature, the first semantic feature, the second local feature, and the second semantic feature are input into the classification discriminator. Based on the annotation information corresponding to the sample images, the few-shot learning network is trained to obtain an image classification model.

[0039] Furthermore, servers 101, 102, and 103 can acquire the image to be classified and input it into the image classification model to obtain the classification result of the image to be classified. The image classification model is obtained based on the above method.

[0040] It should be noted that a server can be either hardware or software. When a server is hardware, it can be any electronic device that provides various services to terminal devices. When a server is software, it can be implemented as multiple software programs or software modules providing various services to terminal devices, or it can be implemented as a single software program or software module providing various services to terminal devices. No specific limitations are made here.

[0041] It should be noted that the model training method and image classification method provided in the embodiments of this disclosure can be executed by servers 101, 102, and 103. Accordingly, the model training device and the image classification device are disposed in servers 101, 102, and 103.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a model training method according to the present disclosure. The model training method includes the following steps:

[0044] Step 210: In response to obtaining the sample image, the annotation information corresponding to the sample image, and the query image, construct the image pair corresponding to the sample image and the query image.

[0045] In this step, the model training method runs on the execution entity (e.g., Figure 1 Servers 101, 102, and 103 can acquire images of various categories via network reading and other methods, with each category corresponding to multiple different images. The aforementioned execution entity can group the acquired images into sample image groups and query image groups. The sample image group can include multiple sample images of different categories, and the sample images are annotated to obtain annotation information. The sample image group serves as the support set for few-shot learning and can include images of N categories, with each category including M sample images. The query image group can include multiple unannotated query images and serves as the query set for few-shot learning.

[0046] After obtaining the sample image group and the query image group, the aforementioned executing entity can select a sample image from the sample image group and a query image from the query image group, and then combine the sample image and the query image into an image pair.

[0047] Step 220: Construct a few-shot learning network that includes a local feature extractor, a semantic feature extractor, and a classification discriminator.

[0048] In this step, the aforementioned execution entity can use neural networks such as Faster R-CNN as the main framework of the model, and on this basis, construct a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator.

[0049] Among them, the local feature extractor can be used to extract features from the input image and extract local features of the input image; the semantic feature extractor can be used to perform semantic analysis on the obtained local features and extract the corresponding semantic features; the classification discriminator can be used to analyze and process the input semantic features and local features to generate the classification result of the input image.

[0050] Step 230: Extract local features from the input local feature extractor of the image to obtain the first local feature corresponding to the sample image and the second local feature corresponding to the query image.

[0051] In this step, the aforementioned local feature extractor can be a commonly used Convolutional Neural Network (CNN) after removing the global pooling layer and its subsequent layers. The execution entity can input the constructed image pair into the local feature extractor, and the local feature extractor can extract local features from the sample image and the query image in the image pair respectively. Thus, the local feature extractor can extract local features from the sample image to obtain the first local feature corresponding to the sample image, and the local feature extractor can extract local features from the query image to obtain the second local feature corresponding to the query image.

[0052] Local feature extractors can be used to extract features from local features. This method is used to extract features from the corresponding sample image and query image. Here, θ1 represents the learnable parameters of the convolutional neural network used, and C, H, and W represent the channels, height, and width, respectively. By merging the spatial dimensions, HW local features of dimension C in both the sample image and the query image can be obtained, which can be represented as...

[0053] Step 240: Input the first local feature and the second local feature into the semantic feature extractor to extract semantic features, and obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

[0054] In this step, after the aforementioned execution entity obtains the first local feature corresponding to the sample image and the second local feature corresponding to the query image through the local feature extractor, it can input the first local feature and the second local feature into the semantic feature extractor respectively. According to the current classification task, the semantic extractor performs semantic analysis and semantic processing on the first local feature to obtain the first semantic feature corresponding to the first local feature. And, according to the current classification task, the semantic extractor performs semantic analysis and semantic processing on the second local feature to obtain the second semantic feature corresponding to the second local feature.

[0055] Step 250: Input the first local feature, the first semantic feature, the second local feature, and the second semantic feature into the classifier, respectively. Based on the annotation information corresponding to the sample image, train the few-shot learning network to obtain the image classification model.

[0056] In this step, after the execution entity obtains the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature, it can input the first local feature and the first semantic feature into the classification discriminator, and also input the second local feature and the second semantic feature into the classification discriminator. The classification discriminator performs classification processing on the first local feature and the first semantic feature, the second local feature and the second semantic feature respectively, and trains the few-shot learning network using machine learning methods based on the annotation information of the input sample image to obtain the image classification model.

[0057] The model training method provided in the embodiments of this disclosure involves the execution entity first responding to the acquisition of a sample image, corresponding annotation information of the sample image, and a query image, constructing an image pair corresponding to the sample image and the query image, and constructing a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator. Then, the image pair is input into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image. Subsequently, the first and second local features are input into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature. Finally... The first local feature, the first semantic feature, the second local feature, and the second semantic feature are respectively input into the classifier. Based on the annotation information corresponding to the sample images, the few-shot learning network is trained to obtain an image classification model. This model can extract corresponding semantic features based on local features, highlight image regions related to the current classification task, and suppress image regions unrelated to the current classification task. It can focus more on image regions related to the current classification task without considering other regions, thereby improving the accuracy of feature aggregation based on semantic features. This overcomes the shortcomings of metric learning techniques and avoids the difficulties of semantic alignment techniques, further improving the accuracy of the image classification model.

[0058] refer to Figure 3 , Figure 3 A flowchart 300 illustrating an embodiment of semantic feature extraction is shown, which may include the following steps:

[0059] Step 310: The first local feature and the second local feature are compressed by the semantic feature extractor to obtain the compressed first local feature and the compressed second local feature.

[0060] The semantic feature extractor may include a global pooling layer, three overlapping fully connected layers, and a ReLU layer.

[0061] In this step, after obtaining the first and second local features, the executing entity can input them into the semantic feature extractor. The first local features are then aggregated globally using a global pooling layer in the semantic feature extractor. Furthermore, the first local features are compressed using three overlapping fully connected layers and a ReLU layer in the semantic feature extractor. Based on the dimensionality reduction ratio, the first local features obtained through global pooling can be compressed into a compact feature space, reducing the dimension from C to D, where D is less than C. As an example, the executing entity can use principal component analysis (PCA) to compress the first local features, obtaining the compressed first local features. PCA, also known as principal component analysis, aims to transform multiple indicators into a few comprehensive indicators (i.e., principal components) using dimensionality reduction. Each principal component reflects most of the information of the original variables, and the information contained within them is non-overlapping.

[0062] Furthermore, the aforementioned execution entity performs global pooling aggregation on the second local features through the global pooling layer in the semantic feature extractor, and uses three overlapping fully connected layers and ReLU layers in the semantic feature extractor to compress the pooled second local features. Based on the dimensionality reduction ratio, the second local features obtained through global pooling aggregation can be compressed into a compact feature space, reducing the second local features from dimension C to dimension D, where the value of D is less than the value of C. As an example, the aforementioned execution entity can use image principal component analysis (PCA) to compress the second local features, obtaining the compressed second local features. PCA, also known as principal component analysis, aims to use dimensionality reduction to transform multiple indicators into a few comprehensive indicators (i.e., principal components), where each principal component reflects most of the information of the original variables, and the information contained is non-overlapping.

[0063] Step 320: The compressed first local feature and the compressed second local feature are expanded in dimension and extracted in semantic feature extraction by the semantic feature extractor to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

[0064] In this step, after the execution entity obtains the compressed first local feature and the compressed second local feature, it can further expand the dimension of the compressed first local feature through three overlapping fully connected layers and ReLU layers in the semantic feature extractor. That is, the dimension of the compressed first local feature can be expanded to the original dimension. For example, if the original dimension is C, the compressed first local feature can be expanded from the compressed dimension D to the original dimension C, where the value of D is less than the value of C. Then, the execution entity can perform semantic feature extraction on the expanded first local feature to the original dimension through the three overlapping fully connected layers and ReLU layers in the semantic feature extractor. The semantic feature extraction can be achieved using the following formula:

[0065]

[0066] in, θ2 and r are semantic feature extraction parameters. j The feature vector represents the local features, where H and W represent the height and width, respectively.

[0067] This disclosure can obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature by using a semantic feature acquisition method in related technologies, based on the compressed first local feature and the compressed second local feature.

[0068] In this implementation, image regions irrelevant to the current classification task are filtered out by compressing the first and second local features, thereby improving the discriminative power of the obtained features and the performance of the model.

[0069] refer to Figure 4 , Figure 4 A flowchart 400 illustrating an embodiment of training a few-shot learning network may include the following steps:

[0070] Step 410: Input the first local feature, the first semantic feature, the second local feature, and the second semantic feature into the classification discriminator for feature aggregation to obtain the first aggregated feature corresponding to the first local feature and the second aggregated feature corresponding to the second local feature.

[0071] In this step, after the execution entity obtains the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature, it can input the first local feature, the first semantic feature, the second local feature, and the second semantic feature into the classification discriminator for feature aggregation. Based on the first semantic feature, the classification discriminator performs feature aggregation on the first local feature to obtain the first aggregated feature corresponding to the first local feature. Based on the second semantic feature, the classification discriminator performs feature aggregation on the second local feature to obtain the second aggregated feature corresponding to the second local feature.

[0072] Step 420: Based on the first aggregated feature, the second aggregated feature, and the annotation information corresponding to the sample images, the few-shot learning network is trained to obtain the image classification model.

[0073] In this step, after the execution entity obtains the first aggregated feature and the second aggregated feature, it can classify the first aggregated feature and the second aggregated feature respectively through a classification discriminator, and train the few-shot learning network using machine learning methods based on the annotation information of the input sample image to obtain an image classification model.

[0074] In this implementation, by aggregating local features based on semantic features, the image regions relevant to the current classification task can be strengthened while other image regions are suppressed. This emphasizes the perception of image semantics and improves the accuracy of aggregated features.

[0075] Reference Figure 5 , Figure 5 A flowchart 500 illustrating one embodiment of feature aggregation is shown, which may include the following steps:

[0076] Step 510: The first weight of the first local feature is obtained by calculating the weights of the first local feature and the first semantic feature through a classification discriminator.

[0077] In this step, after the execution entity obtains the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature, it can calculate the weights of the first local feature and the first semantic feature through a classification discriminator, calculate the norm of the first local feature, and define the norm of the local feature as the attention value of the local feature. This attention value can be used as the first weight of the first local feature.

[0078] The aforementioned executing entity can calculate the norm of the first local feature using the following formula through the classification discriminator:

[0079] ||r j +f|| 2 =(||f||+||r j ||) 2 -2||f||||r j ||(1-cosθ j )

[0080] Where, θ j Representing f and r j The angle between them, r j Representing local features, it can be understood that for two local features r... j and r i If r j Ri If cosθ is more relevant to the semantically perceptual feature f, then j It should be greater than cosθ i This means that the above formula makes r j The norm increment is greater than r i The norm increment. There are two extreme cases as follows: 1) If r j If it is perfectly correlated with semantically perceived features, then the norm increment is ||f||; if r j If it is completely unrelated to semantic-aware features, then the norm increment is -||f||.

[0081] Step 520: Based on the first weight and the first local feature, determine the first aggregated feature corresponding to the first local feature.

[0082] In this step, after the execution entity obtains the first weight of the first local feature, the execution entity can calculate the first aggregate feature corresponding to the first local feature based on the first weight.

[0083] The aforementioned executing entity can use the classification discriminator to calculate the first aggregate feature corresponding to the first local feature using the following formula:

[0084]

[0085] Where, r j +f represents the norm of the first local feature, i.e., the first weight; H and W represent the height and width, respectively.

[0086] Step 530: The weights of the second local features and the second semantic features are calculated by the classification discriminator to obtain the second weights of the second local features.

[0087] In this step, the aforementioned execution entity can calculate the weights of the second local feature and the second semantic feature using a classification discriminator, calculate the norm of the second local feature, and define the norm of the local feature as the attention value of the local feature. This attention value can be used as the second weight of the second local feature.

[0088] The aforementioned executing entity can calculate the norm of the second local feature using the following formula through a classification discriminator:

[0089] ||r j +f|| 2 =(||f||+||r j ||) 2 -2||f||||r j ||(1-cosθ j )

[0090] Where, θ j Representing f and rj The angle between them, r j Representing local features, it can be understood that for two local features r... j and r i If r j R i If cosθ is more relevant to the semantically perceptual feature f, then j It should be greater than cosθ i This means that the above formula makes r j The norm increment is greater than r i The norm increment. There are two extreme cases as follows: 1) If r j If it is perfectly correlated with semantically perceived features, then the norm increment is ||f||; if r j If it is completely unrelated to semantic-aware features, then the norm increment is -||f||.

[0091] Step 540: Based on the second weight and the second local feature, determine the second aggregated feature corresponding to the second local feature.

[0092] In this step, after the execution entity obtains the second weight of the second local feature, the execution entity can calculate the second aggregate feature corresponding to the second local feature based on the second weight.

[0093] The aforementioned executing entity can use the classification discriminator to calculate the second aggregate feature corresponding to the second local feature using the following formula:

[0094]

[0095] Where, r j +f represents the norm of the second local feature, i.e., the second weight; H and W represent the height and width, respectively.

[0096] In this implementation, the weight value of each local feature is determined by semantic features, which avoids assigning more weight to image regions that are irrelevant to the current classification task and improves the accuracy of the weight values. Thus, using semantic features to determine the aggregated features can suppress image regions that are irrelevant to the current classification task and highlight image regions that are relevant to the current classification task, thereby improving the accuracy of feature aggregation.

[0097] refer to Figure 6 , Figure 6 A flowchart 600 illustrating an embodiment of obtaining an image classification model is shown, which may include the following steps:

[0098] Step 610: Determine the cross-entropy loss function based on the first aggregation feature and the second aggregation feature.

[0099] In this step, after the aforementioned execution entity obtains the data, it can determine the corresponding cross-entropy loss function based on the first aggregation feature and the second aggregation feature.

[0100] Specifically, after obtaining the first aggregated feature and the second aggregated feature, the aforementioned executing entity can calculate the average feature of the sample image:

[0101]

[0102] in, This represents the aggregated feature of the i-th sample image belonging to the m-th supporting category in the sample image group, where N is the total number of sample images in the sample image group.

[0103] The aforementioned execution entity can then calculate the probability that the k-th query image in the query image group belongs to the m-th supported category based on the following formula:

[0104]

[0105] Where d(·,·) represents the cosine distance, and τ represents the scale factor. This represents the aggregated feature of the Kth sample image belonging to the mth supporting category in the query image group.

[0106] Finally, the aforementioned execution entity can determine the cross-entropy loss function as follows:

[0107]

[0108] in, Let I(·) represent the label of the k-th query image, and let I(·) represent the indicator function, which is equal to 1 if its argument is true, and equal to 0 otherwise.

[0109] Step 620: Based on the cross-entropy loss function and the annotation information corresponding to the sample images, the few-shot learning network is jointly trained to obtain the image classification model.

[0110] In this step, after the aforementioned execution entity determines the cross-entropy loss function, it can use machine learning methods to jointly train the few-shot learning network based on the cross-entropy loss function and the annotation information of the input sample images, adjust the network parameters in the few-shot learning network, and obtain the image classification model.

[0111] In this implementation, the few-shot learning network is jointly trained using the cross-entropy loss function and the annotation information corresponding to the sample images to obtain an image classification model, thereby improving the training effect and accuracy of the image classification model.

[0112] refer to Figure 7 , Figure 7A flowchart 700 illustrating an embodiment of an image classification method may include the following steps:

[0113] Step 710: Obtain the image to be classified.

[0114] In this step, the aforementioned executing entity obtains the image to be classified through methods such as network reading.

[0115] Step 720: Input the image to be classified into the image classification model to obtain the classification result of the image to be classified.

[0116] In this step, after the aforementioned executing entity obtains the image to be classified, it can input the image to be classified into the image classification model. The image classification model processes the image to be classified and outputs a classification result corresponding to the image to be classified. This classification result can characterize the image category information of the image to be classified. The aforementioned image classification model is based on the above... Figures 2-6 The model training method is obtained.

[0117] The image classification apparatus provided in the above embodiments of this disclosure first acquires an image to be classified, then inputs the image to be classified into an image classification model to obtain a classification result of the image to be classified. The image classification model is acquired based on the above method. By classifying the image to be classified through the image classification model, the classification efficiency and accuracy of the image to be classified can be improved.

[0118] Further reference Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model training apparatus. This apparatus embodiment is similar to... Figure 2 The method embodiments shown correspond to those described.

[0119] like Figure 8 As shown, the model training device 800 in this embodiment may include: a construction module 810, a local feature extraction module 820, a semantic feature extraction module 830, and a training module 840.

[0120] The construction module 810 is configured to, in response to obtaining a sample image, the annotation information corresponding to the sample image, and a query image, construct an image pair corresponding to the sample image and the query image; and construct a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator.

[0121] The local feature extraction module 820 is configured to input the image pair into the local feature extractor to extract local features, thereby obtaining a first local feature corresponding to the sample image and a second local feature corresponding to the query image.

[0122] The semantic feature extraction module 830 is configured to input the first local feature and the second local feature into the semantic feature extractor respectively to extract semantic features, thereby obtaining the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature;

[0123] The training module 840 is configured to input the first local feature and the first semantic feature, the second local feature and the second semantic feature into the classification discriminator respectively, and train the few-shot learning network based on the annotation information corresponding to the sample image to obtain an image classification model.

[0124] In some optional implementations of this embodiment, the semantic feature extraction module 830 is further configured to: compress the first local feature and the second local feature respectively through the semantic feature extractor to obtain the compressed first local feature and the compressed second local feature; and expand the dimension and extract semantic features from the compressed first local feature and the compressed second local feature respectively through the semantic feature extractor to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

[0125] In some optional implementations of this embodiment, the training module 840 includes: a feature aggregation unit configured to input the first local feature and the first semantic feature, the second local feature and the second semantic feature into the classification discriminator respectively for feature aggregation, to obtain the first aggregated feature corresponding to the first local feature and the second aggregated feature corresponding to the second local feature; and a training unit configured to train the few-shot learning network based on the first aggregated feature, the second aggregated feature and the annotation information corresponding to the sample image to obtain the image classification model.

[0126] In some optional implementations of this embodiment, the feature aggregation unit is further configured to: calculate the weights of the first local feature and the first semantic feature using a classification discriminator to obtain the first weight of the first local feature; determine the first aggregated feature corresponding to the first local feature based on the first weight and the first local feature; calculate the weights of the second local feature and the second semantic feature using a classification discriminator to obtain the second weight of the second local feature; and determine the second aggregated feature corresponding to the second local feature based on the second weight and the second local feature.

[0127] In some optional implementations of this embodiment, the training unit is further configured to: determine the cross-entropy loss function based on the first aggregated feature and the second aggregated feature; and jointly train the few-shot learning network based on the cross-entropy loss function and the annotation information corresponding to the sample images to obtain an image classification model.

[0128] The model training apparatus provided in the above embodiments of this disclosure, wherein the execution entity first responds to acquiring a sample image, the corresponding annotation information of the sample image, and a query image, constructs an image pair corresponding to the sample image and the query image, and constructs a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator. Then, the image pair is input into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image. Subsequently, the first local feature and the second local feature are respectively input into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature. The first local feature, the first semantic feature, the second local feature, and the second semantic feature are then input into the classifier. Based on the annotation information corresponding to the sample images, the few-shot learning network is trained to obtain an image classification model. This model can extract corresponding semantic features based on local features, highlight image regions related to the current classification task, and suppress image regions unrelated to the current classification task. It focuses more on image regions related to the current classification task without considering other regions, thereby improving the accuracy of feature aggregation based on semantic features. This overcomes the shortcomings of metric learning techniques and avoids the difficulties of semantic alignment techniques, further improving the accuracy of the image classification model.

[0129] Those skilled in the art will understand that the above-described apparatus also includes other well-known structures, such as processors and memories. To avoid unnecessarily obscuring the embodiments of this disclosure, these well-known structures are... Figure 8 Not shown in the image.

[0130] Further reference Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an image classification device. This device embodiment is similar to... Figure 7 The method embodiments shown correspond to those described.

[0131] like Figure 9 As shown, the image classification device 900 of this embodiment may include an acquisition module 910 and a classification module 920.

[0132] Among them, the acquisition module 910 is configured to acquire the image to be classified;

[0133] The classification module 920 is configured to input the image to be classified into an image classification model to obtain a classification result for the image to be classified, wherein the image classification model is based on the above... Figure 2-6 The method shown is used to obtain it.

[0134] The image classification apparatus provided in the above embodiments of this disclosure first acquires an image to be classified, then inputs the image to be classified into an image classification model to obtain a classification result of the image to be classified. The image classification model is acquired based on the above method. By classifying the image to be classified through the image classification model, the classification efficiency and accuracy of the image to be classified can be improved.

[0135] Those skilled in the art will understand that the above-described apparatus also includes other well-known structures, such as processors and memories. To avoid unnecessarily obscuring the embodiments of this disclosure, these well-known structures are... Figure 9 Not shown in the image.

[0136] The following is for reference. Figure 10 The diagram illustrates a structural schematic of an electronic device 1000 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as smart screens, laptops, tablets, PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The terminal device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0137] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0138] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 10 Each box shown can represent a device or multiple devices as needed.

[0139] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium of embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor. For example, a processor may include a construction module, a local feature extraction module, a semantic feature extraction module, and a training module; or, a processor may include an acquisition module and a classification module. The names of these modules do not necessarily limit the functionality of the module itself.

[0143] In another aspect, this application also provides a computer-readable medium, which may be included in the aforementioned electronic device or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: in response to acquiring a sample image, corresponding annotation information of the sample image, and a query image, construct an image pair corresponding to the sample image and the query image; construct a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator; input the image pair into the local feature extractor for local feature extraction to obtain a first local feature corresponding to the sample image and a second local feature corresponding to the query image; input the first local feature and the second local feature into the semantic feature extractor for semantic feature extraction to obtain a first semantic feature corresponding to the first local feature and a second semantic feature corresponding to the second local feature; input the first local feature, the first semantic feature, the second local feature, and the second semantic feature into the classification discriminator, and train the few-shot learning network based on the annotation information corresponding to the sample image to obtain an image classification model. Alternatively, the electronic device may: acquire an image to be classified; input the image to be classified into an image classification model to obtain the classification result of the image to be classified, wherein the image classification model is acquired based on a model training method.

[0144] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A model training method, the method comprising: In response to obtaining a sample image, the annotation information corresponding to the sample image, and a query image, an image pair corresponding to the sample image and the query image is constructed; Construct a few-shot learning network that includes a local feature extractor, a semantic feature extractor, and a classification discriminator; The image is input to the local feature extractor to extract local features, thereby obtaining the first local feature corresponding to the sample image and the second local feature corresponding to the query image; The first local feature and the second local feature are respectively input into the semantic feature extractor for semantic feature extraction to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature; The first local feature and the first semantic feature, the second local feature and the second semantic feature are respectively input into the classification discriminator for feature aggregation to obtain the first aggregated feature corresponding to the first local feature and the second aggregated feature corresponding to the second local feature; Based on the first aggregated feature, the second aggregated feature, and the annotation information corresponding to the sample image, the few-shot learning network is trained to obtain an image classification model.

2. The method according to claim 1, wherein, The step of inputting the first local feature and the second local feature into the semantic feature extractor for semantic feature extraction, respectively, to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature, includes: The semantic feature extractor performs feature compression on the first local feature and the second local feature respectively to obtain the compressed first local feature and the compressed second local feature; The semantic feature extractor performs dimensional expansion and semantic feature extraction on the compressed first local feature and the compressed second local feature respectively, to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

3. The method according to claim 1, wherein inputting the first local feature and the first semantic feature, the second local feature and the second semantic feature into the classification discriminator for feature aggregation to obtain a first aggregated feature corresponding to the first local feature and a second aggregated feature corresponding to the second local feature, comprises: The first weight of the first local feature is obtained by calculating the weights of the first local feature and the first semantic feature using the classification discriminator. Based on the first weight and the first local feature, the first aggregated feature corresponding to the first local feature is determined; The classification discriminant calculates the weights of the second local feature and the second semantic feature to obtain the second weight of the second local feature. Based on the second weight and the second local feature, the second aggregated feature corresponding to the second local feature is determined.

4. The method according to claim 1, wherein, The step of training the few-shot learning network based on the first aggregated feature, the second aggregated feature, and the annotation information corresponding to the sample image to obtain an image classification model includes: Based on the first aggregation feature and the second aggregation feature, determine the cross-entropy loss function; Based on the cross-entropy loss function and the annotation information corresponding to the sample images, the few-shot learning network is jointly trained to obtain an image classification model.

5. An image classification method, the method comprising: Obtain the image to be classified; The image to be classified is input into an image classification model to obtain the classification result of the image to be classified, wherein the image classification model is obtained based on the method of claims 1-4 above.

6. A model training apparatus, the apparatus comprising: The construction module is configured to, in response to obtaining a sample image, the annotation information corresponding to the sample image, and a query image, construct an image pair corresponding to the sample image and the query image; and construct a few-shot learning network including a local feature extractor, a semantic feature extractor, and a classification discriminator. The local feature extraction module is configured to input the image pair into the local feature extractor to extract local features, thereby obtaining a first local feature corresponding to the sample image and a second local feature corresponding to the query image. The semantic feature extraction module is configured to input the first local feature and the second local feature into the semantic feature extractor respectively to extract semantic features, thereby obtaining the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature; The training module includes a feature aggregation unit and a training unit. The feature aggregation unit is configured to input the first local feature and the first semantic feature, the second local feature and the second semantic feature into the classification discriminator respectively for feature aggregation, so as to obtain a first aggregated feature corresponding to the first local feature and a second aggregated feature corresponding to the second local feature. The training unit is configured to train the few-shot learning network based on the first aggregated feature, the second aggregated feature, and the annotation information corresponding to the sample image to obtain an image classification model.

7. The apparatus according to claim 6, wherein, The semantic feature extraction module is further configured to: The semantic feature extractor performs feature compression on the first local feature and the second local feature respectively to obtain the compressed first local feature and the compressed second local feature; The semantic feature extractor performs dimensional expansion and semantic feature extraction on the compressed first local feature and the compressed second local feature respectively, to obtain the first semantic feature corresponding to the first local feature and the second semantic feature corresponding to the second local feature.

8. The apparatus according to claim 6, wherein, The feature aggregation unit is further configured as follows: The first weight of the first local feature is obtained by calculating the weights of the first local feature and the first semantic feature using the classification discriminator. Based on the first weight and the first local feature, the first aggregated feature corresponding to the first local feature is determined; The classification discriminant calculates the weights of the second local feature and the second semantic feature to obtain the second weight of the second local feature. Based on the second weight and the second local feature, the second aggregated feature corresponding to the second local feature is determined.

9. The apparatus according to claim 6, wherein, The training unit is further configured as follows: Based on the first aggregation feature and the second aggregation feature, determine the cross-entropy loss function; Based on the cross-entropy loss function and the annotation information corresponding to the sample images, the few-shot learning network is jointly trained to obtain an image classification model.

10. An image classification apparatus, the apparatus comprising: The acquisition module is configured to acquire images to be classified. The classification module is configured to input the image to be classified into an image classification model to obtain the classification result of the image to be classified, wherein the image classification model is obtained based on the method of claims 1-4 above.

11. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images