Object classification method, device, computer readable medium and electronic device

Abstract

The embodiments of the present application provide an object classification method, device, computer-readable medium and electronic device, the method comprising: extracting a first depth feature corresponding to the first image sample data of each category in the first sample data set through an object classification model; extracting a second depth feature of the image to be classified in the object classification model; constructing a first feature pair corresponding to each first depth feature; encoding the first depth feature and the second depth feature respectively based on the attention information between the first depth feature and the second depth feature in the first feature pair to obtain a second feature pair including the encoded first depth feature and the encoded second depth feature; determining the object classification result according to the distance information corresponding to each second feature pair. The embodiments of the present application reduce background interference and improve the accuracy of object classification. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to an object classification method, device, computer-readable medium, and electronic device. Background Art

[0002] With the development of artificial intelligence, object classification models based on deep learning technology have been applied to various fields.

[0003] Existing object classification models have poor ability to distinguish between objects and backgrounds in images, and are easily disturbed by the background of the image when classifying the image, resulting in low classification accuracy of the object classification model. Summary of the invention

[0004] The embodiments of the present application provide an object classification method, apparatus, computer-readable medium, and electronic device, which can reduce background interference at least to a certain extent, thereby improving the accuracy of object classification.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, an object classification method is provided, the method comprising: extracting a first depth feature corresponding to first image sample data of each category in a first sample data set through an object classification model, wherein the object classification model is obtained by training based on the first sample data set, and the first sample data set includes first image sample data belonging to multiple categories respectively; inputting an image to be classified into the object classification model to extract a second depth feature of the image to be classified in the object classification model; constructing a first feature pair corresponding to each first depth feature, the first feature pair including the first depth feature and the second depth feature; for each first feature pair, based on attention information between the first depth feature and the second depth feature in the first feature pair, encoding the first depth feature and the second depth feature respectively to obtain a second feature pair including the encoded first depth feature and the encoded second depth feature, the attention information being used to measure the attention strength of one depth feature to another depth feature; determining the distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair, and determining the object classification result of the image to be classified according to the distance information corresponding to each second feature pair.

[0007] According to one aspect of an embodiment of the present application, an object classification device is provided, the device comprising: an extraction unit, configured to extract a first depth feature corresponding to first image sample data of each category in a first sample data set through an object classification model, wherein the object classification model is obtained by training based on the first sample data set, and the first sample data set comprises first image sample data belonging to a plurality of categories respectively; an input unit, configured to input an image to be classified into the object classification model, so as to extract a second depth feature of the image to be classified in the object classification model; a construction unit, configured to construct a first feature pair corresponding to each first depth feature, the first feature pair comprising the first depth feature and the second depth feature; an encoding unit, configured to encode the first depth feature and the second depth feature respectively based on attention information between the first depth feature and the second depth feature in the first feature pair for each first feature pair, so as to obtain a second feature pair comprising an encoded first depth feature and an encoded second depth feature, wherein the attention information is used to measure the attention strength of one depth feature to another depth feature; a classification unit, configured to determine distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair, and determine an object classification result of the image to be classified according to the distance information corresponding to each second feature pair.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the encoding configuration is: encoding the first depth feature based on the attention information of the first depth feature to the second depth feature and the attention information of the second depth feature to the first depth feature; encoding the second depth feature based on the attention information of the second depth feature to the first depth feature and the attention information of the first depth feature to the second depth feature.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit is configured to: extract the original first depth features corresponding to the first image sample data of each category in the first sample data set through the backbone network of the object classification model, and input the original first depth features into the local classifier of the object classification model to obtain the first prediction probability vector output by the local classifier; according to the position of the maximum probability in the first prediction probability vector, extract the first feature vector in the weight matrix of the local classifier, and update the original first depth features according to the first feature vector to obtain the first depth features; the input unit is configured to: extract the original second depth features corresponding to the image to be classified through the backbone network of the object classification model, and input the original second depth features into the local classifier of the object classification model to obtain the second prediction probability vector output by the local classifier; according to the position of the maximum probability in the second prediction probability vector, extract the second feature vector in the weight matrix of the local classifier, and update the original second depth features according to the second feature vector to obtain the second depth features.

[0010] In some embodiments of the present application, based on the aforementioned scheme, before extracting the first depth feature corresponding to the first image sample data of each category in the first sample data set through the object classification model, the extraction unit is also used to: train the pre-trained object classification model according to the first sample data set to obtain the object classification model.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit is configured to: initialize the weight matrix of the original local classifier of the pre-trained object classification model according to the original first depth features corresponding to the first image sample data of each category; train the original local classifier based on the first sample data set to obtain a local classifier.

[0012] In some embodiments of the present application, based on the aforementioned scheme, before training the pre-trained object classification model according to the first sample data set to obtain the object classification model, the extraction unit is also used to: pre-train the original object classification model according to the second sample data set including multiple second sample data to obtain the pre-trained object classification model.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the second sample data set includes a training set and a validation set, and the second sample data in the training set and the validation set belong to different categories, and the extraction unit is configured to: perform multiple task data set extraction operations on the training set of the second sample data set to randomly extract a first support set and a first query set through each task data set extraction operation, the first support set and the first query set both include second sample data corresponding to a predetermined number of categories, and the predetermined number of categories corresponding to the first support set and the first query set are the same; pre-train the original object classification model based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the original object classification model also includes a global classifier and a metric classifier, and the extraction unit is configured to: input the extracted first support set and the first query set into the original object classification model to generate enhanced first depth features corresponding to the second image sample data of each category in the first support set and multiple enhanced second depth features corresponding to the second image sample data in the first query set; input the enhanced first depth features and the enhanced second depth features into the metric classifier to obtain the first object classification result of the second image sample data in the first query set; determine the metric classifier loss according to the first object classification result corresponding to the second image sample data in the first query set and the category of the second image sample data; input the enhanced second depth features corresponding to the category of the second image sample data among the multiple enhanced second depth features into the global classifier to obtain the second object classification result of the second image sample data; determine the global classifier loss according to the second object classification result of the second image sample data and the category of the second image sample data; pre-train the original object classification model according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the original object classification model also includes a rotation classifier. When executing the step of inputting the extracted first support set and the first query set into the original object classification model, the extraction unit is further configured to: rotate the second image sample data in the first query set by multiple predetermined angles respectively to obtain multiple rotated first query sets; input the extracted first support set and the rotated first query set into the original object classification model to obtain the angle classification result corresponding to the second image sample data in the rotated first query set output by the rotation classifier; when executing the step of pre-training the original object classification model according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model, the extraction unit is further configured to: determine the rotation classifier loss according to the angle classification result corresponding to the second image sample data and the predetermined angle by which the second image sample data is rotated; obtain the final loss according to the metric classifier loss, the global classifier loss and the rotation classifier loss, and update the parameters of the original object classification model according to the final loss to obtain a pre-trained object classification model.

[0016] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the object classification method as described in the above embodiment is implemented.

[0017] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the object classification method as described in the above embodiments.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the object classification method described in the above embodiments.

[0019] In the technical solutions provided in some embodiments of the present application, first, the first depth features corresponding to the first image sample data of each category in the first sample data set and the second depth features of the image to be classified are respectively extracted in the object classification model, and the corresponding first feature pairs are obtained by combining the second depth features with each first depth feature respectively; then, the first depth features and the second depth features are respectively encoded based on the attention information between the first depth features and the second depth features in each first feature pair to obtain the second feature pairs, and attention enhancement processing is implemented on the depth features in the first feature pairs, so that the encoded first depth features and the encoded second depth features in the second feature pairs can highlight and enhance the features of the target object. Therefore, the model can pay more attention to the target object in the image when extracting the encoded depth features, and significantly reduce the interference of the background image; on this basis, by determining the distance information corresponding to each second feature pair and determining the object classification result of the image to be classified according to the distance information corresponding to each second feature pair, the classification accuracy of the object classification model can be improved.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A schematic diagram of the principle of a small sample classification method based on metric learning in the related art is shown;

[0023] Figure 2 A schematic diagram showing an exemplary application scenario to which the technical solution of the embodiments of the present application can be applied;

[0024] Figure 3 A flow chart of an object classification method according to an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of the overall framework of an algorithm for implementing object classification according to an embodiment of the present application is shown;

[0026] Figure 5 An embodiment according to the present application is shown. Figure 3 Flowchart of steps before step 330 in the embodiment;

[0027] Figure 6 An embodiment according to the present application is shown. Figure 5 A flowchart of the details of step 320 in an embodiment;

[0028] Figure 7 An embodiment according to the present application is shown. Figure 5 Flowchart of steps before step 320 in the embodiment;

[0029] Figure 8 An embodiment according to the present application is shown. Figure 7 A flowchart of the details of step 310 in an embodiment;

[0030] Fig. 9 A flowchart of pre-training an original object classification model based on a first support set and a first query set extracted through multiple task data set extraction operations according to an embodiment of the present application is shown;

[0031] Fig.10 An embodiment according to the present application is shown. Figure 3 A flowchart of the details of step 360 in an embodiment;

[0032] Fig.11 A schematic diagram of a Transformer cross-correlation attention module according to an embodiment of the present application is shown;

[0033] Fig.12 A schematic diagram showing the principle of an old-class-new-class feature interaction module according to an embodiment of the present application is shown;

[0034] Fig.13 A block diagram of an object classification device according to an embodiment of the present application is shown;

[0035] Fig.14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0040] Due to a series of factors such as privacy, security or high data labeling costs, many real-world application scenarios (such as in industry, medicine, and finance) do not have the conditions to obtain enough labeled training samples. Therefore, how to enable a deep learning system to efficiently learn and generalize its cognitive ability from a very small number of samples has important significance and application value. Compared with ordinary deep learning methods that require a large amount of labeled data, small sample learning methods only require a small number of samples to learn high-performance models.

[0041] In the related technologies, there are three main small sample classification methods based on metric learning, namely: MatchingNet, ProtoNet and RelationNet. Figure 1 The schematic diagram shows the principle of the small sample classification method based on metric learning in the related art. Figure 1 As shown, Figure 1 (a) Figure 1 (b) Figure 1 (c) The structural principle diagrams of MatchingNet, ProtoNet and RelationNet, respectively, where S is a small set of training samples, Q is the sample to be classified, and f θ For the backbone network, is the classification result. Figure 1 Taking MatchingNet shown in (a) as an example, the specific process of this solution is as follows: First, use the backbone network fθ Perform deep feature extraction on a small number of input training sample sets S and samples to be classified Q to obtain the corresponding features f S and f Q ; Then, the cosine distance is used as a similarity measure to calculate f S and f Q The similarity between Q The most similar f S The corresponding category is the classification result of the sample Q to be classified The basic principles of ProtoNet and RelationNet are roughly the same as those of MatchingNet, except that in ProtoNet and RelationNet, the mean vector μ of each type of sample needs to be calculated, and ProtoNet measures similarity by Euclidean distance, while RelationNet uses the Relation Module to measure similarity.

[0042] Although the above three small sample classification methods based on metric learning have achieved small sample classification, they have at least the following defects:

[0043] First, the ability to distinguish between the target object and the background is poor and is easily disturbed by the background.

[0044] Second, the backbone network has poor generalization performance, and the extracted features are not representative enough, making it difficult to distinguish between samples within and between classes, resulting in low classification accuracy.

[0045] To this end, the present application first provides an object classification method. The object classification method provided based on the embodiment of the present application can overcome the above-mentioned defects. The object classification method can not only be applied to small sample classification scenarios, that is, when there are only a few images per class in the sample data used for training the model, it can still achieve a high classification accuracy, and can effectively distinguish the target object from the background, reduce the interference of the background, and can also improve the representation ability of the deep features, thereby improving the performance of object classification as a whole.

[0046] Figure 2 A schematic diagram showing an exemplary application scenario to which the technical solution of the embodiment of the present application can be applied. Figure 2 As shown, the embodiment of the present application is applied to the item sorting process in the logistics industry, and specifically to the intelligent item sorting system. The intelligent item sorting system can be used to classify various types of items (such as cucumbers, cabbages, eggplants, tomatoes and other vegetables). Figure 2The application scheme shown is as follows: the intelligent item sorting system is used to sort the items to be classified 210 according to the categories of the items. The intelligent item sorting system includes a camera 220, a system brain 230 and a sorting mechanism 240, wherein the camera 220 establishes a communication connection with the system brain 230, and the system brain 230 establishes a communication connection with the sorting mechanism 240, and the system brain 230 is an implementation terminal of the embodiment of the present application. The specific process of the intelligent item sorting system sorting the items to be classified 210 is as follows: first, the camera 220 shoots the items to be classified 210, and transmits the captured image to the system brain 230; then, the system brain 230 inputs the received image into the small sample classification algorithm provided in the embodiment of the present application, and obtains the classification result of the items in the image output by the small sample classification algorithm; finally, the system brain 230 sends the classification result to the sorting mechanism 240, and the sorting mechanism 240 sorts the items to be classified 210 to the corresponding position according to the classification result.

[0047] In some embodiments of the present application, the small sample classification algorithm is a small sample classification model pre-trained based on a number of image sample data, and the category of the objects recorded in the number of image sample data is consistent with the category of the object 210 to be classified.

[0048] In some embodiments of the present application, the object to be classified 210 will move, and the camera 220 can capture multiple images of the object to be classified 210 during the movement, and send the multiple images to the system brain 230; the system brain 230 will determine the classification results corresponding to each image respectively through a small sample classification algorithm, and determine the final classification result based on the multiple classification results; the system brain 230 will send the final classification result to the sorting mechanism 240.

[0049] In some embodiments of the present application, the system brain 230 is deployed on a server.

[0050] It should be understood that Figure 2 The number of cameras, system brains and sorting mechanisms in the embodiment is only illustrative. According to the implementation requirements, any number of cameras, system brains and sorting mechanisms can be provided. For example, multiple cameras can be used to take multiple images of the object to be classified 210 at different angles from different angles, and the system brain can obtain the classification result based on the multiple images; there can be multiple system brains, which are respectively deployed on multiple servers in the server cluster; there can also be multiple sorting mechanisms, so as to achieve large-scale sorting.

[0051] It should be noted that Figure 2 The embodiment shown is only one embodiment of the present application. Figure 2In the solution of the embodiment, the system brain 230 is deployed on a server, but in other embodiments of the present application, the system brain 230 is deployed on various types of terminal devices such as smart phones, desktop computers, workstations, tablet computers, and vehicle-mounted terminals. That is, the implementation terminal of the embodiment of the present application can be the various terminal devices mentioned above; although in Figure 2 In the scheme of the embodiment, the object classification method is applied to the intelligent object sorting system, but in other embodiments of the present application, the object classification method can also be applied to various scenarios that require classification, such as the diagnosis scenario of lung nodules; although in Figure 2 In the scheme of the embodiment, the object classification method is used to classify physical objects, but in other embodiments of the present application, it can also be applied to the scene of classifying virtual objects, for example, it can be applied to the scene of classifying abstract graphics in images. The embodiment of the present application does not make any limitation to this, and the protection scope of the present application should not be limited in any way.

[0052] It is easy to understand that the object classification method provided in the embodiment of the present application is generally executed by the server, and accordingly, the object classification device is generally set in the server. However, in other embodiments of the present application, the terminal device can also have similar functions as the server, so as to execute the object classification scheme provided in the embodiment of the present application.

[0053] Therefore, the embodiments of the present application can be applied to a terminal or a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0054] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0055] Figure 3 The flowchart of an object classification method according to an embodiment of the present application is shown. The object classification method can be executed by various devices capable of computing and processing, such as user terminals or cloud servers. User terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, portable devices, etc. The embodiment of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 3As shown, the object classification method at least includes the following steps:

[0056] In step 330, a first depth feature corresponding to the first image sample data of each category in the first sample data set is extracted through an object classification model, wherein the object classification model is trained based on the first sample data set, and the first sample data set includes first image sample data belonging to multiple categories respectively.

[0057] The object classification model is a trained classification model that can classify images to determine the categories of objects contained in the images. The objects contained in the images can be various types of physical objects such as fruits, vegetables, household appliances, furniture, vehicles, kitchen utensils, etc., and can also be virtual objects such as symbols.

[0058] The first sample data set includes multiple first image sample data, each of which belongs to only one category. It is easy to understand that the category to which the first image sample data belongs is the category to which the object contained in the first image sample data belongs. The first sample data set includes first image sample data corresponding to multiple categories respectively; for each category, the first sample data set includes one or more first image sample data corresponding to the category. The number of first image sample data corresponding to each category may not exceed 10, or even may not exceed 5. The category of the first image sample data in the first sample data set is known, and the category is usually set in the form of a label.

[0059] The first sample data set D novel It can be a project data set used in actual application of the project.

[0060] In one embodiment of the present application, the first image sample data belonging to each category in the first sample data set are the same.

[0061] Taking the solution of the embodiment of the present application as an example of a small sample classification task, the small sample task is an N-way K-shot classification task, where N-way refers to N categories and K-shot refers to the number of labeled samples K provided for each category. Then, the first sample data set D novel N categories can be set in, and each category includes K labeled first image sample data, the first sample data set D novel There are N*K first image sample data in total.

[0062] Combine the following Figure 4 The overall algorithm framework shown further introduces the solution of the embodiment of the present application. Figure 4 The schematic diagram of the overall framework of the algorithm for implementing object classification according to an embodiment of the present application is shown. Figure 4As shown, the overall framework of the algorithm is the object classification model, which specifically includes the backbone network f θ , TMNAM (Transformer Mutual-Nearest Attention Module), BNI (Base-Novel Interaction) including Local Classifier, Metric Classifier, Global Classifier, Rotation Classifier and Multi-Task Loss Module, Backbone Network f θ The ResNet-12 network can be used. Of course, other networks such as WideResNet, SENet, and HRNet can also be used.

[0063] exist Figure 4 In the illustrated embodiment, the support set may be a first sample data set D novel ,Support set=[S 1 ,…,S k ,…,S N ] represents the first image sample data of the support set with N categories. When the support set is input to the backbone network f of the object classification model θ After that, we can get the backbone network f θ The first depth features [P 1 ,…,P k ,…,P N ], where P k Represents the first deep feature corresponding to category k among N categories. Deep features are also called embedding features.

[0064] Specifically, when each category of the support set includes multiple first image sample data, the first depth feature corresponding to the category can be obtained using the depth features corresponding to the multiple first image sample data. For example, a depth feature can be randomly selected from the depth features corresponding to the multiple first image sample data as the first depth feature, or the average value of the depth features corresponding to the multiple first image sample data can be determined, and the average value can be used as the first depth feature.

[0065] The following describes the process of obtaining an object classification model through model training:

[0066] Figure 5 An embodiment according to the present application is shown. Figure 3 Flowchart of steps before step 330 in the embodiment. Figure 5 As shown, before step 330, the following may be included:

[0067] In step 320, the pre-trained object classification model is trained according to the first sample data set to obtain an object classification model.

[0068] The pre-trained object classification model is a pre-obtained object classification model. As mentioned above, the object classification model is obtained by training based on the first sample data set, and the object classification model can be obtained by further training on the basis of the pre-trained object classification model obtained by pre-training. Since there is no need to train from scratch, the training of the pre-trained object classification model is equivalent to fine-tuning training.

[0069] Figure 6 An embodiment according to the present application is shown. Figure 5 Flow chart of details of step 320 in the embodiment. Figure 6 As shown, the specific steps of obtaining the object classification model by training the pre-trained object classification model are:

[0070] In step 321, the weight matrix of the original local classifier of the pre-trained object classification model is initialized according to the original first depth features corresponding to the first image sample data of each category.

[0071] The local classifier in the BNI of the pre-trained object classification model is the original local classifier and needs further training.

[0072] When the object classification model is not trained, after the first image sample data of each category in the first sample data set is input into the pre-trained object classification model, the backbone network of the pre-trained object classification model extracts the original first depth features of each category.

[0073] Fig.12 FIG. 1 shows a schematic diagram of the principle of an old-class-new-class feature interaction module according to an embodiment of the present application. Fig.12 The upper left figure shows the original first depth feature P corresponding to the first image sample data of each category = [P 1 ,…,P k ,…,P N ] is input into the L2Norm (L2 normalization) layer of BNI to obtain the output of the L2Norm layer, and then the output of the L2Norm layer is used as the weight matrix W = w of the original local classifier 1 ,…,w k ,…,w N], thus initializing the weight matrix of the original local classifier, and the sizes of P and W are Where N is the category, c is the dimension of each category; the original local classifier includes a linear layer (Linear), and the weight matrix W of the original local classifier is the weight of the linear layer. The above method of initializing the weight matrix is ​​the proxy initialization (ProtoInit) provided in the embodiment of this application, please refer to Fig.12 In the lower left figure, compared with random initialization of the weight matrix (Random init), proxy initialization (ProtoInit) can approach the target weight faster.

[0074] In step 322, the original local classifier is trained based on the first sample data set to obtain a local classifier.

[0075] Based on the weight matrix generated by initialization, the original local classifier can be further fine-tuned using the first sample data set. Fig.12 The upper left image, x i s is the first image sample data. After the first image sample data is input into the pre-trained object classification model, the feature extracted by the backbone network of the pre-trained object classification model is f θ (x i s ), f θ (x i s ) is input to the original local classifier to obtain the classification result Y output by the original local classifier L , according to Y L and the category of the first image sample data in the first sample data set, the original local classifier can be fine-tuned and trained, thereby updating the weight matrix of the original local classifier, and finally obtaining the local classifier.

[0076] Figure 7 An embodiment according to the present application is shown. Figure 5 Flowchart of steps before step 320 in the embodiment. Figure 7 As shown, before training the pre-trained object classification model according to the first sample data set to obtain the object classification model, the object classification method may further include the following steps:

[0077] In step 310, the original object classification model is pre-trained according to a second sample data set including a plurality of second sample data to obtain a pre-trained object classification model.

[0078] The second sample data is also image sample data. Generally, the category of the second sample data in the second sample data set is different from the category of the first image sample data in the first sample data set. Similar to the first sample data set, the second sample data set includes second sample data corresponding to multiple categories respectively; for each category, the second sample data set includes multiple second sample data corresponding to the category.

[0079] In one embodiment of the present application, the second sample data set includes a training set and a validation set, and the second sample data in the training set and the validation set belong to different categories.

[0080] Figure 8 An embodiment according to the present application is shown. Figure 7 Flow chart of details of step 310 in the embodiment. Figure 8 As shown, step 310 may specifically include the following steps:

[0081] In step 311, multiple task data set extraction operations are performed on the training set of the second sample data set, so as to randomly extract a first support set and a first query set through each task data set extraction operation.

[0082] The first support set and the first query set both include second sample data corresponding to a predetermined number of categories, and the predetermined number of categories corresponding to the first support set and the first query set are the same.

[0083] A first support set and a first query set can be randomly extracted through each task data set extraction operation.

[0084] Specifically, the second sample dataset can be a public dataset miniImageNet, that is, the model can be pre-trained on the public dataset. If the public dataset miniImageNet is used as the second sample dataset, then the second sample dataset D base The second sample data set D includes second sample data corresponding to 100 categories, and the second sample data corresponding to each category can be the same. base Divide into training set D train and validation set D val , where the training set D train Includes second sample data corresponding to 80 categories for training; validation set D val Including second sample data corresponding to the other 20 categories for verification. Of course, in other embodiments of the present application, other public datasets such as tieredImageNet and ImageNet can also be used as the second sample dataset.

[0085] In step 312, the original object classification model is pre-trained based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model.

[0086] In one embodiment of the present application, the original object classification model is pre-trained based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model, including: pre-training the original object classification model based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a candidate pre-trained object classification model; performing a task data set extraction operation on a verification set of a second sample data set to obtain a randomly extracted second support set and a second query set; performing a performance evaluation on the candidate pre-trained object classification model based on the second support set and the second query set to obtain an evaluation result; and selecting a candidate pre-trained object classification model with the best corresponding evaluation result from each candidate pre-trained object classification model as the pre-trained object classification model.

[0087] Specifically, the specific process of pre-training the original object classification model is as follows: (a) Task dataset extraction operation: train Randomly extract N-way K-shot data as the first support set (SupportSet), which includes N*K second sample data in total, and then extract m samples from each class of the N classes with the same category as the second sample data of the first support set as the first query set (QuerySet), which includes N*m second sample data in total; (b) Repeat the task data set extraction operation in step (a) above t times, and perform a model training iteration after each extraction, then iterate t times in total, at this time, the model parameters will be updated to obtain a candidate pre-trained object classification model; (c) Use the task data set extraction operation in step (a) to extract the model from the validation set D val Randomly extract the second support set and the second query set to evaluate the performance of the candidate pre-trained object classification model; (d) repeat steps (a) to (c) for a total of e times to obtain the performance evaluation results corresponding to the e candidate pre-trained object classification models, and use the candidate pre-trained object classification model with the best performance evaluation result as the pre-trained object classification model M. In this way, when the first sample data set D is used in the subsequent novel When fine-tuning the pre-trained object classification model M, the parameters of the model other than BNI are fixed to obtain the optimal object classification model M'.

[0088] Fig. 9 A flowchart of pre-training the original object classification model based on the first support set and the first query set extracted through multiple task dataset extraction operations according to an embodiment of the present application is shown. Fig. 9As shown, the following steps may be specifically included:

[0089] In step 910, the extracted first support set and the first query set are input into the original object classification model to generate enhanced first depth features corresponding to the second image sample data of each category in the first support set and multiple enhanced second depth features corresponding to the second image sample data in the first query set.

[0090] Please continue to see Figure 4 As shown, the input data passes through the backbone network f θ After processing with TMNAM, the enhanced first depth feature is obtained And the enhanced second depth feature The generation method of the enhanced first depth feature and the enhanced second depth feature will be introduced in the subsequent content.

[0091] In step 920, the enhanced first depth feature and the enhanced second depth feature are input into a metric classifier to obtain a first object classification result of the second image sample data in the first query set.

[0092] Please continue to see Figure 4 As shown, after the enhancement, the first depth feature And the enhanced second depth feature Afterwards, the feature pairs are calculated The cosine similarity between the two is used as the similarity score, and the category with the largest similarity score is used as the classification result Y output by the metric classifier. M .

[0093] In step 930 , a metric classifier loss is determined according to the first object classification result corresponding to the second image sample data in the first query set and the category of the second image sample data.

[0094] According to the classification result Y M and class calculate cross entropy loss as a metric classifier loss L M .

[0095] In step 940, the enhanced second depth features corresponding to the category of the second image sample data among the plurality of enhanced second depth features are input into a global classifier to obtain a second object classification result of the second image sample data.

[0096] Specifically, if the category of the second image sample data is k, then the enhanced second depth feature corresponding to this category is Therefore, Input to the global classifier, after being processed by the linear layer of the global classifier, the output classification result Y is obtained G.

[0097] In step 950 , a global classifier loss is determined according to the second object classification result of the second image sample data and the category of the second image sample data.

[0098] According to the classification result Y G The cross entropy loss is calculated as the global classifier loss L G .

[0099] In step 960, the original object classification model is pre-trained according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model.

[0100] In one embodiment of the present application, the extracted first support set and the first query set are input into the original object classification model, including: rotating the second image sample data in the first query set by multiple predetermined angles to obtain multiple rotated first query sets; inputting the extracted first support set and the rotated first query set into the original object classification model to obtain an angle classification result output by the rotation classifier corresponding to the second image sample data in the rotated first query set;

[0101] The original object classification model is pre-trained according to the metric classifier loss and the global classifier loss to obtain the pre-trained object classification model, including: determining the rotation classifier loss according to the angle classification result corresponding to the second image sample data and the predetermined angle by which the second image sample data is rotated; obtaining the final loss according to the metric classifier loss, the global classifier loss and the rotation classifier loss, and updating the parameters of the original object classification model according to the final loss to obtain the pre-trained object classification model.

[0102] Specifically, the second image sample data in the first query set can be rotated four times by 0°, 90°, 180° and 270° respectively to obtain the four rotated first query sets [x q 0° , x q 90° , x q 180° , x q 270° ], and then the first support set is combined with each rotated first query set to pre-train the original object classification model. The rotation classifier will output the angle classification result Y R , Y R It can be one of 0°, 90°, 180° and 270°. The rotation classifier loss L can be obtained according to the angle classification result and the predetermined angle rotated by the second image sample data. RIn addition, the rotated first query set can also be used to train the tasks corresponding to the global classifier and the metric classifier. It can be seen that each classifier in the global classifier, the metric classifier and the rotated classifier corresponds to a training task.

[0103] exist Figure 4 The multi-task loss module shown calculates the final loss Loss through the following expression:

[0104]

[0105]

[0106] Among them, [w G ,w R ] is the loss weight, [b G ,b R ] is the loss bias, [σ G ,σ R ] is the learnable weight, λ is the weight bias parameter, and λ can be set to λ = 2.0 based on experience.

[0107] As you can see, w G and w R between and b G and b R There is a certain constraint relationship between them, through which the training tasks can be more balanced; the training tasks corresponding to the global classifier and the rotation classifier are auxiliary tasks, and the training tasks corresponding to the metric classifier are small sample learning tasks. G ,w R ] can balance small sample learning tasks and auxiliary tasks, improve the representation ability of deep features, and thus improve the accuracy of small sample classification.

[0108] Please continue to see Figure 3 As shown, in step 340, the image to be classified is input into the object classification model to extract the second depth feature of the image to be classified in the object classification model.

[0109] See also Figure 4 As shown, the image Query to be classified is input into the object classification model. The backbone network f of the object classification model θ The second deep feature Q will be extracted.

[0110] In one embodiment of the present application, extracting a first depth feature corresponding to first image sample data of each category in a first sample data set through an object classification model includes: extracting an original first depth feature corresponding to first image sample data of each category in the first sample data set through a backbone network of the object classification model, and inputting the original first depth feature into a local classifier of the object classification model to obtain a first prediction probability vector output by the local classifier; extracting a first feature vector from a weight matrix of the local classifier according to a position of maximum probability in the first prediction probability vector, and updating the original first depth feature according to the first feature vector to obtain the first depth feature;

[0111] Extracting a second depth feature of an image to be classified in an object classification model includes: extracting an original second depth feature corresponding to the image to be classified through a backbone network of the object classification model, and inputting the original second depth feature into a local classifier of the object classification model to obtain a second prediction probability vector output by the local classifier; extracting a second feature vector from a weight matrix of the local classifier according to a position of maximum probability in the second prediction probability vector, and updating the original second depth feature according to the second feature vector to obtain a second depth feature.

[0112] See also Fig.12 The upper right figure shows the first image sample data x i Input to the backbone network, the backbone network extracts the original second depth feature f θ (x i ), and the original second depth feature f θ (x i ) is input into the local classifier. After being processed by the linear layer of the local classifier, the first predicted probability vector is output. The position k with the maximum probability in the first predicted probability vector corresponds to the category Y output by the local classifier. L , the size is The first predicted probability vector and the weight matrix W′ of the local classifier are 1 ′,…,w k ′,…,w N ′], we can get w k 'The first feature vector, based on the first feature vector, the original first depth feature is updated by performing the following operations on the original first depth feature:

[0113]

[0114] Among them, ‖·‖2 is the L2 norm normalization operation, ← the f on the right θ (x i ) is the original first depth feature, ← f on the left θ (xi ) is the first depth feature.

[0115] Similarly, the original second deep features can also be updated in a similar way. Finally, the old-class-new-class feature interaction module outputs the updated features as embedded features.

[0116] See also Fig.12 As shown in the lower right figure, the above feature update operation is the ClassShift operation, which can make the feature differences between samples of different categories more significant.

[0117] In step 350 , a first feature pair corresponding to each first depth feature is constructed, where the first feature pair includes a first depth feature and a second depth feature.

[0118] The second depth feature Q is respectively combined with each first depth feature [P 1 ,…,P k ,…,P N ] are combined respectively to obtain the first feature pair [P k ,Q](k=1,2,…,N).

[0119] In step 360, for each first feature pair, based on the attention information between the first depth feature and the second depth feature in the first feature pair, the first depth feature and the second depth feature are respectively encoded to obtain a second feature pair including the encoded first depth feature and the encoded second depth feature, and the attention information is used to measure the attention intensity of one depth feature to another depth feature.

[0120] After encoding the first feature pair, the encoded first depth feature is obtained And the second deep feature after encoding

[0121] Fig.10 An embodiment according to the present application is shown. Figure 3 Flow chart of details of step 360 in the embodiment. Fig.10 As shown, step 360 may specifically include the following steps:

[0122] In step 361 , the first depth feature is encoded based on the attention information of the first depth feature to the second depth feature and the attention information of the second depth feature to the first depth feature.

[0123] Specifically, the attention information of the first depth feature to the second depth feature refers to the attention weight of the first depth feature to the second depth feature; the attention information of the second depth feature to the first depth feature refers to the attention weight of the second depth feature to the first depth feature. Encoding the first depth feature is to perform weighted processing on the first depth feature using the attention weight, so as to achieve the purpose of attention enhancement. The encoded first depth feature is the first depth feature after attention enhancement.

[0124] In step 362, the second depth feature is encoded based on the attention information of the second depth feature to the first depth feature and the attention information of the first depth feature to the second depth feature.

[0125] Similar to the way of encoding the first depth feature, encoding the second depth feature is to use the attention information as the attention weight and perform weighted processing on the second depth feature to achieve the purpose of attention enhancement. The encoded second depth feature is the second depth feature after attention enhancement.

[0126] Fig.11 Schematic diagram of the principle of the Transformer cross-correlation attention module according to an embodiment of the present application. Fig.11 As shown in (c), two columns of images are shown on the far left. The two images in the first column represent the input image to be classified and a first image sample data, and the two images in the second column represent the second depth feature corresponding to the image to be classified and the first depth feature corresponding to the first image sample data, that is, the first feature pair [f p ,f q ]. The Transformer mutual-nearest attention module includes two TMNAs (Transformer Mutual-Nearest Attention, Transformer mutual-nearest attention layer). After being processed by the two TMNAs, the second feature pair [f p ′,f q ′].

[0127] See also Fig.11 (a) and Fig.11 As shown in (b), it can be seen that compared with the standard Transformer layer, the Transformer cross-correlation attention layer proposed in the embodiment of the present application adopts multi-head cross-correlation attention. The input of the Transformer cross-correlation attention layer is the first feature pair [f p ,f q] More importantly, the internal structure of the Transformer cross-correlation attention layer is different from that of the standard Transformer layer. Of course, in other embodiments of the present application, the same structure as the standard Transformer layer can also be adopted.

[0128] In the Transformer cross-correlation attention layer, MatMul represents matrix multiplication, ⊙ represents the product at the element level, Row Softmax represents the Softmax operation from the horizontal direction, Column Softmax represents the Softmax operation from the vertical direction, and W Q represents the weight matrix corresponding to the query vector, Q represents the query vector, W K Represents the weight matrix corresponding to the key vector, K represents the key vector, W V Represents the weight matrix corresponding to the value vector, and V represents the value vector (valuevector).

[0129] Therefore, the multi-head cross-correlation attention operation performed in the Transformer cross-correlation attention layer can be expressed as:

[0130] MNA(Q,K,V)=σ(σ(QK T )⊙(σ(KQ T )) T )V,

[0131] Among them, σ is the Softmax operation, ⊙ represents the product at the element level, Q represents the query vector, K represents the key vector, and V represents the value vector.

[0132] In the above expression, σ(QK T ) represents the attention paid by another feature to this feature, σ(KQ T ) represents the attention of this feature to another feature. The attention here is the attention information. If the attention of the two is the same, the corresponding value will be amplified, thereby achieving the effect of enhancing attention.

[0133] In step 370, the distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair is determined, and the object classification result of the image to be classified is determined according to the distance information corresponding to each second feature pair.

[0134] Similar to the classification method in the training phase, after the enhancement, the first deep feature And the enhanced second depth feature Afterwards, the feature pairs are calculated The cosine similarity between the two is used as the similarity score, and the category with the largest similarity score is used as the object classification result Y output by the metric classifier. M Therefore, the distance information here is the cosine distance. Of course, in other embodiments of the present application, the object classification result can also be determined by calculating other distances such as Euclidean distance, Manhattan distance, Minkowski distance, etc.

[0135] In summary, the object classification method provided in the embodiments of the present application can at least achieve the following technical effects:

[0136] 1) Even with very little data (only a few images per class), a high classification accuracy can still be achieved.

[0137] 2) When training the network, add auxiliary tasks and train them together with the small sample learning tasks. The auxiliary tasks can be used to improve the representation ability of deep features, thereby improving the small sample classification accuracy.

[0138] 3) A transformer-based cross-correlation attention module TMNAM is proposed, which allows the deep network to focus more on the target area when extracting features and reduce background interference.

[0139] 4) An old-class-new-class feature interaction module BNI is proposed, which allows the model to learn on a small number of samples and improves the feature discrimination ability.

[0140] The following describes an apparatus embodiment of the present application, which can be used to execute the object classification method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the object classification method of the present application.

[0141] Fig.13 A block diagram of an object classification device according to an embodiment of the present application is shown.

[0142] Reference Fig.13As shown, according to an embodiment of the present application, an object classification device 1300 includes: an extraction unit 1310 , an input unit 1320 , a construction unit 1330 , an encoding unit 1340 and a classification unit 1350 . The extraction unit 1310 is used to extract a first depth feature corresponding to the first image sample data of each category in the first sample data set through an object classification model, wherein the object classification model is obtained by training based on the first sample data set, and the first sample data set includes first image sample data belonging to multiple categories respectively; the input unit 1320 is used to input the image to be classified into the object classification model to extract the second depth feature of the image to be classified in the object classification model; the construction unit 1330 is used to construct a first feature pair corresponding to each first depth feature, and the first feature pair includes the first depth feature and the second depth feature; the encoding unit 1340 is used to encode the first depth feature and the second depth feature respectively for each first feature pair based on the attention information between the first depth feature and the second depth feature in the first feature pair, to obtain a second feature pair including the encoded first depth feature and the encoded second depth feature, and the attention information is used to measure the attention strength of one depth feature to another depth feature; the classification unit 1350 is used to determine the distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair, and determine the object classification result of the image to be classified according to the distance information corresponding to each second feature pair.

[0143] In some embodiments of the present application, based on the aforementioned scheme, the encoding unit 1340 is configured to: encode the first depth feature based on the attention information of the first depth feature to the second depth feature and the attention information of the second depth feature to the first depth feature; encode the second depth feature based on the attention information of the second depth feature to the first depth feature and the attention information of the first depth feature to the second depth feature.

[0144] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1310 is configured to: extract the original first depth features corresponding to the first image sample data of each category in the first sample data set through the backbone network of the object classification model, and input the original first depth features into the local classifier of the object classification model to obtain the first prediction probability vector output by the local classifier; according to the position of the maximum probability in the first prediction probability vector, extract the first feature vector in the weight matrix of the local classifier, and update the original first depth features according to the first feature vector to obtain the first depth features; the input unit 1320 is configured to: extract the original second depth features corresponding to the image to be classified through the backbone network of the object classification model, and input the original second depth features into the local classifier of the object classification model to obtain the second prediction probability vector output by the local classifier; according to the position of the maximum probability in the second prediction probability vector, extract the second feature vector in the weight matrix of the local classifier, and update the original second depth features according to the second feature vector to obtain the second depth features.

[0145] In some embodiments of the present application, based on the aforementioned scheme, before extracting the first depth features corresponding to the first image sample data of each category in the first sample data set through the object classification model, the extraction unit 1310 is also used to: train the pre-trained object classification model according to the first sample data set to obtain the object classification model.

[0146] In some embodiments of the present application, based on the aforementioned scheme, the extraction unit 1310 is configured to: initialize the weight matrix of the original local classifier of the pre-trained object classification model according to the original first depth features corresponding to the first image sample data of each category; and train the original local classifier based on the first sample data set to obtain a local classifier.

[0147] In some embodiments of the present application, based on the aforementioned scheme, before training the pre-trained object classification model according to the first sample data set to obtain the object classification model, the extraction unit 1310 is also used to: pre-train the original object classification model according to the second sample data set including multiple second sample data to obtain the pre-trained object classification model.

[0148] In some embodiments of the present application, based on the aforementioned scheme, the second sample data set includes a training set and a validation set, and the second sample data in the training set and the validation set belong to different categories. The extraction unit 1310 is configured to: perform multiple task data set extraction operations on the training set of the second sample data set to randomly extract a first support set and a first query set through each task data set extraction operation, wherein the first support set and the first query set both include second sample data corresponding to a predetermined number of categories, respectively, and the predetermined number of categories corresponding to the first support set and the first query set are the same; pre-train the original object classification model based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model.

[0149] In some embodiments of the present application, based on the aforementioned scheme, the original object classification model also includes a global classifier and a metric classifier, and the extraction unit 1310 is configured to: input the extracted first support set and the first query set into the original object classification model to generate enhanced first depth features corresponding to the second image sample data of each category in the first support set and multiple enhanced second depth features corresponding to the second image sample data in the first query set; input the enhanced first depth features and the enhanced second depth features into the metric classifier to obtain the first object classification result of the second image sample data in the first query set; determine the metric classifier loss according to the first object classification result corresponding to the second image sample data in the first query set and the category of the second image sample data; input the enhanced second depth features corresponding to the category of the second image sample data among the multiple enhanced second depth features into the global classifier to obtain the second object classification result of the second image sample data; determine the global classifier loss according to the second object classification result of the second image sample data and the category of the second image sample data; pre-train the original object classification model according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model.

[0150] In some embodiments of the present application, based on the aforementioned scheme, the original object classification model also includes a rotation classifier. When executing the step of inputting the extracted first support set and the first query set into the original object classification model, the extraction unit 1310 is further configured to: rotate the second image sample data in the first query set by multiple predetermined angles respectively to obtain multiple rotated first query sets; input the extracted first support set and the rotated first query set into the original object classification model to obtain the angle classification result corresponding to the second image sample data in the rotated first query set output by the rotation classifier; when executing the step of pre-training the original object classification model according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model, the extraction unit 1310 is further configured to: determine the rotation classifier loss according to the angle classification result corresponding to the second image sample data and the predetermined angle by which the second image sample data is rotated; obtain the final loss according to the metric classifier loss, the global classifier loss and the rotation classifier loss, and update the parameters of the original object classification model according to the final loss to obtain the pre-trained object classification model.

[0151] Fig.14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0152] It should be noted that Fig.14 The computer system 1400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Fig.14 As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1402 or the program loaded from the storage part 1408 to the random access memory (RAM) 1403, such as executing the method described in the above embodiment. In the RAM 1403, various programs and data required for system operation are also stored. The CPU 1401, the ROM 1402 and the RAM 1403 are connected to each other through the bus 1404. The input / output (I / O) interface 1405 is also connected to the bus 1404.

[0154] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0155] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1409, and / or installed from a removable medium 1411. When the computer program is executed by a central processing unit (CPU) 1401, various functions defined in the system of the present application are executed.

[0156] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0157] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0159] As one aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0160] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0161] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0162] It is understandable that in the specific implementation of this application, data related to entities is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0163] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0164] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for object classification, characterized in that: The method comprises: Extracting the first depth feature corresponding to the first image sample data of each category in the first sample data set in the following manner: extracting the original first depth feature corresponding to the first image sample data of each category in the first sample data set through the backbone network of the object classification model, and inputting the original first depth feature into the local classifier of the object classification model to obtain a first prediction probability vector output by the local classifier, wherein the object classification model is trained based on the first sample data set, and the first sample data set includes first image sample data belonging to a plurality of categories respectively; extracting a first feature vector in the weight matrix of the local classifier according to the position of the maximum probability in the first prediction probability vector, and updating the original first depth feature according to the first feature vector to obtain the first depth feature; Inputting an image to be classified into the object classification model, extracting original second depth features corresponding to the image to be classified through a backbone network of the object classification model, and inputting the original second depth features into a local classifier of the object classification model to obtain a second prediction probability vector output by the local classifier; Extracting a second feature vector from the weight matrix of the local classifier according to the position of the maximum probability in the second predicted probability vector, and updating the original second depth feature according to the second feature vector to obtain a second depth feature; Constructing a first feature pair corresponding to each first depth feature, wherein the first feature pair includes the first depth feature and the second depth feature; For each first feature pair, based on the attention weight of the first depth feature to the second depth feature and the attention weight of the second depth feature to the first depth feature, the first depth feature is weighted, and based on the attention weight of the second depth feature to the first depth feature and the attention weight of the first depth feature to the second depth feature, the second depth feature is weighted, to obtain a second feature pair including the encoded first depth feature and the encoded second depth feature; Determine the distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair, and determine the object classification result of the image to be classified according to the distance information corresponding to each second feature pair.

2. The object classification method according to claim 1, characterized in that: Before extracting the first depth feature corresponding to the first image sample data of each category in the first sample data set through the object classification model, the method further includes: The pre-trained object classification model is trained according to the first sample data set to obtain an object classification model.

3. The object classification method according to claim 2, characterized in that: The step of training the pre-trained object classification model according to the first sample data set to obtain the object classification model includes: Initializing a weight matrix of an original local classifier of the pre-trained object classification model according to the original first depth features corresponding to the first image sample data of each category; The original local classifier is trained based on the first sample data set to obtain a local classifier.

4. The object classification method according to claim 3, characterized in that: Before training the pre-trained object classification model according to the first sample data set to obtain the object classification model, the method further includes: The original object classification model is pre-trained according to a second sample data set including a plurality of second sample data to obtain a pre-trained object classification model.

5. The object classification method according to claim 4, characterized in that: The second sample data set includes a training set and a validation set, the second sample data in the training set and the second sample data in the validation set belong to different categories, and the pre-training of the original object classification model according to the second sample data set including the plurality of second sample data to obtain the pre-trained object classification model includes: Performing multiple task data set extraction operations on the training set of the second sample data set to randomly extract a first support set and a first query set through each task data set extraction operation, wherein the first support set and the first query set both include second sample data corresponding to a predetermined number of categories, and the predetermined number of categories corresponding to the first support set and the first query set are the same; The original object classification model is pre-trained based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model.

6. The object classification method according to claim 5, characterized in that: The original object classification model further includes a global classifier and a metric classifier. The original object classification model is pre-trained based on the first support set and the first query set extracted through multiple task data set extraction operations to obtain a pre-trained object classification model, including: Inputting the extracted first support set and the first query set into the original object classification model to generate enhanced first depth features corresponding to the second image sample data of each category in the first support set and multiple enhanced second depth features corresponding to the second image sample data in the first query set; Inputting the enhanced first depth feature and the enhanced second depth feature into the metric classifier to obtain a first object classification result of the second image sample data in the first query set; Determine a metric classifier loss based on the first object classification result corresponding to the second image sample data in the first query set and the category of the second image sample data; Inputting the enhanced second depth features corresponding to the category of the second image sample data among the plurality of enhanced second depth features into the global classifier to obtain a second object classification result of the second image sample data; determining a global classifier loss according to a second object classification result of the second image sample data and a category of the second image sample data; The original object classification model is pre-trained according to the metric classifier loss and the global classifier loss to obtain a pre-trained object classification model.

7. The object classification method according to claim 6, characterized in that: The original object classification model further includes a rotation classifier, and the step of inputting the extracted first support set and the first query set into the original object classification model includes: Rotating the second image sample data in the first query set by a plurality of predetermined angles respectively to obtain a plurality of rotated first query sets; Inputting the extracted first support set and the rotated first query set into the original object classification model to obtain an angle classification result output by the rotation classifier corresponding to the second image sample data in the rotated first query set; The pre-training of the original object classification model according to the metric classifier loss and the global classifier loss to obtain the pre-trained object classification model includes: determining a rotation classifier loss according to an angle classification result corresponding to the second image sample data and a predetermined angle by which the second image sample data is rotated; A final loss is obtained according to the metric classifier loss, the global classifier loss and the rotation classifier loss, and the parameters of the original object classification model are updated according to the final loss to obtain a pre-trained object classification model.

8. An object classification device, characterized in that: The device comprises: an extraction unit, configured to extract first depth features corresponding to first image sample data of each category in a first sample data set by using an object classification model, wherein the object classification model is trained based on the first sample data set, and the first sample data set includes first image sample data belonging to a plurality of categories respectively; An input unit, used for inputting the image to be classified into the object classification model, so as to extract the second depth feature of the image to be classified in the object classification model; A construction unit, configured to construct a first feature pair corresponding to each first depth feature, wherein the first feature pair includes the first depth feature and the second depth feature; an encoding unit, configured to, for each first feature pair, perform weighted processing on the first depth feature based on an attention weight of the first depth feature on the second depth feature and an attention weight of the second depth feature on the first depth feature, and perform weighted processing on the second depth feature based on an attention weight of the second depth feature on the first depth feature and an attention weight of the first depth feature on the second depth feature, to obtain a second feature pair including an encoded first depth feature and an encoded second depth feature; a classification unit, configured to determine distance information between the encoded first depth feature and the encoded second depth feature in each second feature pair, and determine an object classification result of the image to be classified according to the distance information corresponding to each second feature pair; The extraction unit is configured to: extract the original first depth feature corresponding to the first image sample data of each category in the first sample data set through the backbone network of the object classification model, and input the original first depth feature into the local classifier of the object classification model to obtain the first prediction probability vector output by the local classifier; extract the first feature vector in the weight matrix of the local classifier according to the position of the maximum probability in the first prediction probability vector, and update the original first depth feature according to the first feature vector to obtain the first depth feature; The input unit is configured as follows: extracting the original second depth feature corresponding to the image to be classified through the backbone network of the object classification model, and inputting the original second depth feature into the local classifier of the object classification model to obtain a second prediction probability vector output by the local classifier; extracting a second feature vector from the weight matrix of the local classifier according to the position of the maximum probability in the second prediction probability vector, and updating the original second depth feature according to the second feature vector to obtain a second depth feature.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object classification method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the object classification method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the object classification method according to any one of claims 1 to 7.

Images