Image classification method, training method of image classification model, and related device

Abstract

The embodiment of the application provides a kind of image classification method, the training method of image classification model and related equipment, belong to artificial intelligence technical field.Image classification method includes: obtaining initial sample, initial sample includes initial new class sample and multiple initial base class samples;Initial sample is input into initial image classification model, and the similarity matrix determined by the preset composite measure function is selected from multiple initial base class samples Target base class sample;According to target base class sample and the similarity matrix corresponding to target base class sample, initial new class sample is dynamically corrected, and target new class sample is obtained;Target new class sample is used to train initial image classification model until reaching training stop condition, and the trained image classification model is obtained;Obtain the target image to be classified, input the target image into the trained image classification model, and obtain the corresponding target classification result.The application can improve the accuracy of image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an image classification method, a training method of an image classification model, and related equipment. BACKGROUND

[0002] Image classification refers to assigning an input image to one of the predefined categories. Among them, few-shot learning image classification is a common image classification task, which aims to train a model to achieve efficient task execution under the condition of only a few labeled samples.

[0003] The related technology focuses on the generation of samples, and helps the model to contact more sample data in the training process by generating more samples. However, such an image classification method ignores the real feature distribution of the target category group represented by the few samples, resulting in deviation between the generated samples and the few samples, and thus causing poor image classification accuracy of the trained model. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose an image classification method, a training method of an image classification model, and related equipment, which aims to improve the accuracy of image classification.

[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes an image classification method, comprising:

[0006] Obtaining initial samples, the initial samples including initial new class samples and a plurality of initial base class samples;

[0007] Inputting the initial samples into an initial image classification model, and determining a similarity matrix corresponding to the initial samples based on a preset composite measure function, the similarity matrix being used to represent the feature matching degree of the initial new class samples and each initial base class sample in multiple dimensions;

[0008] Selecting a target base class sample from the plurality of initial base class samples based on the similarity matrix;

[0009] According to the target base class sample and the similarity matrix corresponding to the target base class sample, dynamically correcting the initial new class sample to obtain a target new class sample;

[0010] Training the initial image classification model using the target new class sample until a training stop condition is reached, to obtain a trained image classification model;

[0011] Obtaining a target image to be classified, inputting the target image into the trained image classification model, and obtaining a corresponding target classification result.

[0012] In some embodiments, the initial new-class sample includes a plurality of initial new-class features, and each initial base-class sample includes a plurality of initial base-class features.

[0013] The similarity matrix is determined based on the composite metric function, including:

[0014] The new-class feature matrix of the initial new-class sample and the base-class feature matrix of the initial base-class sample are determined respectively;

[0015] Based on the plurality of initial new-class features and the plurality of initial base-class features, a first similarity value of the initial new-class sample and each initial base-class sample in a first dimension is determined, and a second similarity value of the initial new-class sample and each initial base-class sample in a second dimension is determined;

[0016] Based on the new-class feature matrix and the base-class feature matrix, a third similarity value of the initial new-class sample and each initial base-class sample in a third dimension is determined;

[0017] According to the composite metric function provided with a preset hyperparameter, the first similarity value, the second similarity value and the third similarity value are fused to obtain the similarity matrix.

[0018] In some embodiments, the new-class feature matrix of the initial new-class sample and the base-class feature matrix of the initial base-class sample are determined respectively, including:

[0019] Each initial new-class feature is transposed to obtain a corresponding new-class transposed feature;

[0020] Based on the cross product of each initial new-class feature and the corresponding new-class transposed feature, the new-class feature matrix is obtained;

[0021] Based on the plurality of initial base-class features, the initial base-class mean of each initial base-class sample is obtained;

[0022] Each initial base-class mean is transposed to obtain a corresponding base-class transposed mean;

[0023] Based on the cross product of each initial base-class mean and the corresponding base-class transposed mean, the base-class feature matrix is obtained.

[0024] In some embodiments, based on the plurality of initial new-class features and the plurality of initial base-class features, the first similarity value of the initial new-class sample and each initial base-class sample in the first dimension is determined, and the second similarity value of the initial new-class sample and each initial base-class sample in the second dimension is determined, including:

[0025] A first probability distribution set of the initial new-class sample is determined, and a second probability distribution set of the initial base-class sample is determined;

[0026] According to the first probability distribution set and the second probability distribution set, a joint distribution set is composed;

[0027] determine a first similarity value based on a distance expectation value of any initial new class feature and any initial base class feature under a joint distribution set;

[0028] determine a second similarity value based on a divergence value between the first probability distribution set and the second probability distribution set.

[0029] In some embodiments, the new class feature matrix comprises a plurality of first matrix elements, and the base class feature matrix comprises a plurality of second matrix elements;

[0030] determine a third similarity value of the initial new class sample and each initial base class sample under a third dimension based on the new class feature matrix and the base class feature matrix, comprising:

[0031] calculate a square sum between any first matrix element and any second matrix element to obtain an element square value;

[0032] perform square root processing on the element square value to obtain an element square root value;

[0033] determine the third similarity value based on the element square root value.

[0034] In some embodiments, based on the similarity matrix, a target base class sample is selected from the plurality of initial base class samples, comprising:

[0035] based on a preset normalization operation operator, normalize the similarity matrix of all initial base class samples to obtain an adjusted similarity matrix, wherein the adjusted similarity matrix comprises a plurality of third matrix elements;

[0036] for any adjusted similarity matrix, if all third matrix elements are greater than a preset similarity threshold, determine that the corresponding initial base class sample is the target base class sample.

[0037] In some embodiments, based on the target base class sample and the similarity matrix corresponding to the target base class sample, the initial new class sample is dynamically corrected to obtain a target new class sample, comprising:

[0038] based on the plurality of initial base class features, obtain an initial base class covariance value corresponding to each initial base class sample;

[0039] based on the similarity matrix, update the initial base class mean and the initial base class covariance value respectively to obtain a target base class mean and a target base class covariance value;

[0040] based on the target base class mean and the target base class covariance value, migrate the target base class sample to the initial new class sample to obtain the dynamically corrected target new class sample.

[0041] In some embodiments, before determining the similarity matrix based on the composite metric function, further comprising:

[0042] If the preset distribution hyperparameter value of the initial image classification model is not equal to the preset parameter threshold, performing logarithmic processing on all initial new class samples to obtain updated initial new class samples;

[0043] If the preset distribution hyperparameter value of the initial image classification model is equal to the preset parameter threshold, performing exponential processing on all initial new class samples to obtain updated initial new class samples.

[0044] To achieve the above object, a second aspect of the embodiment of the present application proposes a training method of an image classification model, comprising:

[0045] obtaining initial samples, the initial samples comprising initial new class samples and a plurality of initial base class samples;

[0046] inputting the initial samples into an initial image classification model, and determining a similarity matrix corresponding to the initial samples based on a preset composite metric function, the similarity matrix being used to represent a feature matching degree of the initial new class samples and each of the initial base class samples in multiple dimensions;

[0047] selecting a target base class sample from the plurality of initial base class samples based on the similarity matrix;

[0048] performing dynamic correction on the initial new class samples according to the target base class sample and the similarity matrix corresponding to the target base class sample to obtain a target new class sample;

[0049] training the initial image classification model using the target new class sample until a training stop condition is reached to obtain a trained image classification model.

[0050] To achieve the above object, a third aspect of the embodiment of the present application proposes an image classification device, comprising:

[0051] an obtaining module configured to obtain initial samples, the initial samples comprising initial new class samples and a plurality of initial base class samples;

[0052] a similarity matrix determining module configured to input the initial samples into an initial image classification model, and determine a similarity matrix corresponding to the initial samples based on a preset composite metric function, the similarity matrix being used to represent a feature matching degree of the initial new class samples and each of the initial base class samples in multiple dimensions;

[0053] a target base class sample selecting module configured to select a target base class sample from the plurality of initial base class samples based on the similarity matrix;

[0054] a correction module, configured to perform dynamic correction on the initial new-class sample according to the target base-class sample and the similarity matrix corresponding to the target base-class sample, to obtain a target new-class sample;

[0055] a training module, configured to train the initial image classification model by using the target new-class sample until a training stop condition is reached, to obtain a trained image classification model;

[0056] an application module, configured to obtain a target image to be classified, and input the target image into the trained image classification model to obtain a corresponding target classification result.

[0057] To achieve the above object, a fourth aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the image classification method of the first aspect or the training method of the image classification model of the second aspect when executing the computer program.

[0058] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image classification method of the first aspect or the training method of the image classification model of the second aspect.

[0059] The image classification method, the training method of the image classification model and the related device provided by the present application, the image classification method comprises obtaining initial samples, the initial samples comprise an initial new-class sample and a plurality of initial base-class samples; inputting the initial samples into an initial image classification model, and determining a similarity matrix corresponding to the initial samples based on a preset composite metric function, the similarity matrix is used to represent the feature matching degree of the initial new-class sample and each initial base-class sample in multiple dimensions; the similarity matrix determined based on the composite metric function provides more position and distribution information, and improves the accuracy of the feature distribution description between the two types of samples; then, based on the similarity matrix, a target base-class sample is selected from the plurality of initial base-class samples; the initial new-class sample is dynamically corrected according to the target base-class sample and the similarity matrix corresponding to the target base-class sample, so as to compensate for the problem of scarcity of effective training features in the initial new-class sample by accurately capturing a more similar target base-class sample, to obtain a target new-class sample; then, the target new-class sample is used to train the initial image classification model until a training stop condition is reached, to obtain a trained image classification model; then, a target image to be classified is obtained, and the target image is input into the trained image classification model to obtain a corresponding target classification result. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is an application scenario of the image classification device provided by the embodiment of the present application;

[0061] Figure 2 is an optional flowchart of the image classification method provided by the embodiment of the present application;

[0062] Figure 3 is an optional sample processing schematic diagram of the image classification method provided by the embodiment of the present application;

[0063] Figure 4 is an optional correction schematic diagram of the image classification method provided by the embodiment of the present application;

[0064] Figure 5 is an optional flowchart of the training method of the image classification model provided by the embodiment of the present application;

[0065] Figure 6 is an optional flowchart of the image classification device provided by the embodiment of the present application;

[0066] Figure 7 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0070] First, the terms involved in the present application are analyzed:

[0071] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0072] Among them, machine learning is a subfield of artificial intelligence, machine learning aims to make relevant models automatically learn to improve their performance through data without explicit programming. In other words, the goal of machine learning is to let the computer "learn" the law from the data, and use these laws for intelligent prediction or decision-making.

[0073] Further, few-shot learning (small sample learning): is a task in the field of machine learning, aiming to solve classification or regression problems when the training data is very limited. Unlike traditional supervised learning methods, few-shot learning assumes that the model only encounters a small number of samples (usually 1 to 5) for each class during the training phase, and needs to infer and classify new classes during the test phase.

[0074] Image classification refers to assigning an input image to one of the predefined categories. Among them, few-shot learning image classification is a common image classification task, which aims to train the model to achieve efficient task execution under the condition of only a few labeled samples.

[0075] The related technology focuses on the generation of samples, and helps the model to contact more sample data during the training process by generating more samples. However, such image classification method ignores the true feature distribution of the target class group represented by few samples, resulting in deviation of the generated samples from few samples, and further causing poor classification accuracy of the model image obtained by training.

[0076] Based on this, the embodiment of the present application provides an image classification method, a training method of an image classification model and related equipment, aiming to improve the accuracy of image classification.

[0077] Exemplarily, as Figure 1 shown, Figure 1is a schematic diagram of an application scenario of the image classification apparatus provided in the embodiments of the present application. In an optional application scenario, a client 11 is in communication connection with a server 12, the image classification apparatus provided in the embodiments of the present application is deployed in the server 12, and an image classification model is arranged in the image classification apparatus. The image classification model is obtained by training based on new class samples in advance. The new class samples refer to samples that have never been seen before by the image classification model. Only a small number of samples with annotations that can be used for training are included in the new class samples. A user inputs a target image to be classified through the client 11. The trained image classification model performs image recognition on the target image, and determines which of the preset categories the target image belongs to based on the recognized image features, thereby completing the image classification task. Since the image classification model provided in the embodiments of the present application is corrected based on the new class sample training method provided in the embodiments of the present application, the feature distribution accuracy of the target category group indicated by the new class sample in the feature space is improved. Therefore, the image classification model arranged in the server 12 and obtained based on the corrected new class sample can perform image classification processing on the input target image, and obtain a corresponding high-accuracy target classification result.

[0078] It should be noted that, in the embodiments of the present application, when it is necessary to obtain user's basic information or information related to user's identity and user characteristics, the user's permission or consent will be obtained first, and the collection, use and processing of the data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained first, and after obtaining the separate permission or separate consent of the user, the necessary data for enabling the embodiments of the present application to operate normally will be obtained. For example, when obtaining initial samples, the embodiments of the present application will first obtain the authorization or consent of the relevant personnel, otherwise the initial samples cannot be used in the embodiments of the present application. In addition, other related data obtained by the image classification apparatus of the present application are authorized data, which will not be described here.

[0079] In the embodiments of the present application, the image classification apparatus will be described from the dimension. The image classification apparatus can be integrated in a computer device, such as a server. As shown in Figure 2 Figure 2 is an optional flowchart of the image classification method provided in the embodiments of the present application, Figure 2 The method in the embodiments of the present application can include but is not limited to the following steps 101 to 106. When the image classification apparatus executes the image classification method, the specific process is as follows. It should be first noted that the order of the steps 101 to 106 in the embodiments of the present application is not limited, and the order of the steps can be adjusted or some steps can be reduced or added according to actual needs. Figure 2

[0080] ​​Step 101: Obtain initial samples, which include initial new class samples and multiple initial base class samples.

[0081] Step 101 will be described in detail below.

[0082] The initial samples refer to the dataset used to train or evaluate the machine learning model. Typically, the initial samples include a large number of initial base class samples with known classification labels under different base class categories, and a small number of initial new class samples with known classification labels under different new class categories, representing initial image classification models that have never seen or are unfamiliar with. Usually, the number of base class categories is much greater than the number of new class categories, with each base class category including multiple initial base class samples and each new class category including one or two initial new class samples. For ease of description, this application's embodiments illustrate an example where each initial base class sample belongs to a different base class category, and each initial new class sample belongs to a different new class category.

[0083] Furthermore, in the image classification task of this application embodiment, both the initial base class sample and the initial new class sample are image samples. The granularity of the classification label of the initial sample can be set according to actual conditions, and this application embodiment does not impose any limitations on this. For example, one initial base class sample a may have the species classification label "cat," and another initial base class sample b may have the human body part classification label "human eye."

[0084] The initial image classification model is already capable of performing good image classification on the initial base class samples. However, without training using the image classification method proposed in this application, the accuracy of image classification results for the initial new class samples is poor. This application uses the initial image classification model as a starting point and performs further optimization training to improve the ability of the trained image classification model to recognize the initial new class samples.

[0085] like Figure 3 As shown, Figure 3 This is an optional sample processing diagram of the image classification method provided in this application embodiment. The samples for training the initial image classification model include initial base class samples and initial new class samples. The initial new class samples include support class samples and query class samples. The support class samples include one sample from each of the five new class categories. The support class samples are used to train the initial image classification model. The target category object represented by the query class sample is one of the five new class category samples. The query class samples are used to test and verify the image classification model obtained after training.

[0086] It should be noted that the initial base class sample is usually obtained from an open source database, and the initial new class sample can be obtained from an open source database or obtained in real time. The initial sample can be obtained in various ways, and the embodiments of the present application are only used for illustration and do not limit them.

[0087] In step 102, the initial sample is input into the initial image classification model, and the similarity matrix corresponding to the initial sample is determined based on the preset composite metric function. The similarity matrix is used to represent the feature matching degree of the initial new class sample and each initial base class sample in multiple dimensions.

[0088] The step 102 is described in detail below.

[0089] The initial image classification model can be a wide residual network, a convolutional neural network model, a recurrent neural network model, a convolutional recurrent neural network model, etc. The specific implementation can be adjusted according to the actual situation, and the embodiments of the present application do not limit it. For example Figure 3 As shown in the figure, the wide residual network (WideResNet) is a deep convolutional neural network improved on the basis of the residual network. Its main feature is to use a wide convolutional layer to increase the number of channels of the network, thereby improving the expression ability and performance of the network.

[0090] The composite metric function is a function for comprehensively evaluating the similarity between the initial base class sample and the initial new class sample (two types of samples) from multiple dimensions. The matrix elements in the similarity matrix generated based on the result of the composite metric function are used to represent the similarity scores between different feature points of the two types of samples. Compared with the feature distribution description obtained by the traditional method in a single dimension, the similarity matrix determined according to the composite metric function provides more position and distribution information, thereby improving the accuracy of the feature distribution description between the two types of samples.

[0091] Further, as shown in the figure Figure 3 After the initial sample is input into the initial image classification model, the initial image classification model performs feature extraction processing on the initial base class sample and the support class sample respectively to obtain a plurality of initial base class features and a plurality of initial new class samples. The feature extraction processing includes data enhancement processing, convolution processing, feature extraction processing, activation, etc. The specific steps of the feature extraction processing can be adaptively adjusted according to the actual situation, and the embodiments of the present application do not limit it.

[0092] In some embodiments, before determining the similarity matrix based on the composite metric function, the following steps are further included:

[0093] (102.a.1) If the preset distribution hyperparameter value of the initial image classification model is not equal to the preset parameter threshold, performing logarithmic processing on all initial new class samples to obtain updated initial new class samples.

[0094] (102.a.2) If the preset distribution hyperparameter value of the initial image classification model is equal to the preset parameter threshold, performing exponential processing on all initial new class samples to obtain updated initial new class samples.

[0095] The steps (102.a.1) to (102.a.2) are described in detail as follows.

[0096] In some embodiments, since the Gaussian distribution can simplify the calculation amount of the initial image classification model, and the feature distribution of the initial new class sample does not necessarily obey the Gaussian distribution, the present embodiment will first perform Tukey's Ladder of Power Transformation on the support class sample S and the query class sample Q as shown in the following formula <1>:

[0097]

[0098] Wherein, a is a preset distribution hyperparameter value, which is used to adjust the distribution deviation; in the present embodiment, 0 is the preset parameter threshold, of course, the preset parameter threshold can be adaptively adjusted according to the actual situation; x i represents the ith feature from the support class sample or the query class sample.

[0099] Wherein, logx i represents that logarithmic processing is performed on all initial new class samples. represents that logarithmic processing is performed on all initial new class samples.

[0100] Further, after the Tukey transformation, the support class sample or the query class sample is updated to and Wherein, y i is the classification label of the support class sample or the query class sample, N is the number of the support class sample or the query class sample, K is the feature dimension of the support class sample, and q is the feature dimension of the query class sample.

[0101] In some embodiments, the similarity matrix is determined based on the composite measurement function, including the following steps:

[0102] (102.b.1) respectively determining the new class feature matrix of the initial new class sample, and the base class feature matrix of the initial base class sample.

[0103] The step (102.b.1) is described in detail as follows.

[0104] Both the new class feature matrix and the base class feature matrix are matrix structures. The new class feature matrix is ​​used to characterize the position and feature attributes of multiple initial new class features in the feature space, and the base class feature matrix is ​​used to characterize the position and feature attributes of multiple initial base class features in the feature space.

[0105] Furthermore, both the new class feature matrix and the base class feature matrix are products of the complex relationships between sample features captured under high-dimensional feature space conditions. The unified feature representation facilitates the subsequent comparison of the similarity between the initial base class samples and the initial new class samples in the same feature space.

[0106] Next, as Figure 3 As shown, after completing the Tuche transform processing of the initial new class sample, it is necessary to determine the feature matrices of the initial new class sample and the initial base class sample respectively, so that the similarity can be compared through the feature matrices of the two classes of samples, and then the target base class sample for correcting the initial new class sample can be selected from the database containing multiple initial base class samples.

[0107] In some embodiments, determining the new class feature matrix of the initial new class sample and the base class feature matrix of the initial base class sample includes the following steps:

[0108] (A.1) Transpose each initial new class feature to obtain the corresponding new class transposed feature.

[0109] (A.2) Based on the cross product of each initial new class feature and the corresponding new class transpose feature, the new class feature matrix is ​​obtained.

[0110] (A.3) Based on multiple initial base class features, the mean value of each initial base class sample is obtained.

[0111] (A.4) Transpose the mean of each initial base class to obtain the corresponding transposed mean of the base class.

[0112] (A.5) Based on the cross product of the mean of each initial base class and the mean of the corresponding base class transpose, the base class feature matrix is ​​obtained.

[0113] Steps (A.1) to (A.5) are described in detail below.

[0114] In some embodiments, by the following formula <2> Determine the feature matrix M of the new class i :

[0115]

[0116] Among them, v i Represents the i-th initial new class feature; T represents the transpose process; for v iTransposition processing is performed to obtain v i The corresponding new class transposed feature · is the cross product (also known as the vector product or outer product).

[0117] Further, compared with the traditional method of describing the initial new class feature from a single dimension direction, the new class feature matrix of the embodiment of the application includes the feature description distribution of the initial new class feature from the original dimension direction, and also includes the feature description distribution after transposition in other dimension directions, so that more comprehensive and rich feature distribution information of the initial new class feature in multiple dimensions can be obtained.

[0118] Further, since the number of initial base class samples is much larger than that of initial new class samples, if the corresponding base class feature matrix is calculated according to formula <2> for each initial base class sample, the initial image classification model will consume a large amount of computing resources due to high complexity calculation. Based on this, the embodiment of the application first obtains the initial base class mean μ j :

[0119]

[0120] wherein n j represents the initial base class feature number of the jth initial base class sample, x i represents the ith initial base class feature in the jth initial base class sample; and ∑ represents summation processing.

[0121] In some other embodiments, if multiple initial base class samples are included in a base class category, n j represents the initial base class sample number of the jth base class category, x i represents the ith initial base class sample in the jth base class category.

[0122] Then, the base class feature matrix M′ j :

[0123]

[0124] wherein μ j represents the initial base class mean corresponding to the jth initial base class sample; T represents transposition processing; and μ j is transposed to obtain μ j The corresponding base class transposed mean · is the cross product.

[0125] In this way, the calculation complexity of the base class feature matrix is reduced from O(n 2) to O(1). When the initial image classification model processes the initial base class samples, especially a large number of initial base class samples, the sample processing speed can be improved, and the waste of computing resources can be reduced.

[0126] Next, steps (102.b.2) to (102.b.4) are described in detail as follows.

[0127] (102.b.2) Based on the plurality of initial new class features and the plurality of initial base class features, a first similarity value of the initial new class sample and each initial base class sample in the first dimension is determined, and a second similarity value of the initial new class sample and each initial base class sample in the second dimension is determined.

[0128] (102.b.3) Based on the new class feature matrix and the base class feature matrix, a third similarity value of the initial new class sample and each initial base class sample in the third dimension is determined.

[0129] (102.b.4) According to a composite metric function provided with preset hyperparameters, the first similarity value, the second similarity value, and the third similarity value are fused to obtain a similarity matrix.

[0130] In some embodiments, the similarity matrix D is determined by the following formula <5> of the composite metric function provided with preset hyperparameters. cm :

[0131] D cm = β · W + ò · F + η · JS <5>

[0132] Wherein, β, ò and η are preset hyperparameters, and their specific values can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this. W, and JS are similarity values of the initial new class sample and each initial base class sample calculated based on the wasserstein distance, the frobenius norm, and the jensen-shannon divergence method.

[0133] It can be understood that the traditional method has limitations in relying on the Euclidean distance in a single dimension as a similarity measurement standard. Especially in a high-dimensional feature space, such a measurement method ignores the shape, variance, and other high-order statistical properties of data distribution, and it is difficult to fully reflect the actual similarity between the initial base class sample and the initial new class sample. Compared with the traditional method, the embodiments of the present application integrate different measurement standards to more comprehensively capture the similarity between the two types of samples.

[0134] It should be noted that the composite metric function for determining the similarity matrix in the embodiments of the present application can also be constructed based on metric methods such as chi-square divergence, Hellinger distance, Bhattacharyya distance, etc., and the specific selection can be made according to actual conditions, and the embodiments of the present application do not make any limitation in this regard.

[0135] In some embodiments, based on the plurality of initial new class features and the plurality of initial base class features, the first similarity value of the initial new class sample and each initial base class sample in the first dimension is determined, and the second similarity value of the initial new class sample and each initial base class sample in the second dimension is determined, including the following steps:

[0136] (B.1) determining a first probability distribution set of the initial new class sample, and determining a second probability distribution set of the initial base class sample.

[0137] (B.2) according to the first probability distribution set and the second probability distribution set, a joint distribution set is composed.

[0138] (B.3) based on the distance expectation value of any initial new class feature and any initial base class feature under the joint distribution set, the first similarity value is determined.

[0139] (B.4) based on the divergence value between the first probability distribution set and the second probability distribution set, the second similarity value is determined.

[0140] The steps (B.1) to (B.4) are described in detail as follows.

[0141] Wherein, the first similarity value W between the initial base class sample and the initial new class sample is measured by the following formula <6>:

[0142]

[0143] Wherein, ∏(p1, p2) is a joint distribution set composed of the first probability distribution set P1 and the second probability distribution set P2; for each possible joint distribution γ, a sample pair (x, y) ~ γ can be obtained, that is, a sample pair composed of any initial new class feature and any initial base class feature; then, the distance ||x-y|| between these sample pairs is evaluated, and the distance expectation value of the sample pair under the joint distribution γ is calculated inf represents the "infimum", and then the Wasserstein distance solves the minimum value of the expectation value in all potential joint distributions; intuitively, It can be understood as the cost of moving a pile of soil from position P1 to position P2 under the path planning γ, and the Wasserstein distance represents the minimum cost under the optimal path planning. One of the key advantages of the Wasserstein distance metric method is that it can measure the distance between two probability distributions, even if their support sets do not overlap.

[0144] The first probability distribution set and the second probability distribution set can be obtained by a parameter estimation method, a maximum likelihood estimation method, a Bayesian estimation method, a non-parametric estimation method, a histogram method, or the like, and can be selected according to actual conditions, and the embodiments of the present application do not limit this.

[0145] Further, the second similarity value JS between the initial base class sample and the initial new class sample is measured by the following formula <7>:

[0146]

[0147] The KL is a Kullback-Leibler divergence (KL divergence) formula. The KL divergence, also known as relative entropy, is a non-symmetrical measure for measuring the difference between two probability distributions. Formula <7> uses the JS divergence as one of the measurement methods by processing the KL divergence, and thus the similarity result obtained by the symmetrical JS divergence measurement method is more accurate when there is a cross between the two probability distributions.

[0148] In some embodiments, based on the new class feature matrix and the base class feature matrix, a third similarity value of the initial new class sample and each initial base class sample in the third dimension is determined, including the following steps:

[0149] (C.1) Calculate the sum of squares between any first matrix element and any second matrix element to obtain an element square value.

[0150] (C.2) Square root processing is performed on the element square value to obtain an element square root value.

[0151] (C.3) Based on the element square root value, the third similarity value is determined.

[0152] The steps (C.1) to (C.3) are described in detail as follows.

[0153] The third similarity value F between the initial base class sample and the initial new class sample is measured by the following formula <8>:

[0154]

[0155] Wherein, M i is the new class feature matrix, M′ jis the base class feature matrix; the Frobenius norm is a generalization of the Euclidean norm (i.e., the length or magnitude of a vector) to the context of matrices. Therefore, the third similarity value in formula <8> is obtained by squaring and summing any first matrix element and any second matrix element, and then taking the square root. Moreover, the Frobenius norm metric method is regarded as the Euclidean length of a matrix in high-dimensional space, which makes the output result easy to understand and interpret.

[0156] In step 103, the target base class sample is selected from the plurality of initial base class samples based on the similarity matrix.

[0157] The step 103 is described in detail below.

[0158] In some embodiments, the initial base class sample that is more similar to the initial new class sample has more similar feature representation information, that is, different initial base class samples have different correction contributions to the initial new class sample. In the traditional correction method, no weight or fixed weight is used to measure the degree of contribution of the initial base class sample, resulting in that the initial base class sample with low similarity occupies a larger weight, and thus causes unreasonable correction of the initial new class sample. Compared with this, in the case of obtaining the similarity matrix based on the composite metric, the embodiments of the present application also dynamically select the target base class sample with higher similarity and better matching from the plurality of initial base class samples based on the dynamically adjusted K value, thereby improving the feature distribution authenticity of the subsequent target new class sample.

[0159] In some embodiments, the target base class sample is selected from the plurality of initial base class samples based on the similarity matrix, including the following steps:

[0160] (103.a.1) Based on a preset normalization operation operator, the similarity matrix of all initial base class samples is normalized to obtain an adjusted similarity matrix, wherein the adjusted similarity matrix includes a plurality of third matrix elements.

[0161] (103.a.2) For any adjusted similarity matrix, if all third matrix elements are greater than a preset similarity threshold, the corresponding initial base class sample is determined as the target base class sample.

[0162] The steps (103.a.1) to (103.a.2) are described in detail below.

[0163] The adjusted similarity matrix is obtained by the following formula <9>:

[0164] W m = f(D cm ) <9>

[0165] Wherein, D cmis a similarity matrix; the similarity matrix of each initial base class sample is normalized to obtain D cm corresponding adjustment similarity matrix W m ; f(·) is a preset normalization operation operator.

[0166] Further, W m includes a plurality of third matrix elements, and if all the third matrix elements are greater than a preset similarity threshold T, it is determined that the corresponding initial base class sample is a target base class sample. In the embodiment of the application, the number of target base class samples is set to K, and if the number of adjustment similarity matrices whose all third matrix elements are greater than the similarity threshold T is less than K, the adjustment similarity matrices can be selected in turn from the most to the least according to the number of third matrix elements greater than the preset similarity threshold T, until the number of selected adjustment similarity matrices is equal to K. The specific value of K can be adaptively adjusted according to actual conditions, and the similarity threshold T is usually set to between 0 and 1.

[0167] Alternatively, the most matching k are first selected from the plurality of adjustment similarity matrices, and then if the selected k adjustment similarity matrices exist below the similarity threshold T, the lower similarity ones are sequentially removed until the number of target base class samples is K, K < k.

[0168] It can be understood that if there are samples with very low similarity in the target base class samples, the deviation between the actual target new class sample obtained by subsequent correction and the expected target new class sample will be large, and the embodiment of the application determines the target base class sample based on the dynamic K value, which can select more matching target base class samples for correction of the initial new class sample on the basis of ensuring similarity.

[0169] As shown in Figure 4 , Figure 4 is an optional correction schematic diagram of the image classification method provided by the embodiment of the application, and the initial new class sample in the dashed circle is required to select the initial base class sample with higher similarity (closer distance in the figure) from a plurality of initial base class samples as the target base class sample in order to ensure the authenticity of the feature distribution of the target new class sample after correction. The initial base class sample in the dashed box is the target base class sample, and the initial base class sample outside the dashed box represents a sample that is not selected, which is not shown in the figure. The target base class sample with higher similarity to the initial new class sample has a higher weight value represented by the element in the similarity matrix, and thus the sample with higher similarity can provide more contribution to the correction of the initial new class sample, so that the feature distribution of the target new class sample after correction is more authentic and accurate.

[0170] At step 104, the initial new-class sample is dynamically corrected according to the target base-class sample and the similarity matrix corresponding to the target base-class sample, to obtain a target new-class sample.

[0171] The step 104 is described in detail as follows.

[0172] In some embodiments, in the image classification task of few-shot learning, the initial new-class sample is dynamically corrected based on the target base-class sample and the similarity matrix corresponding to the target base-class sample, which can compensate for the problem of lack of effective training features in the initial new-class sample by accurately capturing more similar target base-class samples, and then facilitate subsequent use of the target new-class sample with more real and accurate feature distribution to help improve the image classification accuracy of the trained model.

[0173] In some embodiments, the initial new-class sample is dynamically corrected according to the target base-class sample and the similarity matrix corresponding to the target base-class sample to obtain a target new-class sample, including the following steps:

[0174] (104.a.1) Based on the plurality of initial base-class features, obtain the initial base-class covariance value corresponding to each initial base-class sample.

[0175] (104.a.2) Based on the similarity matrix, update the initial base-class mean and the initial base-class covariance value respectively to obtain a target base-class mean and a target base-class covariance value.

[0176] (104.a.3) Based on the target base-class mean and the target base-class covariance value, migrate the target base-class sample to the initial new-class sample to obtain a dynamically corrected target new-class sample.

[0177] The steps (104.a.1) to (104.a.3) are described in detail as follows.

[0178] Wherein, the initial base-class covariance value matrix ∑ corresponding to each initial base-class sample is determined by the following formula <10> j :

[0179]

[0180] Wherein, n j represents the initial base-class feature quantity of the jth initial base-class sample, x i represents the ith initial base-class feature in the jth initial base-class sample; ∑ represents summation processing; T represents transposition processing.

[0181] Further, the target base-class mean and the target base-class covariance value

[0182]

[0183] wherein λ j is the jth third matrix element in the similarity matrix W m is the eigenvector from the updated support class samples S, and δ is a compensation for the intra-class variation of the initial new class samples, which can be adaptively adjusted according to actual conditions.

[0184] Further, based on the target base class mean and the target base class covariance, the feature statistical information of the K target base class samples is migrated to the initial new class samples, and the target new class samples y i satisfy the following formulas <11> and <12>:

[0185]

[0186] wherein, is the learned feature distribution set; and N is the feature distribution between the target new class samples and any target base class sample.

[0187] wherein, the feature statistical information migration can be the classification label of the target base class sample migrated to the initial new class sample to supplement more description of the initial new class sample and help the training of the initial image classification model; or, the target new class sample with a feature distribution consistent with the real distribution of the initial new class sample can be generated according to the feature statistical information, and a large number of target new class samples with correct feature distribution can help improve the image recognition and discrimination ability of the initial image classification model for new class samples in the subsequent process.

[0188] Step 105, training the initial image classification model using the target new class samples until a training stopping condition is reached, to obtain a trained image classification model.

[0189] The step 105 is described in detail as follows.

[0190] wherein, the training stopping condition can be that the training round number reaches a preset maximum value; or, when the change of the loss function used to evaluate the training degree is less than a certain threshold value, it is considered that the initial image classification model has converged, and the training is stopped; or, in the training process, an independent validation set is used to evaluate the performance of the model, and if the performance (such as accuracy, F1 score, etc.) on the validation set no longer improves, the training is stopped. Of course, the training stopping condition can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0191] Further, as Figure 3As shown, after obtaining the trained image classification model, in order to evaluate the generalization ability of the initial image classification model, the trained image classification model needs to be verified by using query class samples. If the image classification model can accurately classify the target object represented by the query class sample, it is generally considered that the training result of the image classification model is good.

[0192] As shown, Figure 3 The target object represented by the query class sample is one of the new class categories to which the support class sample belongs. In this way, the image classification processing ability of the image classification model for new class samples is tested. The number of query class samples can be set according to actual conditions. For example, one corresponding query class sample can be set for each new class category, and the training is completed when each query class sample is correctly classified; or the training is completed when the classification accuracy of all query class samples reaches a preset accuracy threshold.

[0193] Step 106, obtaining a target image to be classified, inputting the target image into the trained image classification model, and obtaining a corresponding target classification result.

[0194] The step 106 is described in detail below.

[0195] The target image refers to an image that needs to be classified by the trained image classification model. The target image can be obtained by real-time shooting by a user or obtained from an open source database. The application embodiment does not limit the way of obtaining the target image.

[0196] The target classification result refers to the predicted class obtained by the image classification model after classifying the input target image. The target classification result is usually a class label or a probability distribution of a group of classes. The target classification result corresponds to different output modes according to different task forms: for single-label classification tasks, the target classification result is a class label; for multi-label classification tasks, the target classification result is a plurality of class labels and their corresponding confidence; the specific output mode can be adjusted according to actual conditions.

[0197] As shown, Figure 5 The method in Figure 5 is an optional flowchart of the training method of the image classification model provided by the application embodiment, Figure 5 The method can include but is not limited to the following steps 101 to 106. When the image classification device executes the training method of the image classification model, the specific process is as follows. It needs to be first pointed out that the order of steps 201 to 205 in Figure 5 The order of steps can be adjusted or some steps can be reduced or increased according to actual needs.

[0198] Step 201, obtaining initial samples, the initial samples including an initial new-class sample and a plurality of initial base-class samples;

[0199] Step 202, inputting the initial samples into an initial image classification model, and determining a similarity matrix corresponding to the initial samples based on a preset composite metric function, the similarity matrix being used to represent a feature matching degree of the initial new-class sample and each of the initial base-class samples in multiple dimensions;

[0200] Step 203, selecting a target base-class sample from the plurality of initial base-class samples based on the similarity matrix;

[0201] Step 204, dynamically correcting the initial new-class sample according to the target base-class sample and the similarity matrix corresponding to the target base-class sample, to obtain a target new-class sample;

[0202] Step 205, training the initial image classification model by using the target new-class sample until a training stop condition is reached, to obtain a trained image classification model.

[0203] The steps 201 to 205 are described in detail as follows.

[0204] The initial samples refer to a data set used for training or evaluating a machine learning model. Generally, the initial samples include a large number of initial base-class samples in different base-class categories with known classification labels, and a small number of initial new-class samples in different new-class categories with known classification labels, which represent initial new-class samples that have never been seen or are unfamiliar to the initial image classification model. Generally, the number of base-class categories is much larger than the number of new-class categories, each base-class category includes a plurality of initial base-class samples, and each new-class category includes one or two initial new-class samples. For ease of description, the embodiments of the present application are exemplarily described in the case that each initial base-class sample belongs to a base-class category and each initial new-class sample belongs to a new-class category.

[0205] Further, in the image classification task of the embodiments of the present application, the initial base-class samples and the initial new-class samples are all image samples. The classification granularity of the classification labels of the initial samples can be set according to actual conditions, which is not limited in the embodiments of the present application. For example, an initial base-class sample a has a species classification label “cat”, and another initial base-class sample b has a human body part classification label “human eye”.

[0206] The initial image classification model can already well perform image classification processing on the initial base class samples, but the image classification result accuracy of the initial new class samples is poor without training by the image classification method proposed in the embodiments of the present application. The embodiments of the present application take the initial image classification model as a starting point, and further optimize the training, thereby improving the recognition ability of the image classification model obtained by training on the initial new class samples.

[0207] It should be noted that the initial base class samples are usually obtained from an open source database, and the initial new class samples can be obtained from an open source database or obtained by real-time shooting. The initial sample acquisition approach is not unique, and the embodiments of the present application are only used for example description, but not limited thereto.

[0208] The composite measurement function is a function for comprehensively evaluating the similarity degree between the two types of samples from multiple dimensions. The matrix elements in the similarity matrix generated based on the result of the composite measurement function are used to represent the similarity scores between different feature points of the two types of samples. Compared with the feature distribution description obtained by the traditional method in a single dimension, the similarity matrix determined according to the composite measurement function in the embodiments of the present application provides more position and distribution information, thereby improving the accuracy of the feature distribution description between the two types of samples.

[0209] In some embodiments, the initial base class samples that are more similar to the initial new class samples have more similar feature representation information, that is, different initial base class samples have different correction contributions to the initial new class samples. In the traditional correction method, no weight or fixed weight is used to measure the contribution degree of the initial base class sample, which leads to that the initial base class sample with low similarity occupies a larger weight, thereby causing unreasonable correction of the initial new class sample. Compared with this, in the case of obtaining the similarity matrix based on the composite measurement, the embodiments of the present application further dynamically select the target base class sample that is more similar and more matched to the initial new class sample from multiple initial base class samples based on the dynamically adjusted K value, thereby improving the feature distribution authenticity of the subsequent target new class sample.

[0210] In some embodiments, in the image classification task of few-shot learning, the initial new class sample is dynamically corrected based on the target base class sample and the corresponding similarity matrix, which can effectively cope with the problems of data scarcity, class imbalance and insufficient feature representation in few-shot learning, thereby improving the image classification accuracy of the model obtained by training.

[0211] The training stopping condition can be: the number of training epochs reaches a preset maximum value; or, when the change in the loss function used to evaluate the training level is less than a certain threshold, the initial image classification model is considered to have converged, and training stops; or, during training, an independent validation set is used to evaluate the model's performance, and if the performance on the validation set (such as accuracy, F1 score, etc.) no longer improves, training stops. Of course, the training stopping condition can be set according to the actual situation, and the embodiments of this application do not impose any limitations on it.

[0212] Furthermore, such as Figure 3 As shown, after obtaining the trained image classification model, in order to evaluate the generalization ability of the initial image classification model, it is necessary to use query class samples to verify the trained image classification model. If the image classification model can accurately classify the target object represented by the query class samples, it is generally considered that the training result of the image classification model is good.

[0213] Furthermore, steps 201 to 205 are similar to steps 101 to 105. Therefore, other details of steps 201 to 205 can be gleaned from steps 101 to 105, and will not be repeated here.

[0214] like Figure 6 As shown, Figure 6 This is an optional flowchart of an image classification apparatus provided in an embodiment of this application. The image classification apparatus includes the following modules 301 to 306:

[0215] The acquisition module 301 is used to acquire initial samples, which include initial new class samples and multiple initial base class samples;

[0216] The similarity matrix determination module 302 is used to input the initial sample into the initial image classification model and determine the corresponding similarity matrix of the initial sample based on a preset composite metric function. The similarity matrix is ​​used to characterize the feature matching degree between the initial new class sample and each of the initial base class samples in multiple dimensions.

[0217] The target base class sample selection module 303 is used to select a target base class sample from a plurality of initial base class samples based on the similarity matrix;

[0218] Correction module 304 is used to dynamically correct the initial new class sample based on the target base class sample and the similarity matrix corresponding to the target base class sample to obtain the target new class sample;

[0219] Training module 305 is used to train the initial image classification model using the target new class samples until the training stopping condition is met, so as to obtain a trained image classification model.

[0220] The application module 306 is configured to obtain a target image to be classified, input the target image into the trained image classification model, and obtain a corresponding target classification result.

[0221] The image classification method, the training method of the image classification model, and the related device provided in the present application include obtaining initial samples, the initial samples including an initial new-class sample and a plurality of initial base-class samples; inputting the initial samples into an initial image classification model, and determining a similarity matrix corresponding to the initial samples based on a preset composite metric function, the similarity matrix being used to represent the feature matching degree of the initial new-class sample and each initial base-class sample in a plurality of dimensions; the similarity matrix determined based on the composite metric function provides more position and distribution information, and improves the accuracy of the feature distribution description between the two types of samples; then, based on the similarity matrix, a target base-class sample is selected from the plurality of initial base-class samples; the initial new-class sample is dynamically corrected based on the target base-class sample and the similarity matrix corresponding to the target base-class sample, so as to compensate for the problem of the scarcity of effective training features in the initial new-class sample by accurately capturing a more similar target base-class sample, and obtain a target new-class sample; then, the target new-class sample with more real and accurate feature distribution is used to help improve the image classification accuracy of the model obtained through training; then, the initial image classification model is trained using the target new-class sample until a training stop condition is reached, and a trained image classification model is obtained; after that, a target image to be classified is obtained, the target image is input into the trained image classification model, and a corresponding target classification result is obtained.

[0222] In addition, the image classification device is also deployed with a graphics processing unit (GPU), which is a hardware specially designed for efficient processing of image and graphics related calculations. The powerful parallel computing capability of the GPU can significantly improve the image processing performance, and thus help improve the training efficiency of the initial image classification model and the image classification efficiency of the image classification model obtained after training in actual application.

[0223] The GPU can be:

[0224] (1) NVIDIA: such as Quadro / RTX A series, Tesla / V100 / H100 series, Jetson series, etc.

[0225] (2) Intel: such as Arc series, Ponte Vecchio / Gaudi2 series, etc.

[0226] It should be noted that the specific type of GPU can be set according to actual conditions, and the above is only an example for illustration, and does not represent a limitation of the embodiments of the present application.

[0227] The specific implementation of the image classification apparatus is basically the same as the specific embodiments of the above image classification method, and will not be repeated here.

[0228] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above image classification method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0229] As Figure 7 shown, Figure 7 is a hardware structure schematic diagram of an electronic device provided by the embodiments of the present application, and the electronic device includes:

[0230] The processor 401 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0231] The memory 402 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 402 and are called and executed by the processor 401 to implement the image classification method of the embodiments of the present application.

[0232] The input / output interface 403 is used to realize information input and output.

[0233] The communication interface 404 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0234] The bus 405 transmits information between various components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device.

[0235] The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are communicatively connected with each other through a bus 405.

[0236] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the image classification method.

[0237] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0238] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0239] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0240] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0241] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0242] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order unless otherwise stated below.

[0243] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0244] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0245] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0246] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0247] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0248] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An image classification method, characterized in that, include: Obtain initial samples, which include initial new class samples and multiple initial base class samples. Each initial new class sample includes multiple initial new class features, and each initial base class sample includes multiple initial base class features. The initial samples are input into the initial image classification model, and the new class feature matrix of the initial new class sample and the base class feature matrix of the initial base class sample are determined respectively. Based on multiple initial new class features and multiple initial base class features, a first similarity value between the initial new class sample and each initial base class sample in the first dimension is determined, and a second similarity value between the initial new class sample and each initial base class sample in the second dimension is determined. Based on the new class feature matrix and the base class feature matrix, the third similarity value of the initial new class sample and each of the initial base class samples in the third dimension is determined. Based on a composite metric function with preset hyperparameters, the first similarity value, the second similarity value, and the third similarity value are fused to obtain a similarity matrix. The similarity matrix is ​​used to characterize the feature matching degree between the initial new class sample and each of the initial base class samples in multiple dimensions. Based on a preset normalization operator, the similarity matrix of all the initial base class samples is normalized to obtain an adjusted similarity matrix, wherein the adjusted similarity matrix includes multiple third matrix elements. For any of the adjusted similarity matrices, if all elements of the third matrix are greater than a preset similarity threshold, the corresponding initial base class sample is determined as the target base class sample. Based on the multiple initial base class features, the corresponding initial base class covariance value of each initial base class sample is obtained; Based on the similarity matrix, the initial base class mean and the initial base class covariance are updated respectively to obtain the target base class mean and the target base class covariance. Based on the target base class mean and the target base class covariance, the target base class samples are transferred to the initial new class samples to obtain dynamically corrected target new class samples; The initial image classification model is trained using the target new class samples until the training stopping condition is met, resulting in a trained image classification model. The target image to be classified is obtained, and the target image is input into the trained image classification model to obtain the corresponding target classification result.

2. The image classification method according to claim 1, characterized in that, The step of determining the new class feature matrix of the initial new class sample and the base class feature matrix of the initial base class sample includes: The initial new class features are transposed to obtain the corresponding new class transposed features; The new class feature matrix is ​​obtained based on the cross product of each of the initial new class features and the corresponding new class transpose features; Based on the multiple initial base class features, the mean value of each initial base class sample is obtained; The initial base class mean is transposed to obtain the corresponding base class transposed mean. The base class feature matrix is ​​obtained based on the cross product of the initial base class mean and the corresponding base class transpose mean.

3. The image classification method according to claim 1, characterized in that, The step of determining a first similarity value between the initial new class sample and each of the initial base class samples in a first dimension, based on multiple initial new class features and multiple initial base class features, and determining a second similarity value between the initial new class sample and each of the initial base class samples in a second dimension, includes: Determine the first probability distribution set of the initial new class samples, and determine the second probability distribution set of the initial base class samples; A joint distribution set is formed based on the first probability distribution set and the second probability distribution set; The first similarity value is determined based on the expected distance between any of the initial new class features and any of the initial base class features in the joint distribution set; The second similarity value is determined based on the divergence value between the first probability distribution set and the second probability distribution set.

4. The image classification method according to claim 1, characterized in that, The new class feature matrix includes multiple first matrix elements, and the base class feature matrix includes multiple second matrix elements; The step of determining the third similarity value of the initial new class sample and each of the initial base class samples in the third dimension based on the new class feature matrix and the base class feature matrix includes: Calculate the sum of squares between any element of the first matrix and any element of the second matrix to obtain the square value of the element; The square root of the element is obtained by taking the square root of the element. The third similarity value is determined based on the square root value of the element.

5. The image classification method according to claim 1, characterized in that, Before determining the similarity matrix corresponding to the initial sample based on the preset composite metric function, the method further includes: If the preset distribution hyperparameter value of the initial image classification model is not equal to the preset parameter threshold, logarithmic processing is performed on all the initial new class samples to obtain the updated initial new class samples; If the preset distribution hyperparameter value of the initial image classification model is equal to the preset parameter threshold, exponential processing is performed on all the initial new class samples to obtain the updated initial new class samples.

6. A training method for an image classification model, characterized in that, include: Obtain initial samples, which include initial new class samples and multiple initial base class samples. Each initial new class sample includes multiple initial new class features, and each initial base class sample includes multiple initial base class features. The initial samples are input into the initial image classification model, and the new class feature matrix of the initial new class sample and the base class feature matrix of the initial base class sample are determined respectively. Based on multiple initial new class features and multiple initial base class features, a first similarity value between the initial new class sample and each initial base class sample in the first dimension is determined, and a second similarity value between the initial new class sample and each initial base class sample in the second dimension is determined. Based on the new class feature matrix and the base class feature matrix, the third similarity value of the initial new class sample and each of the initial base class samples in the third dimension is determined. Based on a composite metric function with preset hyperparameters, the first similarity value, the second similarity value, and the third similarity value are fused to obtain a similarity matrix. The similarity matrix is ​​used to characterize the feature matching degree between the initial new class sample and each of the initial base class samples in multiple dimensions. Based on a preset normalization operator, the similarity matrix of all the initial base class samples is normalized to obtain an adjusted similarity matrix, wherein the adjusted similarity matrix includes multiple third matrix elements. For any of the adjusted similarity matrices, if all elements of the third matrix are greater than a preset similarity threshold, the corresponding initial base class sample is determined as the target base class sample. Based on the multiple initial base class features, the corresponding initial base class covariance value of each initial base class sample is obtained; Based on the similarity matrix, the initial base class mean and the initial base class covariance are updated respectively to obtain the target base class mean and the target base class covariance. Based on the target base class mean and the target base class covariance, the target base class samples are transferred to the initial new class samples to obtain dynamically corrected target new class samples; The initial image classification model is trained using the target new class samples until the training stops, resulting in a well-trained image classification model.

7. An image classification device, characterized in that, include: An acquisition module is used to acquire initial samples, the initial samples including initial new class samples and multiple initial base class samples, the initial new class samples including multiple initial new class features, and each initial base class sample including multiple initial base class features; The similarity matrix determination module is used to input the initial sample into the initial image classification model and determine the new class feature matrix of the initial new class sample and the base class feature matrix of the initial base class sample, respectively. Based on multiple initial new class features and multiple initial base class features, a first similarity value between the initial new class sample and each initial base class sample in the first dimension is determined, and a second similarity value between the initial new class sample and each initial base class sample in the second dimension is determined. Based on the new class feature matrix and the base class feature matrix, the third similarity value of the initial new class sample and each of the initial base class samples in the third dimension is determined. The similarity matrix is ​​obtained by fusing the first similarity value, the second similarity value, and the third similarity value according to a composite metric function with preset hyperparameters. The similarity matrix is ​​used to characterize the feature matching degree between the initial new class sample and each of the initial base class samples in multiple dimensions. The target base class sample selection module and the correction module are used to normalize the similarity matrix of all the initial base class samples based on a preset normalization operation operator to obtain an adjusted similarity matrix, wherein the adjusted similarity matrix includes multiple third matrix elements. For any of the adjusted similarity matrices, if all elements of the third matrix are greater than a preset similarity threshold, the corresponding initial base class sample is determined as the target base class sample. Based on the multiple initial base class features, the corresponding initial base class covariance value of each initial base class sample is obtained; Based on the similarity matrix, the initial base class mean and the initial base class covariance are updated respectively to obtain the target base class mean and the target base class covariance. Based on the target base class mean and the target base class covariance, the target base class samples are transferred to the initial new class samples to obtain dynamically corrected target new class samples; The training module is used to train the initial image classification model using the target new class samples until the training stopping condition is met, so as to obtain a trained image classification model. The application module is used to acquire the target image to be classified, input the target image into the trained image classification model, and obtain the corresponding target classification result.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image classification method according to any one of claims 1 to 5 or the training method of the image classification model according to claim 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification method according to any one of claims 1 to 5 or the training method of the image classification model according to claim 6.

Images