Method, device and computer equipment for training visual recognition model

Abstract

The application relates to a training method and device of a visual recognition model and computer equipment. The method comprises the following steps: inputting each training sample into an initial visual recognition model respectively, fusing the training sample and a target category corresponding to the training sample, obtaining corresponding fusion features, and obtaining an original output vector corresponding to the training sample; determining a reference category number corresponding to the training sample based on a target category sample number corresponding to a target category to which the training sample belongs, determining a target reference category corresponding to the reference category number in the original output vector and a confidence degree corresponding to the target reference category; determining loss information corresponding to the training sample based on the confidence degrees corresponding to each target reference category corresponding to the training sample and the target category, and obtaining a target visual recognition model based on the loss information. The model trained by the method can accurately classify difficult negative samples and reduce category confusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a training method and device of a visual recognition model, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the development of image processing technology, a corresponding model can be used to process images to obtain a processing result related to the images, such as identifying the category corresponding to the image. To ensure that the result of processing the image by the model is correct, the model is often trained using a training data set to adjust the parameters of the model.

[0003] However, data in the real world often follows a long-tail distribution. A small number of categories correspond to a sufficient number of samples, which are referred to as head classes. A large number of categories correspond to a small number of samples, which are referred to as tail classes. A model trained under this distribution usually has high accuracy on head class data, but low accuracy on tail class data. Moreover, for difficult negative samples, i.e., training image data from non-target categories but having strong similarity in vision or semantics with the target category, it is also easy to be incorrectly identified as the target category. The interference of such difficult negative samples seriously reduces the overall performance of the model in long-tail visual recognition. SUMMARY

[0004] Therefore, it is necessary to provide a training method, device, computer device, computer readable storage medium and computer program product of a visual recognition model, which can improve the processing accuracy of the trained model on tail class data without reducing the accuracy of head class, enhance class perception in representation learning, distinguish difficult negative samples and true positive samples, accurately classify difficult negative samples, and reduce the occurrence of class confusion.

[0005] In a first aspect, the present application provides a training method of a visual recognition model, comprising:

[0006] obtaining a plurality of training samples;

[0007] For each training sample, the training sample is input into an initial visual recognition model, the training sample and a target category corresponding to the training sample are fused based on the initial visual recognition model to obtain a corresponding fusion feature, and an original output vector corresponding to the training sample is obtained based on the fusion feature, the original output vector including each reference region category corresponding to the training sample and a confidence degree corresponding to the reference region category;

[0008] determine the number of reference region categories corresponding to the training sample based on the number of target category samples corresponding to the target category to which the training sample belongs, and determine the target reference region category corresponding to the training sample and the confidence degree corresponding to the target reference region category based on the number of reference region categories and the original output vector corresponding to the training sample;

[0009] determine the probability information corresponding to each target reference region category based on the confidence degree corresponding to the target reference region category corresponding to the training sample, and determine the loss information corresponding to the training sample based on the probability information corresponding to each target reference region category corresponding to the training sample and the target category corresponding to the training sample;

[0010] adjust the model parameters of the initial visual recognition model based on the loss information corresponding to each training sample to obtain the target visual recognition model.

[0011] In a second aspect, the present application further provides a training device of a visual recognition model, comprising:

[0012] The acquisition module is configured to acquire a plurality of training samples.

[0013] The feature module is configured to input each training sample into an initial visual recognition model, perform fusion processing on the training sample and the target category corresponding to the training sample based on the initial visual recognition model, obtain corresponding fusion features, and obtain an original output vector corresponding to the training sample based on the fusion features, wherein the original output vector comprises each reference region category corresponding to the training sample and the confidence degree corresponding to the reference region category.

[0014] The distinguishing module is configured to determine the number of reference region categories corresponding to the training sample based on the number of target category samples corresponding to the target category to which the training sample belongs, and determine the target reference region category corresponding to the training sample and the confidence degree corresponding to the target reference region category based on the number of reference region categories and the original output vector corresponding to the training sample.

[0015] The loss module is configured to determine the probability information corresponding to each target reference region category based on the confidence degree corresponding to the target reference region category corresponding to the training sample, and determine the loss information corresponding to the training sample based on the probability information corresponding to each target reference region category corresponding to the training sample and the target category corresponding to the training sample.

[0016] The adjustment module is configured to adjust the model parameters of the initial visual recognition model based on the loss information corresponding to each training sample to obtain the target visual recognition model.

[0017] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above method when executing the computer program.

[0018] In a fourth aspect, the present application also provides a computer readable storage medium, having stored thereon a computer program, which when executed by a processor implements the steps described in the above method.

[0019] In a fifth aspect, the present application also provides a computer program product, comprising a computer program which when executed by a processor implements the steps described in the above method.

[0020] The training method and device of the visual recognition model, the computer device, the computer readable storage medium and the computer program product described above, by obtaining a plurality of training samples, for each training sample, respectively inputting each training sample into an initial visual recognition model, based on the initial visual recognition model, the training sample and the target category corresponding to the training sample are fused to obtain the corresponding fusion feature, so that the original output vector obtained based on the fusion feature can carry the corresponding category information, that is, the category semantic information is injected into the fusion feature, thereby improving the feature discrimination degree under the long-tail distribution. Then, based on the target category sample number corresponding to the target category to which the training belongs, the reference category number corresponding to the training sample is determined, the target reference category and the confidence degree corresponding to the target reference category are determined based on the reference category number and the original output vector corresponding to the training sample, the probability information corresponding to each target reference category is determined based on the confidence degree corresponding to the target reference category corresponding to the training sample, the loss information corresponding to the training sample is determined based on the probability information corresponding to each target reference category corresponding to the training sample and the target category, and the model parameters of the visual recognition model are adjusted based on the loss information corresponding to each training sample to obtain a target visual recognition model. In this way, when training the initial image encoding model, the number of candidate categories can be determined according to the number of samples, so that more candidate categories can be selected for tail class data to calculate the loss information, so that the discrimination between tail class and other classes can be strengthened, the discrimination degree between difficult categories can be improved, and the model can learn a better decision boundary to improve the accuracy of recognizing tail data categories without compromising the accuracy of head data categories. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0022] Figure 1 It is a flowchart of the training method of the visual recognition model in an embodiment.

[0023] Figure 2 A structural block diagram of an image encoding layer and a category encoding layer in one embodiment;

[0024] Figure 3 A model performance diagram corresponding to a target visual recognition model in one embodiment;

[0025] Figure 4 A structural block diagram of a training device of a visual recognition model in one embodiment;

[0026] Figure 5 An internal structure diagram of a computer device in one embodiment;

[0027] Figure 6 An internal structure diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0028] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0029] In one embodiment, as shown in Figure 1 a training method of a visual recognition model is provided, and the present embodiment takes the method applied to a terminal as an example for illustration. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is realized through the interaction of the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In the present embodiment, the method includes the following steps:

[0030] In step 102, a plurality of training samples are obtained.

[0031] The plurality of training samples are input into an initial visual recognition model as a same batch for training the initial visual recognition model. The plurality of training samples are in a long-tail distribution. The training samples are, for example, image samples, text samples, etc.

[0032] Exemplarily, the long-tail distribution dataset is randomly sampled into batches, and each batch of training samples is input into the initial visual recognition model for forward propagation to obtain an original output vector corresponding to each training sample in each batch.

[0033] In this embodiment, since the original output vector needs to be adjusted based on the frequency of the category to which the currently processed training sample belongs in the plurality of training samples, after obtaining the plurality of training samples, the plurality of training samples can be processed first to obtain the number of training samples corresponding to each category in the plurality of training samples, thereby determining the frequency of each category appearing in the plurality of training samples.

[0034] In step 104, for each training sample, the training sample is input into the initial visual recognition model, the training sample and the target category corresponding to the training sample are fused based on the initial visual recognition model to obtain a corresponding fusion feature, and the original output vector corresponding to the training sample is obtained based on the fusion feature.

[0035] The original output vector includes each reference region category corresponding to the training sample and the confidence of the reference region category. For example, the original initial vector is {cat: 2.5, dog: 1.5}.

[0036] The initial visual recognition model processes each training sample respectively to obtain an original output vector corresponding to each training sample.

[0037] Reference Figure 2 In this embodiment, the initial visual recognition model includes an image encoding layer 201 and a category encoding layer 202. The category encoding layer 202 is inserted into each feature extraction layer of the image encoder (i.e., the image encoding layer 201 described above) to modulate the output result of the image encoder (i.e., to fuse the output result of the category encoding layer 202 on the basis of the result output by the image encoding layer 201 alone). The category encoding layer 202 is used to extract features of the training sample and the target category corresponding to the training sample, so that the initial visual recognition model extracts features of the training sample that fuse the relevant information of the target category. In this way, the discrimination between the features obtained based on the category can be improved, which is beneficial to help the model make better decisions on the boundary. Figure 2 Only one category encoding layer 202 is inserted into the corresponding feature extraction layer in the above embodiment.

[0038] In the conventional technology, only the original feature extraction layer in the image encoder is used to extract features of the input training sample, and the training sample is not processed by the category encoding layer.

[0039] Exemplarily, the training sample is processed by the feature extraction layer in the image encoder to obtain original features. The visual features input into the image encoder are input into the category encoding layer 202, and the following steps are performed: the visual features are down-sampled to obtain down-sampled features; and the target category (i.e. the actual category) corresponding to the training sample is extracted to obtain category semantic features; the down-sampled features and the category semantic features are spliced to obtain spliced features; the spliced features are processed based on an activation function to obtain activated features; and finally the activated features are up-sampled to obtain target features with the same dimension as the original features. Then, based on the original features output by each feature extraction layer and the corresponding target features, the original output vector corresponding to the training sample is determined.

[0040] Exemplarily, after obtaining the fusion features, the fusion features can be input into the cosine classifier to obtain the original output vector corresponding to the training sample.

[0041] In the training process of the initial visual recognition model, the parameters adjusted based on the subsequently obtained loss information are the parameters in the category encoding layer 202 and / or the cosine classifier.

[0042] Step 106: determining the reference category number corresponding to the training sample based on the target category sample number corresponding to the target category to which the training sample belongs, and determining the target reference category and the confidence degree corresponding to the target reference category based on the reference category number and the original output vector corresponding to the training sample.

[0043] The target category sample number is the frequency of the target category appearing in the input multiple training samples. If the number of training samples belonging to the target category in the multiple training samples is 10, then the corresponding target category sample number is 10. The reference category number is determined based on the target category sample number. The reference category number is used to filter the reference category used to obtain the loss information subsequently. The target reference category is a category similar to the target category. The original initial vector records the confidence degree of the training sample in each category. However, because the input data may have a long-tail distribution, the model may not be able to well distinguish the tail class data from other class data, and therefore the probability corresponding to the confidence degree in the original output vector output by the model may not be the target category. Therefore, in order to reduce the risk of feature confusion and misclassification, more similar categories are selected from the original output vector for the tail class sample in this embodiment, so that the model can better distinguish the difficult negative sample.

[0044] Exemplarily, the target class sample number corresponding to the target class to which the current training sample belongs is determined, and the reference category number corresponding thereto is determined based on the target class sample number. The reference category number and the target class sample number can be in an inverse proportional relationship. That is, when the target class sample number is larger, the determined reference category number is smaller. When the target class sample number is smaller, the determined reference category number is larger. In this way, for tail class data, more similar categories can be selected, thereby strengthening the differentiation between the tail class and other categories, so as to better distinguish difficult negative samples. After the reference category number is determined, the target reference categories with a total number of the reference category number and the confidence corresponding to the target reference categories in the original output vector are determined based on the original output vector. Exemplarily, after the target reference categories are determined based on the original output vector, the original output vector is adjusted using the class prior information to obtain an adjusted output vector, and the confidence corresponding to the target reference categories is selected from the adjusted output vector. In other embodiments, the values in the original output vector can also be directly used as the confidence corresponding to the target reference categories. Among them, the value corresponding to the target reference category in the original output vector is greater than the value corresponding to the remaining categories. That is, the value corresponding to the target reference category in the original output vector is in the top. For example, it is determined that the reference category number corresponding to the current training sample is 10, then 10 target reference categories with the confidence in the top 10 are selected from the original output vector, and the confidence corresponding to the target reference categories is selected from the adjusted output vector corresponding to the original output vector, so as to determine the probability information corresponding to each target reference category based on the confidence, thereby calculating the loss information corresponding to the current training sample.

[0045] Optionally, the reference category number can be determined by obtaining a preset total value, and determining the reference category number based on the difference between the preset total value and the target class sample number.

[0046] Optionally, the reference category number can also be determined by the difference between the sample number of the category with the largest sample number in the plurality of training samples and the target class sample number.

[0047] In step 108, the probability information corresponding to each target reference category is determined based on the confidence corresponding to the target reference category corresponding to the training sample, and the loss information corresponding to the training sample is determined based on the probability information corresponding to each target reference category corresponding to the training sample and the target class corresponding thereto.

[0048] Exemplarily, after determining the confidence corresponding to each target reference region category classification corresponding to the current training sample, the probability information corresponding to each target reference region category classification is determined based on the confidence corresponding to each target reference region category classification. It can be understood that the sum of the probability information corresponding to each target reference region category classification is 1. Then, based on the target category corresponding to the current training sample and each target reference region category classification, the label (for example, one-hot label) corresponding to the current training sample is determined. Then, based on the label corresponding to the current training sample and the probability information corresponding to each target reference region category classification corresponding to the current training sample, cross entropy processing is performed to obtain the loss information corresponding to the current training sample.

[0049] Optionally, after obtaining the original output vector, the original output vector can be adjusted to obtain an updated output vector (i.e., the above-mentioned adjusted output vector). The subsequent steps are then performed based on the updated output vector. Specifically, the original output vector is adjusted using class prior information, which is obtained by statistically processing a plurality of training samples. The adjustment can occur in the inference phase (i.e., the model application phase) or the original output vector can be adjusted in the training phase to affect the loss and thus the training optimization process of the model, so that the final obtained model can obtain a more fair prediction for the input sample. The CE loss form of Logit Adjustment is shown in the following formula:

[0050] l y,f x = -log e f y x + τ·log π y ∑ y ' ∈[L] e f y ' x + τ·log π y '

[0051] wherein, is the corresponding item of the class in the original output vector output by the model, is a temperature coefficient, is the prior of the class in the training set, i.e., the inverse of the number of corresponding samples.

[0052] Step 110, adjusting the model parameters of the initial visual recognition model based on the loss information corresponding to each training sample to obtain a target visual recognition model.

[0053] In this embodiment, the adjusted model parameters include all parameters in the class encoding layer 202 (i.e., the CMA module in the Figure 2 and the model parameters in the cosine classifier, and the original pre-training parameters in the image encoding layer 201 of the visual recognition model are not adjusted. For reference Figure 2The parameters of each module marked as a frozen layer (e.g., an adaptive multi-layer perceptron, a normalization layer, a context fusion layer, a normalization layer) are not adjusted, each module marked as a training layer (e.g., an up-sampling module, a down-sampling module, a class embedding module) is trained, and the corresponding parameters are adjusted.

[0054] In the training method of the visual recognition model, a plurality of training samples are obtained, for each training sample, each training sample is input into an initial visual recognition model, the training sample and a target category corresponding to the training sample are fused based on the initial visual recognition model, and a corresponding fusion feature is obtained. The original output vector obtained based on the fusion feature can carry corresponding category information, i.e., the category semantic information is injected into the fusion feature, thereby improving the feature discrimination degree under the long-tail distribution. Then, based on the target category sample number corresponding to the target category to which the training belongs, the reference category number corresponding to the training sample is determined, the target reference category corresponding to the training sample and the confidence corresponding to the target reference category are selected from the original output vector of the training sample based on the reference category number, the probability information corresponding to each target reference category is determined based on the confidence corresponding to the target reference category corresponding to the training sample, the loss information corresponding to the training sample is determined based on the probability information corresponding to each target reference category corresponding to the training sample and the target category corresponding to the training sample, and the model parameters of the visual recognition model are adjusted based on the loss information corresponding to each training sample to obtain a target visual recognition model. In this way, when training the initial image encoding model, the number of candidate categories can be determined according to the number of samples, so that more candidate categories can be selected for tail class data to calculate the loss information, thereby strengthening the distinction between tail classes and other classes, helping to improve the discrimination degree between difficult categories, and further helping the model to learn a better decision boundary and improve the accuracy of recognizing tail data categories without compromising the accuracy of head data categories.

[0055] In an example embodiment, the reference category number can be determined by whether the target category to which the current training sample belongs is a head category or a tail category in the plurality of training samples (e.g., the current training batch), and thus the number of samples belonging to each category in the plurality of training samples needs to be determined to determine whether the category to which the current training sample belongs is a head category or a tail category. If it is a head category, the corresponding target reference category number is smaller. If it is a tail category, the corresponding target reference category number is larger. Illustratively, determining the reference category number corresponding to the training sample based on the target category sample number corresponding to the target category to which the training sample belongs includes: determining the number of training samples corresponding to each category in the plurality of training samples based on the category label information corresponding to each training sample; determining a first training sample number and a second training sample number based on the extreme value in the number of training samples corresponding to each category; the first training sample number is greater than the second training sample number; determining the target category sample number corresponding to the training sample from the number of training samples corresponding to each category based on the category label information corresponding to the training sample; and determining the reference category number corresponding to the training sample based on the target category sample number, the first training sample number, and the second training sample number.

[0056] In the example, there are three categories in the plurality of training samples, the number of training samples corresponding to category 1 is 10, the number of training samples corresponding to category 2 is 5, and the number of training samples corresponding to category 3 is 1. The first training sample number is the maximum number of the number of training samples corresponding to each category. The second training sample number is the minimum number of the number of training samples corresponding to each category. After determining the number of training samples corresponding to each category in the plurality of training samples, the specific position of the target category sample number corresponding to the target category to which the current training sample belongs between the first training sample number and the second training sample number can be determined, i.e., whether the target category belongs to a head category or a tail category can be determined, and thus the target reference category and the corresponding confidence required for subsequent loss information calculation can be determined.

[0057] Optionally, the reference category number can also be determined based on a difference between the first training sample number and the target category sample number. For example, after determining the difference, the reference category number corresponding to the difference is determined according to a preset association table. The preset association table is used to represent the association relationship between the difference and the reference category number. For example, when the difference is 1, the corresponding reference category number is 1. When the difference is 10, the corresponding reference category number is 5. That is, the difference between the first training sample number and the target category sample number is in direct proportion to the reference category number. It can be understood that if the reference category number is determined based on the difference between the second training sample number and the target category sample number, the difference between the second training sample number and the target category sample number can be in inverse proportion to the reference category number.

[0058] Optionally, the reference category number can also be determined based on a difference between the first training sample number and the target category sample number, and a difference between the first training sample number and the second training sample number.

[0059] In the embodiment, the reference category number corresponding to the current training sample is determined based on the target category sample number corresponding to the current training sample by determining the first training sample number and the second training sample number corresponding to the plurality of training samples. In this way, more candidate samples can be selected for the tail class samples, so that the model can improve the classification ability for difficult negative samples and reduce the confusion of samples between categories.

[0060] In one of the embodiments, the following steps can be performed to determine the reference category number: determining a first difference between the first training sample number and the target category sample number, determining a second difference between the first training sample number and the second training sample number; determining the reference category number corresponding to the training sample based on a ratio between the first difference and the second difference.

[0061] Specifically, after determining the first difference between the first training sample number and the target category sample number, and determining the second difference between the first training sample number and the second training sample number, a ratio between the first difference and the second difference is obtained. Then, the ratio is processed based on a preset hyperparameter to obtain the reference category number. Specifically, the preset hyperparameter includes a first hyperparameter and a second hyperparameter. The first hyperparameter is greater than the second hyperparameter. A hyperparameter difference between the first hyperparameter and the second hyperparameter is obtained, a product of the hyperparameter difference and the ratio is obtained, and the reference category number is determined based on a sum of the product and the second hyperparameter.

[0062] For example, the following formula is used to obtain the reference category number:

[0063]

[0064] wherein, denotes the reference region class number, denotes the first training sample number, denotes the second training sample number, denotes the target class sample number, denotes the first hyperparameter, denotes the second hyperparameter.

[0065] In this embodiment, by determining the first difference value between the maximum training sample number and the target class sample number, the second difference value between the maximum training sample number and the minimum training sample number is determined, and the reference region class number corresponding to the training sample is determined based on the ratio between the first difference value and the second difference value. In this way, it can be accurately determined whether the current training sample belongs to the head class sample or the tail class sample in a plurality of training samples (such as the current training batch), thereby improving the accuracy of the subsequent determined reference region class number.

[0066] It can be understood that in order to train the initial visual recognition model, the prediction loss is determined based on the gap between the class predicted by the initial visual recognition model and the actual class (i.e., the target class described above), and the model parameters of the initial visual recognition model are adjusted based on the prediction loss until the output prediction loss is within the preset loss range. Therefore, after determining the reference region class number, the target reference region class number corresponding to the reference region class number can be determined according to the original output vector, and the loss information corresponding to the current training sample can be obtained based on these target reference region class numbers and their corresponding confidence. Illustratively, based on the probability information corresponding to each target reference region class corresponding to the training sample and the corresponding target class, the loss information corresponding to the training sample is determined, including: when the target class exists in each of the target reference region classes, the target probability information corresponding to the target class is determined from the probability information corresponding to each of the target reference region classes, and the loss information corresponding to the training sample is determined based on the target probability information; when the target class does not exist in each of the target reference region classes, the minimum confidence in the confidence of each of the target reference region classes that is screened out is updated based on the confidence of the target class, the probability information corresponding to each of the target reference region classes is determined based on the updated confidence of each of the target reference region classes, the target probability information corresponding to the target class is determined from the probability information corresponding to each of the target reference region classes, and the loss information corresponding to the training sample is determined based on the target probability information.

[0067] The target reference category class refers to a set of possible classes predicted by the initial visual recognition model for the training sample. The target class is the real label corresponding to the training sample. The probability information corresponds to the confidence score of each class predicted by the initial visual recognition model. The loss information is an error index for guiding the optimization of the model parameters of the initial visual recognition model.

[0068] Specifically, after obtaining the original output vector, the original output vector is adjusted using the class prior information to obtain an adjusted output vector. The class prior information can be the prior information of each class in the training data set, for example, the reciprocal of the number of samples of each class.

[0069] Then, the target reference category classes corresponding to the number of reference category classes are determined based on the original output vector. For example, the class corresponding to the value arranged in the front is selected from the original output vector as the target reference category class. Then, the confidence corresponding to the target reference category class is selected from the adjusted output vector. The probability information corresponding to each target reference category class is determined based on the confidence corresponding to each target reference category class.

[0070] If the selected target reference category class includes the target class, the probability information corresponding to each target reference category class and the label information (for example, one-hot label) corresponding to the target class are cross-entropy loss processed to obtain the final loss information.

[0071] If the selected target reference category class does not include the target class, after determining the target reference category classes corresponding to the number of reference category classes based on the original output vector, the smallest confidence in the confidence corresponding to each target reference category class is replaced by the confidence corresponding to the target class (obtained from the adjusted output vector) to obtain the updated confidence corresponding to each target reference category class. That is, the updated confidence corresponding to each target reference category class includes the confidence corresponding to the target class. Then, the confidence corresponding to each target reference category class is normalized to obtain the probability information corresponding to each target reference category class, i.e., the probability distribution information. Then, the probability distribution information and the label information corresponding to the target class are cross-entropy loss processed to obtain the final loss information.

[0072] In this embodiment, by further classifying the target class and several difficult classes, the confidence of the target class is improved, and the confidence of the difficult class is weakened, thereby further improving the discrimination ability of the model.

[0073] In an exemplary embodiment, the visual recognition model is adjusted, and the class encoding layer 202 is inserted into each layer of the image encoding layer 201, as shown in FIG. 2B. The image encoding layer 201 and the class encoding layer 202 are connected in series, and the output of the image encoding layer 201 is input into the class encoding layer 202. Figure 2The input training sample is forward propagated in each network layer of the image encoder until the output of the last layer. The sample feature output by the image encoder is input into the cosine classifier to obtain the original output vector corresponding to the training sample. For example, based on the initial visual recognition model, the training sample and the target category corresponding to the training sample are fused to obtain the corresponding fusion feature, and the original output vector corresponding to the training sample is obtained based on the fusion feature, including: obtaining the visual feature input into the image encoding layer of the initial visual recognition model; performing feature extraction on the visual feature to obtain the corresponding original feature; encoding the target category corresponding to the training sample based on the category encoding layer in the initial visual recognition model to obtain the category semantic feature, and splicing the category semantic feature with the visual feature to obtain the target feature; obtaining the corresponding fusion feature based on the original feature and the target feature; inputting the fusion feature into the cosine classifier in the initial visual recognition model to obtain the original output vector corresponding to the training sample.

[0074] The initial visual recognition model refers to a deep learning network for feature extraction and classification. The image encoding layer 201 is, for example, each feature extraction layer of the Transformer architecture. The original feature is the feature obtained by the image encoding layer 201 performing feature extraction on the training sample. The category semantic feature is a high-dimensional semantic vector obtained by fusing the target category (such as "cat" and "dog") and the visual feature through the category encoding layer 202 (including a Concat module for feature splicing, a ReLU function for introducing nonlinear activation, an up-sampling and down-sampling module for adjusting feature resolution). The visual feature can be the feature input into the image encoding layer 201 from the previous layer, or the training sample. The target feature is the fusion representation formed by splicing the category semantic feature and the visual feature, and the fusion feature is an enhanced feature obtained by further combining the original feature and the target feature. The cosine classifier is a module for classifying by calculating the cosine similarity between the feature vector and the category prototype.

[0075] Specifically, first, the visual feature input into the image encoding layer 201 is obtained. The visual feature can be the feature output by the previous feature extraction layer in the image encoding layer 201, or the feature output after performing feature extraction on the training sample. Then, the input visual feature is further extracted by the current feature extraction layer to obtain the original feature and enhance the representation capability. In this process, the category encoding layer 202 converts the target category into a category semantic feature, and splices the category semantic feature with the processed visual feature to form the target feature after processing the visual feature. This process can inject category information into the feature by explicitly fusing semantic priori and visual content, and improve the feature discrimination in the long-tail distribution. Subsequently, the original feature and the target feature are combined to generate the fusion feature. Finally, the fusion feature is input into the cosine classifier to obtain the original output vector.

[0076] In this embodiment, compared with the method of relying only on single modal features or simple fusion in the prior art, the semantic-guided feature interaction and fusion can more effectively adapt to the class distribution offset or sample scarcity scenario, while enhancing the robustness of the model to visual noise, because the class semantic features can be used as a regularization constraint to suppress irrelevant visual interference. In this embodiment, the class prior information is introduced into the original output vector, which can help the initial visual recognition model learn better decision boundaries and improve the recognition accuracy of the tail classes without compromising the head class accuracy.

[0077] It can be understood that after obtaining the target visual recognition model, the target visual recognition model can be used for image class recognition. However, in the process of applying the target visual recognition model, the image input into the model does not know the class to which the image belongs in advance, so the average class semantic feature needs to be used, and the average class semantic feature is obtained by processing each target class input in the process of training the initial visual recognition model through the class encoding layer 202. For example, after obtaining the target visual recognition model, the image data is feature-extracted based on the image encoding layer 201 in the target visual recognition model to obtain corresponding original features; the fusion features are obtained based on the original features and the average class semantic features; the fusion features are input into the cosine classifier in the target visual recognition model to obtain the original output vector corresponding to the image data; and the predicted class corresponding to the image data is determined based on the original output vector corresponding to the image data.

[0078] Specifically, referring to Figure 2 , the feature to be processed input into the class encoding layer 202 is first down-sampled, and then the average semantic feature is obtained from the target class table, and the average semantic feature is spliced with the down-sampled feature to be processed to obtain a spliced feature. Then, the spliced feature is fused with the down-sampled feature to be processed (for example, the elements at corresponding positions in the spliced feature and the down-sampled feature to be processed are added), to obtain an activated feature. Then, the activated feature is input into an activation function, and then subjected to up-sampling processing to increase the dimension to the dimension corresponding to the feature to be processed, and then multiplied by a scaling parameter, to finally obtain the target feature output by the class encoding layer 202. The subsequent steps are then performed to obtain the corresponding original output vector.

[0079] In this embodiment, by using the average semantic feature obtained based on the training samples in the process of applying the model, since the average semantic feature can be used as a stable anchor point to suppress sample noise interference, the representation deviation of the minority class samples can be effectively alleviated.

[0080] In some embodiments, the determination process of the above target category table can refer to the following steps: determining a semantic category corresponding to the category semantic feature in the target category table, and storing the category semantic feature into a feature set associated with the semantic category; obtaining a category frequency corresponding to the semantic category in the target category table, processing the feature set associated with the semantic category based on the category frequency, and obtaining a category semantic feature corresponding to the semantic category. That is, the target category table is obtained by summarizing the features obtained by processing the target categories corresponding to each training sample during training.

[0081] In addition, in order to facilitate image processing combined with category semantic information in subsequent model application, each category semantic feature can be weighted and processed based on the category frequency corresponding to each semantic category in the target category table to obtain an average category semantic feature. That is, in subsequent model application, the application image and the average category semantic feature are fused and processed based on the target visual recognition model to obtain corresponding fusion features, and subsequent processing is performed based on the fusion features.

[0082] Specifically, each category semantic feature is stored in the target category table. If the semantic category corresponding to the category semantic feature does not exist in the current target category table, a new empty feature set is created in the target category table, the empty set corresponds to the semantic category corresponding to the category semantic feature, and the category semantic feature is stored in the empty set. First, find the semantic category corresponding to the current category semantic feature in the target category table, and store it in the associated feature set. This step aims to gradually accumulate the semantic representation of the same sample; then, obtain a target category frequency corresponding to the semantic category in the target category table, process the feature set associated with the semantic category based on the target category frequency, and obtain a category semantic feature corresponding to the semantic category.

[0083] For example, the semantic feature of a certain category stored in the target category table is the average of the category semantic features of all samples of this category. For category A, the current category feature and the number of samples of category A that have been processed are recorded. When the next semantic feature belonging to category A is processed, the category feature is superimposed and the number is increased by 1 to obtain the category semantic feature corresponding to category A. It can be understood that the category feature corresponding to each category stored in the target category table can be processed according to the above steps.

[0084] After that, the average class semantic feature can be updated based on the content of the target class table together after the training ends. Or during the training process, the average class semantic feature is updated every time a new class semantic feature is stored. To obtain the average class semantic feature, the frequency of occurrence of each class in the target class table (i.e., class frequency) can be obtained, and the class semantic feature is weighted based on the class frequency to generate the average class semantic feature, which ensures that the average feature can more accurately reflect the typical distribution of the class.

[0085] In this embodiment, by obtaining the average class semantic feature, the fairness of the model in the long-tail distribution classification task can be significantly improved. Compared with the simple average or frequency-ignoring method in the existing scheme, the frequency weighting can effectively suppress the overfitting of the majority class and enhance the representation ability of the minority class, so that the model can still maintain stable classification performance in the class imbalance scene.

[0086] In one embodiment, after obtaining the target visual recognition model, the target visual recognition model can be tested to verify the performance of the target visual recognition model. In order to further improve the performance of the model and achieve a suitable trade-off between efficiency and accuracy, the following steps can be referred to:

[0087] Illustratively, a test image set is obtained; for each test image in the test image set, the resolution of the test image is adjusted to a plurality of target resolutions, obtaining a plurality of adjusted test images corresponding thereto; a preset size image is respectively cut from the center region of each adjusted test image to obtain a corresponding target test image; each target test image is respectively input into the target visual recognition model to obtain a corresponding model output result, and a prediction result corresponding to the test image is determined based on each model output result. Wherein, the preset size is the size required by the model input.

[0088] Specifically, each test image is adjusted to three different resolutions: R, R+δ_1 and R+δ_2, where R is the resolution accepted by the model, and δ is the change amount of the resolution adjustment. Then, the center is cropped to a size of R. The three target test images of size R obtained are input into the target visual recognition model to obtain corresponding model output results. Then, the average result corresponding to the three model output results is taken as the prediction result of the test image. This process generates 3 views with scale perception while keeping the spatial size consistent, thereby enhancing the robustness to target scale changes.

[0089] Exemplarily, the following test method can also be adopted: obtaining a test image set; for each test image in the test image set, scaling the test image to a target size, and then cropping a plurality of target test images from the test image; inputting each target test image into the target visual recognition model respectively to obtain a corresponding model output result, and determining a prediction result corresponding to the test image based on each model output result; wherein each target test image is located at a different position in the test image, and the size of each target test image is a preset size.

[0090] Specifically, the test image is first scaled to wherein represents the change in scaling, and then three positions (for example, the upper left corner, the center, and the lower right corner) are selected to crop a region of size R. In this way, the diversity of the test image set in space can be improved.

[0091] Optionally, in the process of determining a plurality of target test images based on a plurality of target test images, the cropped target position image can also be horizontally flipped, and more target test images can be obtained based on the target position image before flipping and the target position image after flipping. Then, the prediction result corresponding to the target test image (i.e., the corresponding target model output result) is obtained based on the average of the model output results corresponding to each of the expanded target test images. In this way, the diversity of the test image set in the direction can be improved.

[0092] In one specific embodiment, the initial visual recognition model is a multi-layer neural network such as Figure 2The structure is shown. The input training sample is forward propagated in each layer of the image encoding layer 201 in the initial visual recognition model until the output of the last layer. In the image encoding layer 201 in the initial visual recognition model, the input training sample (or the result output by the last layer) is sequentially subjected to a normalization layer (LN layer), a context fusion layer (for example, MHSA layer), a normalization layer (LN layer), and an adaptive multilayer perception (Adapt MLP layer), and then the processed result of the layer is output. In this embodiment, a category encoding layer 202 is inserted in each layer of the image encoding layer 201 in the initial visual recognition model. Specifically, the input feature of the category encoding layer 202 is the feature processed by the context fusion layer, and the output feature of the category encoding layer 202 is connected with the output result corresponding to the image encoding layer 201 to obtain a feature with superimposed category semantics. Further, the original output vector is obtained according to the feature with superimposed semantics. In the category encoding layer 202, the input feature is first subjected to down-sampling processing to obtain a down-sampled feature, and the target category to which the current input feature belongs is encoded to obtain a category semantic feature. The down-sampled feature and the category semantic feature are spliced to obtain a spliced feature. The spliced feature is used to connect with the down-sampled feature to obtain a connection feature. The connection feature is input into an activation function to obtain an activation feature. Finally, the activation feature is up-sampled and scaled to obtain the corresponding target feature. For details, see the following process:

[0093] I. Training process of the initial visual recognition model.

[0094] First, a plurality of training samples are obtained.

[0095] Specifically, when determining the target reference region category later, it is necessary to use the number of training samples belonging to the target category in the plurality of training samples to which the target category of the current training sample belongs. That is, the number of training samples belonging to the target category in the plurality of training samples needs to be used.

[0096] For each training sample, the training sample is input into the initial visual recognition model to obtain a visual feature input into the image encoding layer 201 of the initial visual recognition model.

[0097] The visual feature is a feature obtained after being processed by the LN layer and the MHSA layer.

[0098] In the following formula, represents the length of the visual feature encoding, represents the dimension, represents the total number of categories.

[0099] Specifically, the visual feature is normalized by a layer normalization (LN) operation, and then down-sampled to , to obtain . Wherein, , the formula is as follows:

[0100]

[0101] Next, the target class corresponding to the training sample is encoded based on the class encoding layer 202 in the initial visual recognition model to obtain a class semantic feature. The class semantic feature and the visual feature are spliced to obtain a target feature.

[0102] That is, the class semantic information (i.e., the above target class) is injected into the visual feature and modulated. Specifically, for an input sample of a certain class (i.e., the current training sample, is the target class to which the current training sample belongs), its class label label (corresponding to the target class) is sent to the class embedding module (Class Emb module) in Figure 2 to obtain a class semantic encoding (i.e., the above class semantic feature) . Then, the above obtained and are spliced (Concat) to each other, and then pass through a multi-layer perceptron to obtain . The formula is as follows:

[0103]

[0104] The visual feature injected with the class semantic information is then up-sampled to the original feature space, which facilitates its calculation and forward propagation in the original feature encoder (i.e., the above image encoder). Specifically, the above and corresponding position elements are added, sent to a ReLU activation function, and then pass through a mapping to elevate to the dimension of the original feature , and then multiply a learnable scaling parameter to obtain the final output of the CMA module (i.e., the above target feature). The formula is as follows:

[0105]

[0106] In the category encoding layer, a category encoding table (i.e., the following target category table) is also constructed to store the category semantic features corresponding to each training sample. For example, the semantic category corresponding to the category semantic feature is determined in the target category table, and the category semantic feature is stored in the feature set associated with the semantic category; the category frequency corresponding to the semantic category in the target category table is obtained, and the feature set associated with the semantic category is processed based on the category frequency to obtain the category semantic feature corresponding to the semantic category. Specifically, a category encoding table is constructed In the labeled training, the category semantic encoding obtained in the above process is added to the corresponding category encoding in . That is, the semantic category corresponding to the category semantic feature is determined in the target category table, and the category semantic feature is stored in the feature set associated with the semantic category; the target category frequency corresponding to the semantic category in the target category table is obtained, and the feature set associated with the semantic category is processed based on the target category frequency to obtain the category semantic feature corresponding to the semantic category.

[0107] In addition, the average category semantic feature needs to be obtained based on the content of the target category table. After obtaining the target category table, each category semantic feature is weighted based on the category frequency corresponding to each semantic category in the target category table to obtain the average category semantic feature. That is, the category semantic encoding of all categories is weighted based on the category frequency to obtain the average category semantic encoding (i.e., the above average category semantic feature), and the formula is as follows:

[0108]

[0109] wherein, represents the number of semantic categories stored in the target category table, represents the Cth category semantic feature stored in the target category table, represents the category frequency corresponding to the Cth category semantic feature.

[0110] Thus, in the inference stage (i.e., the following application process) and the case of missing labels, the average category semantic encoding is used to replace in formula (2) to perform the category semantic information injection operation.

[0111] After that, the visual features are extracted to obtain corresponding original features; based on the original features and target features, corresponding fusion features are obtained; the fusion features are input into the cosine classifier in the initial visual recognition model to obtain an original output vector corresponding to the training sample. The original output vector includes the classification of each reference area corresponding to the training sample and the confidence corresponding to the classification of the reference area.

[0112] Specifically, the output class token (i.e., class embedding, which acts as a global representation of the entire input) is selected as the feature of the input image sample. The feature is sent to the cosine classifier to output Logit (i.e., the above-mentioned original output vector).

[0113] During the training process, the loss data between the predicted class and the target class output by the model also needs to be calculated, and then the parameters of the model are adjusted based on the loss data to optimize the model to obtain the target visual recognition model. That is, the loss is calculated according to the Logit output by the model, and the model is trained by back propagation. After obtaining the Logit output by the model, the loss value is calculated using the loss function of the TLA method proposed in this embodiment. The detailed process of the TLA method is shown below. Then, the gradient is calculated and the model parameters are updated, and the iteration is continuously performed until the model converges. The converged model can be used for testing. Note that the label information of the sample is not known during testing (i.e., the inference stage), so the CMA module needs to use the average class semantic encoding mentioned above when injecting class semantic information into the feature encoding.

[0114] The process of the TLA method can be seen from the following steps:

[0115] First, determine the number of training samples corresponding to each class based on the target class to which each training sample belongs; determine the first training sample number and the second training sample number from the number of training samples corresponding to each class; the first training sample number is the largest training sample number among the number of training samples corresponding to each class, and the second training sample number is the smallest training sample number among the number of training samples corresponding to each class; based on the target class corresponding to the training sample, determine the number of target class samples corresponding to the training sample from the number of training samples corresponding to each class; determine the first difference between the first training sample number and the number of target class samples, and determine the second difference between the first training sample number and the second training sample number; based on the ratio between the first difference and the second difference, determine the number of reference area classes corresponding to the training sample.

[0116] Specifically, for an image sample with a label , the original Logit output after forward propagation of the model is denoted as , and the candidate class indices with the highest scores are selected from The formula is as follows:

[0117]

[0118] wherein, The function represents the index corresponding to the K maximum values.

[0119] The value of K is determined by the following formula:

[0120]

[0121] wherein, represents the number of candidate classes of any category (i.e., the number of reference region category classes corresponding to the target category to which each training sample belongs), and respectively represent the maximum and minimum number of class samples, and are two hyperparameters used to control the value range. The value of the number of candidate classes is negatively correlated with the number of class samples, ensuring that the tail classes with fewer samples can be distinguished from more difficult category classes during training.

[0122] Then, the corresponding target reference region category and the confidence of the target reference region category corresponding to the target reference region category are selected from the original output vector corresponding to the training sample based on the number of reference region category classes; the probability information corresponding to each target reference region category is determined based on the confidence corresponding to the target reference region category corresponding to the training sample.

[0123] Specifically, the original output vector can also be adjusted to obtain an updated output vector (i.e., the above-mentioned adjusted output vector), and then the confidence corresponding to the target reference region category is determined based on the updated output vector. In this way, the reliability of the subsequent results can be improved. Specifically, the original Logit is subjected to Logit Adjustment to obtain the adjusted Logit, denoted as .

[0124] In the updated output vector, the confidence corresponding to each candidate category index is selected, i.e., a new Logit vector Z adj(K) = Z adj [T] Then, the confidence corresponding to each candidate category index The corresponding confidence obtains the corresponding probability distribution information. If the target reference region category obtained before does not include the target category (that is, the index corresponding to the target category is not included in the above candidate category index), the minimum confidence of each confidence corresponding to each target reference region category is replaced by the confidence corresponding to the target category. Then, the step of normalizing each confidence corresponding to the updated target reference region category to obtain the corresponding probability distribution information is performed. That is, if the target reference region category obtained before includes the target category, the normalization processing can be directly performed to obtain the corresponding probability distribution information. Specifically, the new Logit vector composed of the above formula can be normalized to a probability distribution (that is, the probability information corresponding to each target reference region category is determined based on the confidence corresponding to each updated target reference region category):

[0125]

[0126] Subsequently, the above probability distribution and the one-hot label after re-normalization are cross-entropy loss, and the Top-K loss is obtained, that is, .

[0127] Specifically, the final loss information can be composed of the above loss information and the loss used in the traditional technology. That is . Wherein, represents the loss information used in the Logit Adjustment method in the traditional technology.

[0128] Finally, the model parameters of the visual recognition model are adjusted based on the loss information corresponding to each training sample to obtain the target visual recognition model.

[0129] II. Application process of the target visual recognition model.

[0130] First, based on the target visual recognition model, the image data is feature extracted to obtain the corresponding original feature.

[0131] Next, based on the original feature and the average category semantic feature, the corresponding fusion feature is obtained.

[0132] Then, the fusion feature is input into the cosine classifier in the target visual recognition model to obtain the original output vector corresponding to the image data.

[0133] Finally, based on the original output vector corresponding to the image data, the prediction category corresponding to the image data is determined.

[0134] The above steps address the limitation imposed by difficult negative samples on long-tail visual recognition performance. This study also explores the shortcomings of traditional Logit Adjustment techniques in correctly distinguishing difficult negative samples and mitigating inter-class confusion, revealing that misclassification of difficult negative samples has a particularly severe impact on semantically scarce tail classes. The proposed CM-TLA technique mitigates the interference of difficult negative samples on classification and suppresses inter-class confusion, improving tail class accuracy without sacrificing head class accuracy. Specifically, the TLA method selects several candidate classes most easily confused for each sample and performs deeper discrimination among these classes. Simultaneously, this method adaptively adjusts the number of candidate classes based on the frequency of the sample's class: for tail class samples, more candidate classes are selected for more detailed distinction, effectively alleviating inter-class confusion. The CM fine-tuning module (i.e., the CMA module) injects class information into image encoding, promoting the model's learning of more discriminative features, thus synergizing with TLA to effectively improve the model's ability to distinguish easily confused samples. This technique achieves outstanding performance on long-tail tasks, providing a new solution to the long-tail problem and advancing the field of long-tail visual recognition.

[0135] Furthermore, this application proposes two ensemble testing methods to reduce testing time while ensuring testing performance, achieving a suitable trade-off between testing efficiency and accuracy. In the first ensemble testing method (i.e., the TTE-M method below), each test sample (different from the training sample) is adjusted to three different resolutions, resulting in three corresponding adjusted test samples. Each test sample and its three corresponding adjusted test samples are then input into the target visual recognition model for testing. Specifically, the three different resolutions are R0, R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R2, R3, R4, R5, R6, R7, R8, R9, R10 ... and Where R is the resolution accepted by the model, The amount of change is adjusted for resolution. Then it is centered and cropped to size R. This process generates three scale-aware views while maintaining spatial consistency, thus enhancing robustness to changes in the target scale.

[0136] In the second integration testing method (i.e., the TTE-F method below), the test samples are scaled to obtain scaled test samples. Multiple target regions are then selected from these scaled test samples and horizontally flipped to obtain target test samples. These target test samples are then input into the target visual recognition model to test the model. Specifically, the test samples are first scaled to... ,in This indicates the scaling change, and then three positions (e.g., top left, center, and bottom right) are selected to crop to a size of [size missing]. The cropped regions are then flipped horizontally to finally obtain 6 deterministic views, thus improving the diversity of space and direction.

[0137] By the above integrated test method, the test accuracy of the model can be improved, and the inference overhead is significantly reduced compared to the original TTE technology.

[0138] The application effect corresponding to the above process can be seen from the following content:

[0139] The CM-TLA algorithm proposed in the embodiments of the present application is tested on four commonly used long-tail distribution datasets: CIFAR100-LT, ImageNet-LT, Places-LT and iNaturalist 2018, and good results can be achieved. Each table shows the compared method, the backbone network used by the method, and the Top1 accuracy (%). The part in the gray background uses the method proposed in the embodiments of the present application and is marked as Ours. The performance of CM-TLA and its combination with TTE, TTE-M and TTE-F is verified.

[0140] Table 1 shows the results on the CIFAR100-LT dataset, experiments are conducted under various imbalance rates (ImbalanceFactor) 200, 100, 50, 10, and the corresponding classification accuracies under different imbalance rates are shown. Imbalance refers to the ratio of the maximum number of samples to the minimum number of samples, reflecting the degree of imbalance of the dataset. Tables 2, 3 and 4 respectively show the comparison on the three large-scale datasets ImageNet-LT, Places-LT and iNaturalist 2018. All, Head, Med and Tail in the table represent the classification accuracy of the method for the whole, the classification accuracy of the head class, the classification accuracy of the middle class and the classification accuracy of the tail class. As can be seen from the table, the method adopted in the present application achieves excellent performance (see the last four rows of data in each table).

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147] Table 5 shows the ablation experiments for each component. It can be seen that both CMA and TLA modules can improve the performance of the original LIFT method, and the use of both modules improves the most. Table 6 shows the accuracy and inference time of the proposed TTE-M, TTE-F and the original TTE method. As shown in the table, when the TTE method generates 5 perturbation versions for the original test sample (it can be understood that the number of perturbation versions in Table 6 refers to the number of perturbation versions generated for one test sample), the inference time is 173.40 seconds, while our TTE-M method achieves the same accuracy but only takes 131.74 seconds; TTE (TenCrop) generates 10 perturbation versions, and the accuracy is improved to 83.1%, but the inference time is increased to 366.91 seconds. Our TTE-F method achieves an accuracy of 83.0%, which is only 0.1% lower than that of TTE (TenCrop), but the inference time is only 267.21 seconds.

[0148] Ten last classes and ten first classes are selected from the test set, and their sample features are visualized using the T-SNE technique. As shown in Figure 3 the left column (last classes), compared with the results of the LIFT method, the TLA method improves the separability of the classes, and the CM-TLA method further enhances this effect. The CM-TLA method significantly reduces the confused samples, indicating its effectiveness in alleviating difficult negative samples. The right column shows the visualization of the first class samples, and all methods achieve consistent performance, indicating that the improvement of the tail class performance does not come at the expense of the head class accuracy. Overall, the CM-TLA method effectively enhances the representation and discrimination ability of the tail classes while maintaining the stability of the head class performance.

[0149] It should be understood that although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a training device of a visual recognition model for implementing the training method of the visual recognition model.

[0151] In one exemplary embodiment, as shown in Figure 4 A training device 400 of a visual recognition model is provided, comprising an acquisition module 401, a feature module 402, a distinguishing module 403, a loss module 404 and an adjustment module 405, wherein:

[0152] The acquisition module 401 is configured to acquire a plurality of training samples.

[0153] The feature module 402 is configured to, for each training sample, input the training sample into an initial visual recognition model, perform fusion processing on the training sample and a target category corresponding to the training sample based on the initial visual recognition model, obtain corresponding fusion features, and obtain an original output vector corresponding to the training sample based on the fusion features, wherein the original output vector comprises each reference category of the training sample and a confidence degree corresponding to the reference category.

[0154] The distinguishing module 403 is configured to determine a number of reference categories corresponding to the training sample based on a number of target category samples corresponding to a target category to which the training sample belongs, and determine a target reference category corresponding to the training sample and a confidence degree corresponding to the target reference category based on the number of reference categories and the original output vector corresponding to the training sample.

[0155] The loss module 404 is configured to determine probability information corresponding to each target reference category based on the confidence degree corresponding to the target reference category corresponding to the training sample, and determine loss information corresponding to the training sample based on the probability information corresponding to each target reference category corresponding to the training sample and the target category.

[0156] The adjustment module 405 is configured to adjust model parameters of the initial visual recognition model based on the loss information corresponding to each training sample, so as to obtain a target visual recognition model.

[0157] In some embodiments, in determining the reference region class number corresponding to the training sample based on the target class sample number corresponding to the target class to which the training sample belongs, the distinguishing module 403 is specifically configured to: determine the number of training samples corresponding to each class based on the target class to which each training sample belongs; determine a first training sample number and a second training sample number based on the extreme value in the number of training samples corresponding to each class; the first training sample number is greater than the second training sample number; determine the target class sample number corresponding to the target class to which the training sample corresponds from the number of training samples corresponding to each class; determine the reference region class number corresponding to the training sample based on the target class sample number, the first training sample number and the second training sample number.

[0158] In some embodiments, in determining the reference region class number corresponding to the training sample based on the target class sample number corresponding to the target class to which the training sample belongs, the distinguishing module 403 is specifically configured to: determine the number of training samples corresponding to each class based on the target class to which each training sample belongs; determine a first training sample number and a second training sample number based on the extreme value in the number of training samples corresponding to each class; the first training sample number is greater than the second training sample number; determine the target class sample number corresponding to the target class to which the training sample corresponds from the number of training samples corresponding to each class; determine the reference region class number corresponding to the training sample based on the target class sample number, the first training sample number and the second training sample number.

[0159] In some embodiments, in determining the reference region class number corresponding to the training sample based on the target class sample number corresponding to the target class to which the training sample belongs, the distinguishing module 403 is specifically configured to: determine the number of training samples corresponding to each class based on the target class to which each training sample belongs; determine a first training sample number and a second training sample number based on the extreme value in the number of training samples corresponding to each class; the first training sample number is greater than the second training sample number; determine the target class sample number corresponding to the target class to which the training sample corresponds from the number of training samples corresponding to each class; determine the reference region class number corresponding to the training sample based on the target class sample number, the first training sample number and the second training sample number.

[0160] In some embodiments, in the aspect of fusing the training sample and the target category corresponding to the training sample based on the initial visual recognition model to obtain corresponding fusion features, and obtaining the original output vector corresponding to the training sample based on the fusion features, the feature module 402 is specifically configured to: acquire visual features input into an image encoding layer in the initial visual recognition model; perform feature extraction on the visual features to obtain corresponding original features; encode the target category corresponding to the training sample based on a category encoding layer in the initial visual recognition model to obtain category semantic features, and obtain target features based on the category semantic features and the visual features; obtain corresponding fusion features based on the original features and the target features; and input the fusion features into a cosine classifier in the initial visual recognition model to obtain the original output vector corresponding to the training sample.

[0161] In some embodiments, after the aspect of encoding the target category corresponding to the training sample based on the category encoding layer in the initial visual recognition model to obtain category semantic features, the training device 400 of the visual recognition model is further configured to: determine a semantic category corresponding to the category semantic features in a target category table, and store the category semantic features into a feature set associated with the semantic category; acquire a target category frequency corresponding to the semantic category in the target category table, process the feature set associated with the semantic category based on the target category frequency to obtain a category semantic feature corresponding to the semantic category; and perform weighted processing on each category semantic feature based on the category frequency corresponding to each semantic category in the target category table to obtain an average category semantic feature.

[0162] In some embodiments, after obtaining the target visual recognition model, the present application can further include an application device, which is specifically configured to: perform feature extraction on image data based on the target visual recognition model to obtain corresponding original features; obtain corresponding fusion features based on the original features and the average category semantic features; input the fusion features into a cosine classifier in the target visual recognition model to obtain an original output vector corresponding to the image data; and determine a predicted category corresponding to the image data based on the original output vector corresponding to the image data.

[0163] In some embodiments, the training device 400 of the visual recognition model is further configured to: acquire a test image set; for each test image in the test image set, adjust the resolution of the test image to a plurality of target resolutions to obtain a plurality of adjusted test images corresponding thereto; crop a preset size of image from a center region of each of the adjusted test images to obtain a plurality of target test images corresponding thereto; input each of the target test images into the target visual recognition model to obtain a corresponding model output result, and determine a predicted result corresponding to the test image based on each of the model output results.

[0164] In some embodiments, the training apparatus 400 of the visual recognition model is further configured to: obtain a test image set; for each test image in the test image set, scale the test image to a target size, and crop a plurality of target test images from the test image; input each target test image into the target visual recognition model respectively to obtain a corresponding model output result, and determine a prediction result corresponding to the test image based on each model output result; wherein each target test image is located at a different position in the test image, and each target test image has a preset size.

[0165] Each module in the training apparatus of the visual recognition model described above can be realized by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0166] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data related to the training of the visual recognition model. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a training method of a visual recognition model.

[0167] In one exemplary embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 6The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (NFC) or other technologies. The computer program is executed by the processor to realize a training method of a visual recognition model. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0168] Those skilled in the art can understand that, Figure 5 Or Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0169] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the method embodiments described above.

[0170] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in each of the method embodiments described above.

[0171] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to realize the steps in each of the method embodiments described above.

[0172] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0173] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0174] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0175] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for training a visual recognition model, characterized in that, The method comprises: obtaining a plurality of training samples; the training samples comprise at least one of image samples and text samples; for each of the training samples, inputting the training sample into an initial visual recognition model respectively, obtaining visual features of an image encoding layer in the initial visual recognition model, performing feature extraction on the visual features to obtain corresponding original features, encoding a target category corresponding to the training sample based on a category encoding layer in the initial visual recognition model to obtain category semantic features, and obtaining target features based on the category semantic features and the visual features; obtaining corresponding fusion features based on the original features and the target features; inputting the fusion features into a cosine classifier in the initial visual recognition model to obtain an original output vector corresponding to the training sample; the original output vector comprises each reference category of the training sample and a confidence degree corresponding to the reference category; determining a number of reference categories corresponding to the training sample based on a number of target category samples corresponding to a target category to which the training sample belongs, and determining a target reference category corresponding to the training sample and a confidence degree corresponding to the target reference category based on the number of reference categories and the original output vector corresponding to the training sample; determining probability information corresponding to each of the target reference categories based on the confidence degree corresponding to the target reference category corresponding to the training sample, and determining loss information corresponding to the training sample based on the probability information corresponding to each of the target reference categories corresponding to the training sample and the target category corresponding to the training sample; adjusting model parameters of the initial visual recognition model based on the loss information corresponding to each of the training samples to obtain a target visual recognition model.

2. The method of claim 1, wherein, The method comprises: determining a number of training samples corresponding to each category among the plurality of training samples based on the target category to which each of the training samples belongs; determining a first number of training samples and a second number of training samples based on an extreme value among the number of training samples corresponding to each category; the first number of training samples is greater than the second number of training samples; determining the number of target category samples corresponding to the target category corresponding to the training sample from the number of training samples corresponding to each category; determining the number of reference categories corresponding to the training sample based on the number of target category samples, the first number of training samples, and the second number of training samples.

3. The method of claim 2, wherein, The method comprises: determining a first difference between the first number of training samples and the number of target category samples, and determining a second difference between the first number of training samples and the second number of training samples; determining the number of reference categories corresponding to the training sample based on a ratio between the first difference and the second difference.

4. The method of claim 1, wherein, The step of determining the corresponding target reference class and the confidence level corresponding to the target reference class based on the number of reference distinguishable classes and the original output vector corresponding to the training samples includes: The original output vector is adjusted using prior information to obtain an adjusted output vector; Based on the original output vector, determine the target reference distinction category corresponding to the number of reference distinction categories; Filter the confidence levels corresponding to the target reference distinguishing categories from the adjusted output vector; The step of determining the loss information corresponding to the training sample based on the probability information corresponding to each target reference distinguishing category and the corresponding target category includes: When the target category exists in each of the target reference distinction categories, the target probability information corresponding to the target category is determined from the probability information corresponding to each of the target reference distinction categories, and the loss information corresponding to the training sample is determined based on the target probability information. When the target category is not found in any of the target reference distinction categories, the minimum confidence among the confidence scores of the selected target reference distinction categories is updated based on the confidence score corresponding to the target category. The probability information corresponding to each target reference distinction category is determined based on the updated confidence scores of each target reference distinction category. The target probability information corresponding to the target category is determined from the probability information corresponding to each target reference distinction category. The loss information corresponding to the training sample is determined based on the target probability information.

5. The method of claim 1, wherein, After encoding the target category corresponding to the training sample based on the category encoding layer in the initial visual recognition model to obtain category semantic features, the method further includes: Determine the semantic category corresponding to the semantic features of the category in the target category table, and store the semantic features of the category in the feature set associated with the semantic category; Obtain the target category frequency corresponding to the semantic category in the target category table, and process the feature set associated with the semantic category based on the target category frequency to obtain the category semantic features corresponding to the semantic category; The semantic features of each category are weighted based on the category frequency corresponding to each semantic category in the target category table to obtain the average category semantic features.

6. The method of claim 5, wherein, After obtaining the target visual recognition model, the method further includes: Based on the target visual recognition model, feature extraction is performed on the image data to obtain the corresponding original features; Based on the original features and the average category semantic features, the corresponding fused features are obtained; The fused features are input into the cosine classifier in the target visual recognition model to obtain the original output vector corresponding to the image data; Based on the original output vector corresponding to the image data, the predicted category corresponding to the image data is determined.

7. The method of claim 1, wherein, After obtaining the target visual recognition model, the method further includes: Obtain the test image set; For each test image in the test image set, the resolution of the test image is adjusted to a plurality of target resolutions, obtaining a plurality of corresponding adjusted test images; each of the adjusted test images is cropped from the respective corresponding image center region to obtain a preset size image, to obtain a corresponding each target test image; each of the target test images is input into the target visual recognition model to obtain a corresponding each model output result, and a prediction result corresponding to the test image is determined based on each of the model output results; and / or obtain a test image set; For each test image in the test image set, the test image is scaled to a target size, and a plurality of target test images are cropped from the test image; each of the target test images is input into the target visual recognition model to obtain a corresponding each model output result, and a prediction result corresponding to the test image is determined based on each of the model output results; wherein each of the target test images is located at a different position in the test image, and the size of each of the target test images is a preset size.

8. A training device for a visual recognition model, characterized in that, The device comprises: An acquisition module is configured to acquire a plurality of training samples; the training samples comprise at least one of image samples and text samples; A feature module is configured to, for each of the training samples, input the training sample into an initial visual recognition model respectively, acquire visual features of an image encoding layer input into the initial visual recognition model, perform feature extraction on the visual features to obtain corresponding original features, encode a target category corresponding to the training sample based on a category encoding layer in the initial visual recognition model to obtain category semantic features, and obtain target features based on the category semantic features and the visual features; obtain corresponding fusion features based on the original features and the target features; and input the fusion features into a cosine classifier in the initial visual recognition model to obtain an original output vector corresponding to the training sample; the original output vector comprises each reference category of the training sample and a confidence degree corresponding to the reference category. A distinguishing module is configured to determine a number of reference categories corresponding to the training sample based on a number of target category samples corresponding to a target category to which the training sample belongs, determine a target reference category corresponding to the training sample and a confidence degree corresponding to the target reference category based on the number of reference categories and the original output vector corresponding to the training sample. A loss module is configured to determine probability information corresponding to each of the target reference categories based on the confidence degree corresponding to the target reference category corresponding to the training sample, and determine loss information corresponding to the training sample based on the probability information corresponding to each of the target reference categories corresponding to the training sample and the target category corresponding to the training sample. An adjustment module is configured to adjust model parameters of the initial visual recognition model based on the loss information corresponding to each of the training samples to obtain a target visual recognition model. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images