Model training method and device
By obtaining and updating the visual features of the training and verification objects, using technologies such as generative adversarial networks and graph aggregation networks, the problem of poor visual feature quality in zero-sample learning is solved, and the recognition ability and applicability of the model are improved.
Patent Information
- Application Number
- CN201910995424.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-18
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-10-18
AI Technical Summary
The visual features of the verification objects generated by the existing zero-sample learning algorithm are of poor quality, resulting in poor results in practical applications.
By obtaining the data set of training objects, verification objects data sets and category association information, feature extraction and update, using technologies such as generative adversarial networks and graph aggregation networks to generate and update visual features, model training is carried out based on label propagation algorithms, and visual features are improved to improve visual features applicability.
It improves the effect of model training and enhances the recognition ability in actual work. It is suitable for scenes such as image classification, video classification and object detection.
Smart Images

Figure CN112686277B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning technology, and in particular to a method and device for model training. Background Art
[0002] Machine learning in the field of image recognition can be divided into supervised learning and unsupervised learning. Supervised learning relies on large-scale labeled data. For example, training a model for identifying zebras requires a large number of labeled zebra images. Few-shot learning and zero-shot learning are two important subfields of unsupervised learning. Zero-shot learning does not use images of the verification object for training. For example, when training a model for identifying zebras, zero-shot learning does not use images of zebras for training, but can use images of ordinary horses, tigers, and pandas for training, so that the trained model has the ability to recognize zebras. Compared with zero-shot learning, few-shot learning can use a small number of images of verification objects for training. Compared with supervised learning, unsupervised learning has poorer results because it does not directly use images of verification objects for training. Therefore, it is very important to improve the effect of unsupervised learning.
[0003] A zero-shot learning algorithm is provided in the related art, and its specific training process is as follows: first, a training object data set and a verification object data set are input into the model to be trained, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training sample; the verification object data set includes a label of the verification object and category description information of the verification object; the image of the training object can be an image of one training object or multiple images of the training object; the image of the verification object can be an image of one verification object or multiple images of the verification object. Then, the model to be trained performs feature extraction on the image of each training object to obtain the visual features of the training object. Then, the model to be trained generates the visual features of the verification object based on the visual features of the training object, the category description information of the training object, and the category description information of the training object. Finally, the model to be trained is trained based on the visual features of the verification object and the labels corresponding to the visual features of the verification object. It can be seen that the zero-shot learning algorithm provided in the related art converts the original zero-shot training into traditional supervised training by generating the visual features of the verification object.
[0004] In the process of implementing this application, the inventors found that the related technology has at least the following problems:
[0005] In the related art, the visual features of the verification object are generated based only on the visual features of the training object, the category description information of the training object and the category description information of the verification object. The generated visual features of the verification object are of poor quality, so that the model trained based on the visual features has poor effect in actual work. Summary of the invention
[0006] The embodiments of the present application provide a method and device for model training, which can solve the technical problems existing in the related art. The technical solutions of the method and device for model training are as follows:
[0007] In a first aspect, a model training method is provided, the method comprising:
[0008] Acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes a label of the verification object and category description information of the verification object, and each object includes each training object and each verification object;
[0009] Extracting features from the image of the training object to obtain visual features of the training object;
[0010] generating a visual feature of a verification object based on the visual feature of the training object, the category description information of the training object, and the category description information of the verification object;
[0011] Based on the category association information of each object, the visual features of the verification object and the visual features of the training object, updating the visual features of the verification object and the visual features of the training object;
[0012] Model training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0013] Optionally, the generating the visual feature of the verification object based on the visual feature of the training object, the category description information of the training object and the category description information of the verification object includes:
[0014] The visual features of the training object, the category description information of the training object and the category description information of the verification object are input into a generative adversarial network to generate the visual features of the verification object.
[0015] Optionally, the updating of the visual features of the verification object and the visual features of the training object based on the category association information of each object, the visual features of the verification object, and the visual features of the training object includes:
[0016] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0017] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0018] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0019] The model training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object, including:
[0020] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0021] Optionally, the step of inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object includes:
[0022] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0023] For each visual feature to be updated, based on the category association information and sample association information between the visual feature to be updated and other visual features, determine the category association and sample association between the visual feature to be updated and each other visual feature; based on the category association and sample association between the visual feature to be updated and each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0024] Optionally, the performing model training based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object includes:
[0025] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the labels of the updated visual features of the training object, and the labels of the visual features of the verification object.
[0026] Optionally, the performing model training based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object includes:
[0027] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0028] Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set;
[0029] Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0030] Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set;
[0031] Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix;
[0032] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0033] In a second aspect, a model training method is provided, the method comprising:
[0034] Acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes an image of the verification object, a label of the verification object, and category description information of the verification object, and each object includes each training object and each verification object;
[0035] Extracting features from the image of the training object and the image of the verification object respectively to obtain visual features of the training object and visual features of the verification object;
[0036] Based on the category association information of each object, the visual features of the verification object and the visual features of the training object, updating the visual features of the training object and the visual features of the verification object;
[0037] Model training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels corresponding to the updated visual features of the training object.
[0038] Optionally, before updating the visual features of the training object and the visual features of the verification object based on the category association information of each object, the visual features of the verification object and the visual features of the training object, the method further includes:
[0039] Based on the visual features of the training object, the category description information of the training object and the category description information of the verification object, a new visual feature of the verification object is generated.
[0040] Optionally, the updating of the visual features of the verification object and the visual features of the training object based on the category association information of each object, the visual features of the verification object, and the visual features of the training object includes:
[0041] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0042] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0043] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0044] The model training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object, including:
[0045] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0046] Optionally, the step of inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object includes:
[0047] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0048] For each visual feature to be updated, based on the category association information and sample association information between the visual feature to be updated and other visual features, determine the category association and sample association between the visual feature to be updated and each other visual feature; based on the category association and sample association between the visual feature to be updated and each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0049] Optionally, the performing model training based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object includes:
[0050] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the labels of the updated visual features of the training object, and the labels of the visual features of the verification object.
[0051] Optionally, the performing model training based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object includes:
[0052] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0053] Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set;
[0054] Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0055] Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set;
[0056] Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix;
[0057] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0058] In a third aspect, a model training device is provided, the device comprising:
[0059] an acquisition module, configured to acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes a label of the verification object and category description information of the verification object, and each object includes each training object and each verification object;
[0060] An extraction module, used for performing feature extraction on the image of the training object to obtain visual features of the training object;
[0061] A generating module, configured to generate a visual feature of a verification object based on the visual feature of the training object, the category description information of the training object and the category description information of the verification object;
[0062] an updating module, configured to update the visual features of the verification object and the visual features of the training object based on the category association information of each object, the visual features of the verification object and the visual features of the training object;
[0063] The training module is used to perform model training based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object and the labels of the updated visual features of the training object.
[0064] Optionally, the generating module is used to:
[0065] The visual features of the training object, the category description information of the training object and the category description information of the verification object are input into a generative adversarial network to generate the visual features of the verification object.
[0066] Optionally, the update module is used to:
[0067] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0068] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0069] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0070] The training module is used to:
[0071] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0072] Optionally, the update module is used to:
[0073] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0074] For each visual feature to be updated, based on the category association information and sample association information between the visual feature to be updated and other visual features, determine the category association and sample association between the visual feature to be updated and each other visual feature; based on the category association and sample association between the visual feature to be updated and each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0075] Optionally, the training module is used to:
[0076] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the labels of the updated visual features of the training object, and the labels of the visual features of the verification object.
[0077] Optionally, the training module is used to:
[0078] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0079] Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set;
[0080] Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0081] Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set;
[0082] Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix;
[0083] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0084] In a fourth aspect, a model training device is provided, the device comprising:
[0085] an acquisition module, configured to acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes an image of the verification object, a label of the verification object, and category description information of the verification object, and each object includes each training object and each verification object;
[0086] An extraction module, used to extract features from the image of the training object and the image of the verification object respectively, to obtain visual features of the training object and visual features of the verification object;
[0087] an updating module, configured to update the visual features of the training object and the visual features of the verification object based on the category association information of each object, the visual features of the verification object and the visual features of the training object;
[0088] The training module is used to perform model training based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels corresponding to the updated visual features of the training object.
[0089] Optionally, the device further comprises a generating module, configured to:
[0090] Based on the visual features of the training object, the category description information of the training object and the category description information of the verification object, a new visual feature of the verification object is generated.
[0091] Optionally, the update module is used to:
[0092] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into a first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0093] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0094] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0095] The training module is used to:
[0096] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0097] Optionally, the update module is used to:
[0098] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0099] For each visual feature to be updated, based on the category association information and sample association information between the visual feature to be updated and other visual features, determine the category association and sample association between the visual feature to be updated and each other visual feature; based on the category association and sample association between the visual feature to be updated and each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0100] Optionally, the training module is used to:
[0101] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the labels of the updated visual features of the training object, and the labels of the visual features of the verification object.
[0102] Optionally, the training module is used to:
[0103] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0104] Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set;
[0105] Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0106] Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set;
[0107] Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix;
[0108] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0109] In a fifth aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the model training method as described in the first aspect or the second aspect above.
[0110] In a sixth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the model training method as described in the first aspect or the second aspect above.
[0111] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0112] The technical solution provided in the embodiment of the present application updates the visual features of the training object and the visual features of the verification object based on the category association information of each object, the visual features of the verification object and the visual features of the training object. The updated visual features are more suitable for model training, so that the model completed by subsequent training has better effect in actual work. BRIEF DESCRIPTION OF THE DRAWINGS
[0113] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0114] Figure 1 is a flow chart of a model training method provided in an embodiment of the present application;
[0115] Figure 2 is a flowchart of another model training method provided in an embodiment of the present application;
[0116] Figure 3 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;
[0117] Figure 4It is a structural schematic diagram of another model training device provided in an embodiment of the present application;
[0118] Figure 5 is a structural schematic diagram of a computer device provided in an embodiment of the present application;
[0119] Figure 6 It is a generation framework diagram of an instance-level semantic association graph provided in an embodiment of the present application. DETAILED DESCRIPTION
[0120] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0121] The model training method provided in the embodiment of the present application can be implemented by a computer device, wherein the computer device can be a terminal or a server.
[0122] The model training method provided in the embodiment of the present application, as a branch field of transfer learning, has a wide range of application scenarios. Specifically, it can be used in image classification scenarios, video classification scenarios, and target detection scenarios. Correspondingly, the model training method provided in the embodiment of the present application is used to train an image classification model, a video classification model, and a target detection model, that is, after the training model is completed, it can be used for image classification, video classification, and target detection. The detailed introduction of each scenario can be described as follows.
[0123] Image classification scenario: As one of the most common problems in the industry, image classification is one of the basic perception problems in the visual field, including autonomous driving and medical image analysis. Existing image classification algorithms require a large amount of labeled data, but such conditions are often difficult to achieve in actual application scenarios. Taking medical image analysis as an example, due to issues such as laws and regulations, ethics, and patient privacy, the number of disease-related medical images available for public use is extremely small, which is not enough to drive the training of deep learning models, and existing image classification algorithms are difficult to implement on such data. The model training method provided in the embodiment of the present application can solve such problems. It only needs to provide a very small number of images of the verification object, or even no images of the verification object are required to complete the training of the model, providing a solution to solve intelligent medical care and computer-aided diagnosis.
[0124] Video classification scenario: With the rise of social networks and multimedia, video has gradually become one of the most important data in people's daily lives, and its growth rate is rapid. Faced with massive video data, how to achieve automatic content analysis is a difficult problem to be solved urgently, with broad application prospects and market value. However, compared with other types of data such as images, texts and audio, videos have their own unique characteristics, which pose huge challenges to video content analysis. First, the video data volume is huge, and the training of traditional supervised deep learning on video data takes a long time, requires a large number of server clusters, and annotating video data will consume huge manpower and material resources; second, the video data itself has a variety of content types, but in the sub-field of video action recognition, new categories continue to appear, and traditional supervised learning algorithms cannot be adaptively extended to newly emerging categories; third, the long-tail distribution of video data is obvious, and traditional supervised learning algorithms are extremely prone to overfitting in video classes with small sample sizes, which limits the generalization performance of the model. In view of the above problems, the model training method provided in the embodiment of the present application provides a good solution for video content analysis in actual scenarios, which is conducive to the implementation of video content analysis algorithms based on deep learning in the industry.
[0125] Target detection scenario: As a sub-problem in the underlying perception field of the visual system, the application prospects of target detection are also very broad. In the target detection task, a significant problem is the imbalance of samples, which is particularly prominent for target detection tasks in complex backgrounds. The generalization of traditional supervised algorithms decreases in practical application scenarios such as occlusion, light changes, and object deformation, which limits their application in real environments. The method provided in the embodiment of the present application can improve the detection of targets by deep learning models in complex environments by virtue of its strong generalization ability. In addition, the method provided in the embodiment of the present application requires far fewer training samples than supervised learning, which can significantly improve the training efficiency of the model and improve the scalability of the model, providing a solution for real-time target detection.
[0126] like Figure 1 As shown, the embodiment of the present application provides a model training method, which is a zero-sample learning algorithm. The processing flow of the model training method may include the following steps:
[0127] In step 101, a training object dataset, a verification object dataset, and category association information of each object are obtained.
[0128] The training object data set includes images of the training objects, labels of the training objects and category description information of the training objects, the verification object data set includes labels of the verification objects and category description information of the verification objects, and each object includes each training object and each verification object.
[0129] The training object refers to an object specifically used for training, and the verification object refers to an object that can be recognized by the model after the training of the model is completed. The category association information of each object refers to the category association information between each verification object, between each training object, and between the verification object and the training object. Specifically, it can refer to the similarity between each object. Depending on the trained model, the training object data set and the verification object data set are also different. When the model is an image classification model, the images in the training object data set are pictures, and the label is the category of each picture. When the model is an object detection model, the images in the training object data set are pictures, and the label is the category of the object in each image. When the model is a video classification model, the images in the training object data set are videos, and the label is the category of each video.
[0130] For example, if the model to be trained is a model to be trained for identifying zebras, the verification object is a zebra, and the training objects can be ordinary horses, tigers, and pandas. The training object data set includes images of ordinary horses, tigers, and pandas, labels of ordinary horses, tigers, and pandas (labels indicate which image is an image of an ordinary horse, which image is an image of a tiger, and which image is an image of a panda), and category description information of ordinary horses, tigers, and pandas (for example, a description of the body shape of an ordinary horse, a description of a tiger having stripes, a description of the color of a panda being black and white, etc.). The verification object data set includes the label of a zebra and a category description of a zebra (for example, a description of the color of a zebra being black and white, having stripes and body shape characteristics, etc.). The category association information of each object refers to the category association information between ordinary horses, tigers, and pandas, as well as the category association information of ordinary horses, tigers, and pandas with zebras, respectively. The category association information can include the similarity of each category, for example, the similarity of an ordinary horse with a zebra.
[0131] During implementation, technicians collect and create training object datasets, verification object datasets, and category association information for each object, and input them into the model to be trained.
[0132] For example, the technician collects images of ordinary horses, tigers, and pandas, adds labels to each image, and determines the category description information of ordinary horses, tigers, and pandas, that is, the training data set is obtained. The technician determines the category description information and labels of zebras, that is, the verification data set is obtained. Finally, the technician determines the category association information of each object and inputs it into the model to be trained together with the training object data set and the verification object data set. The computer device can then obtain the training object data set, the verification object data set, and the category association information of each object.
[0133] In step 102, feature extraction is performed on the image of the training object to obtain visual features of the training object.
[0134] The visual feature of the training object is a feature map of the image of the training object, which is essentially a feature matrix used to represent the image of the training object.
[0135] During implementation, the model to be trained performs feature extraction on the image of the training object to obtain the visual features of the training object.
[0136] It should be noted that the algorithm for feature extraction may be an existing algorithm in the art, which will not be described in detail here.
[0137] In step 103, based on the visual features of the training object, the category description information of the training object and the category description information of the verification object, the visual features of the verification object are generated.
[0138] The visual features of the verification object are synthetic visual features, which can be understood as visual features of an image of a virtual verification object.
[0139] In implementation, visual features of the verification object may be generated based on Generative Adversarial Networks (GAN).
[0140] Specifically, the visual features of the training object, the category description information of the training object, and the category description information of the verification object are input into the generative adversarial network, so that the visual features of the verification object can be generated.
[0141] In step 104, based on the category association information of each object, the visual features of the verification object and the visual features of the training object, the visual features of the verification object and the visual features of the training object are updated.
[0142] Among them, the visual features of the verification object and the visual features of the training object are updated, that is, the feature matrices of the training object and the verification object are processed to make them more suitable for model training.
[0143] In implementation, the category association information of each object, the visual features of the verification object and the visual features of the training object are input into the first graph aggregation network, and the visual features of the verification object and the visual features of the training object are initially updated.
[0144] Based on the visual features of the verification object after the initial update, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is updated.
[0145] The updated category association information of each object, the initially updated visual features of the verification object and the visual features of the training object are input into the second graph aggregation network, and the initially updated visual features of the verification object and the visual features of the training object are updated.
[0146] It should be noted that the first graph aggregation network and the second graph aggregation network may be the same or different. The first graph aggregation network may be graphsage, and the second graph aggregation network may be a graph convolutional neural network (GCN).
[0147] Optionally, in order to improve the effect of initial updating of the visual features of the verification object and the visual features of the training object, the first graph aggregation network can be improved based on the multi-head attention mechanism, and the corresponding processing process can be described as follows.
[0148] The principle of the conventional graph aggregation network for initial updating of visual features is:
[0149] For each visual feature to be updated, a plurality of target visual features are randomly selected, and then the visual feature to be updated is initially updated according to the plurality of target visual features and the visual feature to be updated.
[0150] The principle of the graph aggregation network based on the multi-head attention mechanism is:
[0151] Based on the visual features of each verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object.
[0152] For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, the category association degree and sample association degree between the visual feature to be updated and each other visual feature are determined.
[0153] Based on the category association and sample association of the visual feature to be updated with each other visual feature, multiple target visual features are determined. (For example, a visual feature whose category association is greater than a first target threshold and whose sample association is greater than a second target threshold is determined as a target visual feature)
[0154] Based on the visual feature to be updated, each target visual feature, and the sample association degree and category association degree between the visual feature to be updated and each target visual feature, the visual feature to be updated is initially updated.
[0155] It should be noted that each of the above visual features may be a visual feature of a training object or a visual feature of a detection object.
[0156] It can be understood that compared with the conventional graph aggregation network, the graph aggregation network based on the multi-head attention mechanism will not randomly select the target visual features, but determine the target visual features based on the size of the correlation. In addition, the sample correlation and category correlation are taken into account, and the target visual features are weighted using the sample correlation and category correlation, so that the target visual features with large correlation play a greater role in the update process of the visual features to be updated. Then, the visual features to be updated can be initially updated based on multiple weighted target visual features and visual features to be updated.
[0157] In step 105 , training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0158] The category association information of each object is the updated category association information of each object.
[0159] In implementation, the model to be trained can be trained based on the label propagation algorithm. The specific training process can be described as follows:
[0160] According to the updated visual features of the training object, the labels of the updated visual features of the training object, the updated visual features of the verification object and the category association information of each object, based on the relationship propagation algorithm, the labels of the updated visual features of the verification object are predicted.
[0161] The above process is repeated until the predicted labels of the visual features of the verification object converge and the loss value based on the predicted labels and the actual labels is less than the target threshold, and the model training is completed.
[0162] It can be understood that the model trained based on the above training process can be used to identify images of verification objects, and specifically, can be used for target detection, image classification, and video classification.
[0163] Optionally, in order to improve the efficiency and effect of training, a dual relation propagation algorithm can also be used to train the model to be trained. Then the corresponding processing process of step 105 can be described as follows: model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object.
[0164] In implementation, the updated visual features of the verification object and the updated visual features of the training object are divided into two sets, which may be referred to as a training set and a support set.
[0165] Based on the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set, the labels of the visual features in the support set are predicted based on the label propagation algorithm. The predicted labels of the visual features in the support set are combined with the labels of the visual features in the training set to obtain a first label matrix.
[0166] Based on the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set, the labels of the visual features in the training set are predicted based on the label propagation algorithm. The labels of the visual features in the support set and the predicted labels of the visual features in the training set are combined to obtain a second label matrix.
[0167] Repeat the above process. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted labels of the visual features in the support set and the actual labels of the visual features in the support set is less than the target threshold, the model training is completed.
[0168] It should be noted that the updated visual features of the verification object and the updated visual features of the training object are divided into two sets, which can be a random division or a regular division. The training set can include only the visual features of the training object, or it can include both the visual features of the training object and the visual features of the verification object; the verification set can include only the visual features of the verification object, or it can include both the visual features of the verification object and the visual features of the training object. It can be understood that since the first label matrix and the second label matrix are both a combination of the labels of the visual features in the support set and the labels of the visual features in the training set, ideally, the first label matrix and the second label matrix should be the same. The model trained according to the above-mentioned dual relation propagation algorithm can be used to identify images of training objects as well as images of verification objects. Specifically, it can be used for target detection, image classification and video classification.
[0169] like Figure 2 As shown, the present application provides another model training method, which is a small sample learning algorithm. The processing flow of the model training method may include the following steps:
[0170] In step 201, a training object dataset, a verification object dataset, and category association information of each object are obtained.
[0171] The training object data set includes images of training objects, labels of training objects and category description information of training objects; the verification object data set includes images of verification objects, labels of verification objects and category description information of verification objects; and each object includes each training object and each verification object.
[0172] The training object refers to an object specifically used for training, and the verification object refers to an object that can be recognized by the model after the training model is completed. The category association information of each object refers to the category association information between each verification object, between each training object, and between the verification object and the training object. Specifically, it can refer to the similarity between each object.
[0173] For example, if the model to be trained is a model to be trained for identifying zebras, the verification object is a zebra, and the training objects can be ordinary horses, tigers, and pandas. The training object data set includes images of ordinary horses, tigers, and pandas, labels of ordinary horses, tigers, and pandas (labels indicate which image is an image of an ordinary horse, which image is an image of a tiger, and which image is an image of a panda), and category description information of ordinary horses, tigers, and pandas (for example, a description of the body shape of an ordinary horse, a description of a tiger having stripes, and a description of the color of a panda being black and white, etc.). The verification object data set includes images of zebras, labels of zebras (labels indicate which image is an image of a zebra), and category descriptions of zebras (for example, descriptions of the color of zebras being black and white, having stripes and body shape characteristics, etc.). The category association information of each object refers to the category association information between ordinary horses, tigers, and pandas, as well as the category association information of ordinary horses, tigers, and pandas with zebras, respectively. The category association information can include the similarity of each category, for example, the similarity of an ordinary horse with a zebra.
[0174] During implementation, technicians collect and create training object datasets, verification object datasets, and category association information for each object, and input them into the model to be trained.
[0175] For example, the technician collects images of ordinary horses, tigers, and pandas, adds labels to each image, and determines the category description information of ordinary horses, tigers, and pandas, that is, a training data set is obtained. The technician collects images of zebras, adds labels to the images of zebras, and determines the category description information of zebras, that is, a verification data set is obtained. Finally, the technician determines the category association information of each object and inputs it into the model to be trained together with the training object data set and the verification object data set. Then the computer device can obtain the training object data set, the verification object data set, and the category association information of each object.
[0176] In step 202, feature extraction is performed on the image of the training object and the image of the verification object respectively to obtain visual features of the training object and visual features of the verification object.
[0177] The visual feature of the training object is a feature map of the image of the training object, which is essentially a feature matrix used to represent the image of the training object. The visual feature of the verification object is a feature map of the image of the verification object, which is essentially a feature matrix used to represent the image of the verification object.
[0178] During implementation, the model to be trained extracts features from the image of the training object and the image of the verification object respectively to obtain the visual features of the training object and the visual features of the verification object.
[0179] It should be noted that the algorithm for feature extraction may be an existing algorithm in the art, which will not be described in detail here.
[0180] Optionally, new visual features of the verification object can also be generated to expand the samples in the verification data set. The corresponding processing process can be described as follows: based on the visual features of the training object, the category description information of the training object and the category description information of the verification object, new visual features of the verification object are generated.
[0181] The visual features of the new verification object are synthesized visual features, which can be understood as visual features of the image of the virtual verification object.
[0182] In implementation, visual features of the verification object may be generated based on Generative Adversarial Networks (GAN).
[0183] Specifically, the visual features of the training object, the category description information of the training object, and the category description information of the verification object are input into the generative adversarial network, so that the visual features of the verification object can be generated.
[0184] In step 203, based on the category association information of each object, the visual features of the verification object and the visual features of the training object, the visual features of the verification object and the visual features of the training object are updated.
[0185] Among them, the visual features of the verification object and the visual features of the training object are updated, that is, the feature matrices of the training object and the verification object are processed to make them more suitable for model training.
[0186] In implementation, the category association information of each object, the visual features of the verification object and the visual features of the training object are input into the first graph aggregation network, and the visual features of the verification object and the visual features of the training object are initially updated.
[0187] Based on the visual features of the verification object after the initial update, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is updated.
[0188] The updated category association information of each object, the initially updated visual features of the verification object and the visual features of the training object are input into the second graph aggregation network, and the initially updated visual features of the verification object and the visual features of the training object are updated.
[0189] It should be noted that the first graph aggregation network and the second graph aggregation network may be the same or different. The first graph aggregation network may be graphsage, and the second graph aggregation network may be a graph convolutional neural network (GCN).
[0190] Optionally, in order to improve the effect of initial updating of the visual features of the verification object and the visual features of the training object, the first graph aggregation network can be improved based on the multi-head attention mechanism, and the corresponding processing process can be described as follows.
[0191] The principle of the conventional graph aggregation network for initial updating of visual features is:
[0192] For each visual feature to be updated, a plurality of target visual features are randomly selected, and then the visual feature to be updated is initially updated according to the plurality of target visual features and the visual feature to be updated.
[0193] The principle of the graph aggregation network based on the multi-head attention mechanism is:
[0194] Based on the visual features of each verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object.
[0195] For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, the category association degree and sample association degree between the visual feature to be updated and each other visual feature are determined.
[0196] Based on the category association and sample association of the visual feature to be updated with each other visual feature, multiple target visual features are determined. (For example, a visual feature whose category association is greater than a first target threshold and whose sample association is greater than a second target threshold is determined as a target visual feature)
[0197] Based on the visual feature to be updated, each target visual feature, and the sample association degree and category association degree between the visual feature to be updated and each target visual feature, the visual feature to be updated is initially updated.
[0198] It should be noted that each of the above visual features may be a visual feature of a training object or a visual feature of a detection object.
[0199] It can be understood that compared with the conventional graph aggregation network, the graph aggregation network based on the multi-head attention mechanism will not randomly select the target visual features, but determine the target visual features based on the size of the correlation. In addition, the sample correlation and category correlation are taken into account, and the target visual features are weighted using the sample correlation and category correlation, so that the target visual features with large correlation play a greater role in the update process of the visual features to be updated. Then, the visual features to be updated can be initially updated based on multiple weighted target visual features and visual features to be updated.
[0200] In step 204, model training is performed based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the updated labels of the training object.
[0201] The category association information of each object is the updated category association information of each object.
[0202] In implementation, the model to be trained can be trained based on the label propagation algorithm. The specific training process can be described as follows:
[0203] According to the updated visual features of the training object, the labels of the updated visual features of the training object, the updated visual features of the verification object and the category association information of each object, based on the relationship propagation algorithm, the labels of the updated visual features of the verification object are predicted.
[0204] The above process is repeated until the predicted labels of the visual features of the verification object converge and the loss value based on the predicted labels and the actual labels is less than the target threshold, and the model training is completed.
[0205] It can be understood that the model trained based on the above training process can be used to identify images of verification objects, and specifically, can be used for target detection, image classification, and video classification.
[0206] Optionally, in order to improve the efficiency and effect of training, a dual relation propagation algorithm can also be used to train the model to be trained. Then the corresponding processing process of step 105 can be described as follows: model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object.
[0207] In implementation, the updated visual features of the verification object and the updated visual features of the training object are divided into two sets, which may be referred to as a training set and a support set.
[0208] Based on the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set, the labels of the visual features in the support set are predicted based on the label propagation algorithm. The predicted labels of the visual features in the support set are combined with the labels of the visual features in the training set to obtain a first label matrix.
[0209] Based on the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set, the labels of the visual features in the training set are predicted based on the label propagation algorithm. The labels of the visual features in the support set and the predicted labels of the visual features in the training set are combined to obtain a second label matrix.
[0210] Repeat the above process. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted labels of the visual features in the support set and the actual labels of the visual features in the support set is less than the target threshold, the model training is completed.
[0211] It should be noted that the updated visual features of the verification object and the updated visual features of the training object are divided into two sets, which can be a random division or a regular division. The training set can include only the visual features of the training object, or it can include both the visual features of the training object and the visual features of the verification object; the verification set can include only the visual features of the verification object, or it can include both the visual features of the verification object and the visual features of the training object. It can be understood that since the first label matrix and the second label matrix are both a combination of the labels of the visual features in the support set and the labels of the visual features in the training set, ideally, the first label matrix and the second label matrix should be the same. The model trained according to the above-mentioned dual relation propagation algorithm can be used to identify images of training objects as well as images of verification objects. Specifically, it can be used for target detection, image classification and video classification.
[0212] It can be seen that the second model training method mentioned above is different from the first model training method in that its verification object data set contains the verification object image. Therefore, in some cases, it is not necessary to generate the visual features of the verification object. That is, the first model training method is a zero-sample model training method, and the second model training method is a small-sample model training method. Except for the above differences, the other technical features are similar.
[0213] The training methods of the above two models can be divided into two stages. The first stage is the graph generation stage, which is used to generate the instance-level semantic association graph. The second stage is the relationship propagation stage, which is used to train the model based on the instance-level semantic association graph and the relationship propagation algorithm. Among them, the instance-level semantic association graph contains the updated category association information of each object, the updated visual features of the verification object, and the updated visual features of the training object, which is essentially a feature matrix. The specific process can be described as follows:
[0214] Graph generation phase:
[0215] First, a semantic association graph at the category level is established. The semantic association graph at the category level contains the category association information of each object and the category representation of each category. Its essence is a feature matrix, which can be referred to Figure 6 , where each node represents a category, the direction of the edge represents the specific meaning of the association relationship between different semantic categories, the length of the edge represents the strength of the association relationship, and if there is no edge connecting the nodes, it means that the association relationship is 0.
[0216] Then, after completing the modeling of the category-level semantic association graph, the category association information contained in the category-level semantic association graph can be utilized. Taking the semantic association information contained in the category to which the visual feature belongs as the starting point, the visual features represented by different nodes are initially updated using the first graph aggregation network (i.e., the category association information of the category to which each visual feature belongs is used to replace the association information of each visual feature). Each node in the category-level semantic association graph after the initial update represents a visual feature. Before the initial update of the visual features, in order to avoid the problem of domain drift caused by insufficient or missing samples, a generative adversarial network can be used to expand the samples in the verification object data set, that is, to train a conditional generative adversarial network, using word vectors or attribute features as auxiliary conditional information, and using adversarial learning to fit the joint distribution of the visual and semantic features of the trained object to generate the visual features of the verification object. These visual features can effectively alleviate the problem of missing and unbalanced samples of the verification object in the subsequent graph operation stage.
[0217] Finally, if Figure 6As shown in the figure, after the initial update of the visual features, in the representation space composed of the nodes adjusted by the graph neural network, the Gaussian similarity function is used to generate the edges between different nodes, and then the graph convolution network (GCNs) is used to perform a convolution operation on the generated graph, so that the visual features after the initial update are updated again to obtain the final instance-level semantic association graph. In the instance-level semantic association graph, each node represents an updated visual feature, the direction of the edge represents the specific meaning of the association relationship between different semantic categories, and the length of the edge represents the strength of the association relationship. The instance-level semantic association graph embeds category-level semantic association and sample-level visual association information for knowledge transfer in the subsequent relationship propagation stage. In order to prevent the instance-level semantic association graph from being significantly different from the category-level semantic association graph, regularization constraints can also be added.
[0218] The graph generation stage corresponds to steps 101 to 104 in the first model training method, and corresponds to steps 201 to 203 in the second model training method.
[0219] Relation propagation stage: The instance-level semantic association graph generated in the previous stage is fully utilized by the dual relation propagation algorithm based on meta-learning. In the relation propagation stage, the nodes of the instance-level semantic association graph (that is, the updated visual features) are randomly divided into a support set and a training set. The divided support set and training set are used as labeled samples, and the label propagation algorithm is used from them. Constraints are imposed so that the two relation propagation processes form a dual learning relationship, so that the label matrices obtained after label propagation tend to be consistent. The dual learning constraint can make the two relation propagation processes guide each other, and the side with more samples guides the side with less samples, which plays a role in domain adaptation in the final classification stage, making up for the deficiency of existing methods that only perform domain adaptation in the feature extraction stage, and greatly alleviates the problem of algorithm performance skewness caused by data imbalance.
[0220] The relationship propagation stage corresponds to step 105 in the first model training method and corresponds to step 204 in the second model training method.
[0221] The two model training methods provided in the embodiments of the present application update the visual features of the training object and the visual features of the verification object based on the category association information of each object, the visual features of the verification object and the visual features of the training object. The updated visual features are more suitable for model training, so that the model completed by subsequent training has better effect when performing actual work. And the semantic associations at the category level and the semantic associations at the instance level are explicitly modeled, so that the model trained by the model training method provided in the embodiments of the present application can meet the requirements of industrial use.
[0222] Some algorithm formulas used in the embodiments of the present application are introduced below.
[0223] The algorithm formula of the first graph aggregation network can be:
[0224]
[0225]
[0226] Among them, AGGREGATE k represents the aggregation function within the range of k-order neighbors, N(v) represents the set of neighbor nodes of node v, and CONCAT represents the feature concatenation operation.
[0227]
[0228]
[0229]
[0230]
[0231] First, the original input visual features are linearly transformed to obtain Then use the neighbor information and the trainable attention vector The attention weight is obtained and normalized. Finally, the calculated attention score is used in the neighborhood information aggregation to obtain the updated node features. a∈{att C ,att I} represent the class-level and instance-level attention weight vectors respectively.
[0232] The Gaussian similarity determination formula is:
[0233]
[0234] in, represents the edge between the generated midpoints v and u, Φ Θ It is a neural network with Θ as parameter, and δ represents the bandwidth hyperparameter.
[0235] The formula for the second graph aggregation network can be:
[0236]
[0237] The formula of the label propagation algorithm can be:
[0238] Y * =(I-μY L ) -1 Y
[0239] Where I is the identity matrix, Y L is the labeled submatrix of Y and μ is a hyperparameter that controls the amount of information propagation.
[0240] The formula of the relational algorithm for dual relation propagation can be:
[0241]
[0242] Among them, Y S and Y L The labeled sub-matrices in the training set and support set respectively, Represents the Frobenius norm of the matrix.
[0243] In order to prevent the instance-level semantic association graph from being significantly different from the category-level semantic association graph, a regularization constraint can also be added. The formula for the regularization constraint can be:
[0244]
[0245] Among them, GraphKernel(·) is an existing graph kernel algorithm, and graph2vec is used in this application. is the semantic association graph at the category level, It is the instance-level semantic association graph. Represents the error between the category-level semantic association graph and the instance-level semantic association graph.
[0246] During the overall training process of the model, you can also set the total loss function to control the total error. The corresponding formula is:
[0247]
[0248] in, Represents the error between the category-level semantic association graph and the instance-level semantic association graph. d Represents the error between the first label matrix and the second label matrix. Loss c Represents the loss value of the predicted label of the visual feature in the support set and the actual label of the visual feature in the support set. λ1 and λ2 are constants.
[0249] Based on the same technical concept as the first model training method, the embodiment of the present application also provides a model training device, which can be a computer device, such as Figure 3 As shown, the device comprises:
[0250] An acquisition module 301 is used to acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes a label of the verification object and category description information of the verification object, and each object includes each training object and each verification object;
[0251] An extraction module 302 is used to extract features from the image of the training object to obtain visual features of the training object;
[0252] A generating module 303, configured to generate a visual feature of a verification object based on the visual feature of the training object, the category description information of the training object, and the category description information of the verification object;
[0253] An updating module 304, configured to update the visual features of the verification object and the visual features of the training object based on the category association information of each object, the visual features of the verification object and the visual features of the training object;
[0254] The training module 305 is used to perform model training based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels of the updated visual features of the training object.
[0255] Optionally, the generating module 303 is used to:
[0256] The visual features of the training object, the category description information of the training object and the category description information of the verification object are input into the generative adversarial network to generate the visual features of the verification object.
[0257] Optionally, the update module 304 is used to:
[0258] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into the first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0259] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0260] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0261] The training module 305 is used to:
[0262] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the updated labels of the visual features of the training object.
[0263] Optionally, the update module 304 is used to:
[0264] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0265] For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determine the category association and sample association of the visual feature to be updated with each other visual feature; based on the category association and sample association of the visual feature to be updated with each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0266] Optionally, the training module 305 is used to:
[0267] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object.
[0268] Optionally, the training module 305 is used to:
[0269] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0270] According to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set, based on the label propagation algorithm, the labels of the visual features in the support set are predicted;
[0271] Combining the labels of the predicted visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0272] According to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set, based on the label propagation algorithm, the labels of the visual features in the training set are predicted;
[0273] Combining the labels of the visual features in the support set with the labels of the predicted visual features in the training set to obtain a second label matrix;
[0274] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0275] Based on the same technical concept as the second model training method, the present application embodiment also provides another model training device, which can be a computer device, such as Figure 4 As shown, the device comprises:
[0276] An acquisition module 401 is used to acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the verification object data set includes an image of the verification object, a label of the verification object, and category description information of the verification object, and each object includes each training object and each verification object;
[0277] An extraction module 402 is used to extract features from the image of the training object and the image of the verification object to obtain visual features of the training object and visual features of the verification object;
[0278] An updating module 403, configured to update the visual features of the training object and the visual features of the verification object based on the category association information of each object, the visual features of the verification object, and the visual features of the training object;
[0279] The training module 404 is used to perform model training based on the category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the labels corresponding to the updated visual features of the training object.
[0280] Optionally, the device further includes a generating module, configured to:
[0281] Based on the visual features of the training object, the category description information of the training object and the category description information of the verification object, a new visual feature of the verification object is generated.
[0282] Optionally, the update module 403 is used to:
[0283] Inputting the category association information of each object, the visual features of the verification object, and the visual features of the training object into the first graph aggregation network, and initially updating the visual features of the verification object and the visual features of the training object;
[0284] Based on the visual features of the verification object after the initial update, the visual features of the training object, and the Gaussian similarity determination formula, the category association information of each object is updated;
[0285] Inputting the updated category association information of each object, the visual features of the verification object after initial update, and the visual features of the training object into the second graph aggregation network, and updating the visual features of the verification object after initial update and the visual features of the training object;
[0286] The training module 404 is used to:
[0287] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, and the updated labels of the visual features of the training object.
[0288] Optionally, the update module 403 is used to:
[0289] Based on the visual features of the verification object, the visual features of the training object and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain the sample association information of the visual features of each object;
[0290] For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determine the category association and sample association of the visual feature to be updated with each other visual feature; based on the category association and sample association of the visual feature to be updated with each other visual feature, determine multiple target visual features; based on the visual feature to be updated, each target visual feature, and the sample association and category association between the visual feature to be updated and each target visual feature, perform an initial update on the visual feature to be updated.
[0291] Optionally, the training module 404 is used to:
[0292] Model training is performed based on the updated category association information of each object, the updated visual features of the verification object, the updated visual features of the training object, the updated labels of the visual features of the training object, and the labels of the visual features of the verification object.
[0293] Optionally, the training module 404 is used to:
[0294] Dividing the updated visual features of the verification object and the updated visual features of the training object into a training set and a support set;
[0295] According to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set, based on the label propagation algorithm, the labels of the visual features in the support set are predicted;
[0296] Combining the labels of the predicted visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix;
[0297] According to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set, based on the label propagation algorithm, the labels of the visual features in the training set are predicted;
[0298] Combining the labels of the visual features in the support set with the labels of the predicted visual features in the training set to obtain a second label matrix;
[0299] The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
[0300] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0301] It should be noted that: the model training device provided in the above embodiment only uses the division of the above functional modules as an example when performing model training. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model training device provided in the above embodiment and the model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0302] Figure 5 5 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 501 and one or more memories 502, wherein the memory 502 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 501 to implement the above-mentioned model training method. The computer device 500 may be a server or a terminal.
[0303] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the model training method in the above embodiment. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a random access memory (Random Access Memory, RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
[0304] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0305] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A model training method, characterized in that: The model is a model for identifying zebras, and the method comprises: Acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the training objects are ordinary horses, tigers, and pandas, and the category description information of the training objects includes a description of the body shape of ordinary horses, a description of tigers having stripes, and a description of pandas being black and white in color; the verification object data set includes a label of the verification object and category description information of the verification object, the verification object is a zebra, and the category description information of the verification object includes a description of zebras being black and white in color, zebras having stripes, and body shape characteristics of zebras; the category association information of each object includes category association information between ordinary horses, tigers, and pandas, and category association information between ordinary horses, tigers, and pandas and zebras, respectively, and the category association information includes similarity; Perform feature extraction on images of common horses, tigers and pandas to obtain visual features of common horses, tigers and pandas; Input the visual features of an ordinary horse, a tiger and a panda, a description of the body shape of an ordinary horse, a description of the tiger having stripes, a description of the panda being black and white, and a description of the zebra being black and white, the zebra having stripes and the body shape feature of the zebra into a generative adversarial network to generate the visual features of the zebra; Based on the visual features of zebras, common horses, tigers and pandas, and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain sample association information of the visual features of each object; For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determine the category association degree and sample association degree of the visual feature to be updated with each other visual feature; determine the visual feature whose category association degree is greater than a first target threshold and whose sample association degree is greater than a second target threshold as the target visual feature; perform an initial update on the visual feature to be updated based on the visual feature to be updated, each target visual feature, and the sample association degree and category association degree of the visual feature to be updated with each target visual feature; Based on the visual features of the zebra after initial update, the visual features of the common horse, tiger and panda after initial update and the Gaussian similarity determination formula, the category association information of each object is updated; Inputting the updated category association information of each object, the initially updated visual features of the zebra, and the initially updated visual features of the common horse, tiger, and panda into a second graph aggregation network, and updating the initially updated visual features of the zebra and the initially updated visual features of the common horse, tiger, and panda, wherein the second graph aggregation network is a graph convolutional neural network; Dividing the updated visual features of zebras and the updated visual features of common horses, tigers and pandas into a training set and a support set, wherein the training set includes the visual features of common horses, tigers and pandas as well as the visual features of zebras, and the support set also includes the visual features of common horses, tigers and pandas as well as the visual features of zebras; Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set; Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix; Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set; Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix; The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
2. A model training method, characterized in that: The model is a model for identifying zebras, and the method comprises: Acquire a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the training objects are ordinary horses, tigers, and pandas, and the category description information of the training objects includes a description of the body shape of ordinary horses, a description of tigers having stripes, and a description of pandas being black and white in color; the verification object data set includes an image of the verification object, a label of the verification object, and category description information of the verification object, the verification object is a zebra, and the category description information of the verification object includes a description of zebras being black and white in color, zebras having stripes, and body feature descriptions of zebras; the category association information of each object includes category association information between ordinary horses, tigers, and pandas, and similarities between ordinary horses, tigers, and pandas and zebras, respectively; Feature extraction is performed on the images of ordinary horses, tigers, pandas and zebras respectively to obtain visual features of ordinary horses, tigers, pandas and zebras; Generate new visual features of zebra based on the visual features of common horses, tigers and pandas, the description of the body shape of common horses, the description that tigers have stripes, the description that pandas are black and white, and the description that zebras are black and white, zebras have stripes and the body shape features of zebras; Based on the visual features of zebras, common horses, tigers and pandas, and the Gaussian similarity determination formula, the category association information of each object is initially updated to obtain sample association information of the visual features of each object; For each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determine the category association degree and sample association degree of the visual feature to be updated with each other visual feature; determine the visual feature whose category association degree is greater than a first target threshold and whose sample association degree is greater than a second target threshold as the target visual feature; perform an initial update on the visual feature to be updated based on the visual feature to be updated, each target visual feature, and the sample association degree and category association degree of the visual feature to be updated with each target visual feature; Based on the visual features of the zebra after initial update, the visual features of the common horse, tiger and panda after initial update and the Gaussian similarity determination formula, the category association information of each object is updated; Inputting the updated category association information of each object, the initially updated visual features of the zebra, and the initially updated visual features of the common horse, tiger, and panda into a second graph aggregation network, and updating the initially updated visual features of the zebra and the initially updated visual features of the common horse, tiger, and panda, wherein the second graph aggregation network is a graph convolutional neural network; Dividing the updated visual features of zebras and the updated visual features of common horses, tigers and pandas into a training set and a support set, wherein the training set includes the visual features of common horses, tigers and pandas as well as the visual features of zebras, and the support set also includes the visual features of common horses, tigers and pandas as well as the visual features of zebras; Predicting the labels of the visual features in the support set based on a label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object, and the visual features in the support set; Combining the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix; Predicting the labels of the visual features in the training set based on a label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object, and the visual features in the training set; Combining the labels of the visual features in the support set with the labels of the visual features in the predicted training set to obtain a second label matrix; The model is adjusted based on the first label matrix and the second label matrix. When the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted label of the visual features in the support set and the actual label of the visual features in the support set is less than the target threshold, the model training is completed.
3. A model training device, characterized in that: The model is a model for identifying zebras, and the device comprises: an acquisition module, for acquiring a training object data set, a verification object data set, and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object, and category description information of the training object, the training objects are ordinary horses, tigers, and pandas, and the category description information of the training objects includes a description of the body shape of ordinary horses, a description that tigers have stripes, and a description that pandas are black and white in color; the verification object data set includes a label of the verification object and category description information of the verification object, the verification object is a zebra, and the category description information of the verification object includes a description that the zebra is black and white in color, that the zebra has stripes, and a description of the body shape characteristics of the zebra; the category association information of each object includes category association information between ordinary horses, tigers, and pandas, and category association information between ordinary horses, tigers, and pandas and zebras, respectively, and the category association information includes similarity; An extraction module is used to extract features from the images of ordinary horses, tigers and pandas to obtain visual features of ordinary horses, tigers and pandas; A generation module, for inputting the visual features of an ordinary horse, a tiger and a panda, a description of the body shape of an ordinary horse, a description that a tiger has stripes, a description that the color of a panda is black and white, and a description that a zebra is black and white, a zebra has stripes and a body shape feature of a zebra into a generative adversarial network to generate a visual feature of a zebra; The updating module initially updates the category association information of each object based on the visual features of the zebra and the visual features of the common horse, the tiger and the panda, and the Gaussian similarity determination formula, and obtains the sample association information of the visual features of each object; for each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determines the category association degree and sample association degree of the visual feature to be updated with each other visual feature; determines the visual feature whose category association degree is greater than the first target threshold and whose sample association degree is greater than the second target threshold as the target visual feature; based on the visual feature to be updated, each target visual feature and the visual feature to be updated The sample association degree and category association degree of the feature and each target visual feature are used to initially update the visual features to be updated; based on the visual features of the zebra after the initial update, the visual features of the common horse, tiger and panda after the initial update and the Gaussian similarity determination formula, the category association information of each object is updated; the updated category association information of each object, the visual features of the zebra after the initial update, and the visual features of the common horse, tiger and panda after the initial update are input into the second graph aggregation network, and the visual features of the zebra after the initial update and the visual features of the common horse, tiger and panda after the initial update are updated, wherein the second graph aggregation network is a graph convolutional neural network; The training module is used to divide the updated visual features of zebras and the updated visual features of ordinary horses, tigers and pandas into a training set and a support set, wherein the training set includes the visual features of ordinary horses, tigers and pandas as well as the visual features of zebras, and the support set also includes the visual features of ordinary horses, tigers and pandas as well as the visual features of zebras; predict the labels of the visual features in the support set based on the label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set; combine the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix; predict the labels of the visual features in the training set based on the label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set; combine the labels of the visual features in the support set with the predicted labels of the visual features in the training set to obtain a second label matrix; adjust the model based on the first label matrix and the second label matrix, and when the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted labels of the visual features in the support set and the actual labels of the visual features in the support set is less than the target threshold, the model training is completed.
4. A model training device, characterized in that: The model is a model for identifying zebras, and the device comprises: An acquisition module is used to acquire a training object data set, a verification object data set and category association information of each object, wherein the training object data set includes an image of the training object, a label of the training object and category description information of the training object, the training objects are ordinary horses, tigers and pandas, and the category description information of the training objects includes a description of the body shape of ordinary horses, a description of tigers having stripes and a description of the panda's color being black and white; the verification object data set includes an image of the verification object, a label of the verification object and category description information of the verification object, the verification object is a zebra, and the category description information of the verification object includes a description of the zebra's color being black and white, the zebra having stripes and the zebra's body shape features; the category association information of each object includes category association information between ordinary horses, tigers and pandas, and similarities between ordinary horses, tigers and pandas and zebras respectively; An extraction module is used to extract features from the images of ordinary horses, tigers and pandas, and the images of zebras, respectively, to obtain visual features of ordinary horses, tigers and pandas, and visual features of zebras; A generation module, for generating new visual features of a zebra based on the visual features of an ordinary horse, a tiger and a panda, a description of the body shape of an ordinary horse, a description that a tiger has stripes, a description that the panda is black and white in color, and a description that the zebra is black and white in color, that the zebra has stripes and a description of the body shape of the zebra; The updating module is used to initially update the category association information of each object based on the visual features of the zebra, the visual features of the common horse, the tiger and the panda and the Gaussian similarity determination formula to obtain the sample association information of the visual features of each object; for each visual feature to be updated, based on the category association information and sample association information of the visual feature to be updated and other visual features, determine the category association degree and sample association degree of the visual feature to be updated with each other visual feature; determine the visual feature whose category association degree is greater than the first target threshold and whose sample association degree is greater than the second target threshold as the target visual feature; based on the visual feature to be updated, each target visual feature and the visual feature to be updated The sample association degree and category association degree of the feature and each target visual feature are used to initially update the visual features to be updated; based on the visual features of the zebra after the initial update, the visual features of the common horse, tiger and panda after the initial update and the Gaussian similarity determination formula, the category association information of each object is updated; the updated category association information of each object, the visual features of the zebra after the initial update, and the visual features of the common horse, tiger and panda after the initial update are input into the second graph aggregation network, and the visual features of the zebra after the initial update and the visual features of the common horse, tiger and panda after the initial update are updated, wherein the second graph aggregation network is a graph convolutional neural network; The training module is used to divide the updated visual features of zebras and the updated visual features of ordinary horses, tigers and pandas into a training set and a support set, wherein the training set includes the visual features of ordinary horses, tigers and pandas as well as the visual features of zebras, and the support set also includes the visual features of ordinary horses, tigers and pandas as well as the visual features of zebras; predict the labels of the visual features in the support set based on the label propagation algorithm according to the visual features in the training set, the labels of the visual features in the training set, the updated category association information of each object and the visual features in the support set; combine the predicted labels of the visual features in the support set with the labels of the visual features in the training set to obtain a first label matrix; predict the labels of the visual features in the training set based on the label propagation algorithm according to the visual features in the support set, the labels of the visual features in the support set, the updated category association information of each object and the visual features in the training set; combine the labels of the visual features in the support set with the predicted labels of the visual features in the training set to obtain a second label matrix; adjust the model based on the first label matrix and the second label matrix, and when the first label matrix and the second label matrix converge respectively, and the loss value obtained based on the predicted labels of the visual features in the support set and the actual labels of the visual features in the support set is less than the target threshold, the model training is completed.
