Introducing meta-learning model training methods, systems, and relationship category identification methods

Abstract

This application discloses a model training method incorporating meta-learning, comprising: in the first stage of model training, training a relation detection model based on the data of the first stage training relation categories, enabling the relation detection model to learn to predict the object category and the bounding box coordinates of the object, as well as the human behavior category and the bounding box coordinates of the human, to obtain the first-stage relation detection model; the relation detection model includes: an object detection branch and a relation category detection branch; in the second stage of model training, continuing to train the first-stage relation detection model based on the data of the second stage training relation categories, to obtain the final relation detection model; the data of the second stage training relation categories is small sample data, the number of relation categories corresponding to the data of the first stage training relation categories should be greater than the number of relation categories corresponding to the data of the second stage training relation categories, and the sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the model can predict.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training method, apparatus, electronic device, and storage device that incorporates meta-learning. This application also relates to a model training system that incorporates meta-learning, and a method for identifying relation categories. Background Technology

[0002] Currently, the scope of municipal scenarios is becoming increasingly broad, and many scenarios can no longer be addressed by simply identifying targets and adding rules. Examples include getting on and off vehicles in illegal areas, walking dogs outside of designated times, and littering. These scenarios cannot be solved using simple target identification (vehicles, pedestrians, dogs, litter, etc.) plus rules. Taking getting on and off vehicles in illegal areas as an example, using target identification plus rules requires the following steps: (1) identifying motor vehicles and people; (2) determining whether the vehicle is temporarily parked; (3) determining whether the distance between the person and the vehicle is close; (4) determining whether the vehicle is in the open / closed state (vehicle classification model). However, even with these steps, it is still impossible to accurately locate and identify whether a person is getting on or off the vehicle. To address the difficulties in the above scenarios, a neural network model is proposed to identify the relationship between people and objects, thereby determining whether a person is getting on or off the vehicle. To address the problems in the above scenarios, a model for identifying the relationship between people and objects is proposed to uniformly determine the relationship between people and vehicles.

[0003] From a technical perspective, the problem of detecting the relationship between people and objects can be divided into two implementation schemes: one is a two-stage approach, which involves first detecting the target and then identifying the relationship between the person and the object; the other is a one-stage approach, where the model can identify the coordinates of the person and the object and their corresponding relationship through a single inference.

[0004] Under current technology, the one-stage method for detecting human-object relationships is very difficult to collect data in real-world scenarios such as people walking dogs or littering outside of designated time periods, resulting in a severe long-tail distribution of data.

[0005] In summary, existing technologies for detecting the relationship between people and objects suffer from a severe problem of long-tailed data distribution. Summary of the Invention

[0006] This application provides a model training method, apparatus, electronic device, storage device, and system that incorporates meta-learning to address the problem of severe long-tailed data distribution in existing schemes for detecting human-object relationships.

[0007] This application provides a model training system that incorporates meta-learning, including: a platform server and a user terminal;

[0008] The platform server is used to obtain the first-stage training relation category data and the second-stage training relation category data sent by the user terminal; in the first stage of model training, based on the first-stage training relation category data, a relation detection model is trained, enabling the relation detection model to learn to predict the categories of objects and the categories of human behavior, thus obtaining the first-stage relation detection model; in the second stage of model training, based on the second-stage training relation category data, the first-stage relation detection model is further trained to obtain the final relation detection model; wherein, the second-stage training relation category data is small sample data; and the final relation detection model is sent to the user terminal;

[0009] The user terminal is used to send the data of the first stage training relationship categories and the data of the second stage training relationship categories to the platform server; and to obtain the final relationship detection model sent by the platform server.

[0010] This application provides a method for identifying relationship categories, including:

[0011] Obtain the image of the person relationship category to be determined;

[0012] The image of the person-object relationship category to be determined is input into the final relationship detection model to obtain the relationship category of people and objects in the image of the person-object relationship category to be determined;

[0013] The final relationship detection model is obtained in the following manner:

[0014] In the first stage of model training, the relationship detection model is trained based on the data of the relationship categories trained in the first stage. This enables the relationship detection model to learn to predict the categories of objects and the categories of human behavior, thus obtaining the first-stage relationship detection model.

[0015] In the second stage of model training, the first-stage relationship detection model is trained again based on the data of the second-stage training relationship categories to obtain the final relationship detection model; the data of the second-stage training relationship categories is small sample data.

[0016] This application provides a model training method that incorporates meta-learning, including:

[0017] In the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0018] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0019] Optionally, the step of training a relationship detection model based on the data from the first-stage training of relationship categories, enabling the relationship detection model to learn to predict the category of objects and the bounding box coordinates of objects, as well as the category of human behavior and the bounding box coordinates of humans, to obtain the first-stage relationship detection model, includes:

[0020] The data from the first-stage training of relation categories is input into the base network of the relation detection model to obtain the feature data of the first-stage image.

[0021] The feature data of the first stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the first stage are output.

[0022] Based on the feature data of objects and people in the first stage, the object detection branch and the relationship category detection branch are trained respectively, so that the relationship detection model learns to predict the category of objects and the bounding box coordinates of objects, as well as the behavior category of people and the bounding box coordinates of people, thus obtaining the first-stage relationship detection model.

[0023] Optionally, the step of continuing to train the first-stage relation detection model based on the data from the second-stage training of relation categories to obtain the final relation detection model includes:

[0024] The second set of training data for relation categories is input into the base network of the relation detection model, which outputs the feature data of the second-stage image.

[0025] The feature data of the second-stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the second stage are output.

[0026] The data from the second-stage training of relation categories is input into the meta-learning branch of the relation detection model to obtain relation category vectors;

[0027] Based on the relationship category vector and the second-stage human feature data, channel-level operations are performed to obtain the human target feature data.

[0028] Based on the feature data of objects in the second stage and the target feature data of people, the target detection branch and the relationship category detection branch in the first stage are trained respectively to obtain the final relationship detection model.

[0029] Optionally, the step of performing channel-level operations based on the relation category vector and the second-stage human feature data to obtain the human target feature data includes:

[0030] The channel dimension convolution is performed on the relationship category vector and the human feature data of the second stage to obtain the human target feature data.

[0031] Optionally, the object detection branch and the relationship category detection branch in the first stage are trained based on the feature data of the object in the second stage and the target feature data of the person, respectively, to obtain the final relationship detection model, including:

[0032] The target detection branch is trained based on the feature data of the objects in the second stage, so that the first stage relationship detection model can learn to predict the object category and the bounding box coordinates of the object.

[0033] The relationship category detection branch is trained based on the target feature data of people in the second stage and the spatial feature data between people and objects. This enables the first-stage relationship detection model to learn to predict the behavior category of people and the bounding box coordinates of people, thus obtaining the final relationship detection model. Among them, the spatial feature data between people and objects is the data used to characterize the relative position between people and objects in space.

[0034] Optionally, it may also include: obtaining spatial feature data between people and objects.

[0035] Optionally, obtaining the spatial feature data between people and objects includes:

[0036] Based on the human characteristic data from the second stage, obtain the bounding box coordinates of the human;

[0037] Obtain the bounding box coordinates of the object based on the object's feature data from the second stage;

[0038] Based on the bounding box coordinates of people and objects, spatial feature data between people and objects is obtained.

[0039] Optionally, after the step of training the first-stage relation detection model based on the target relation category vector to obtain the final relation detection model, the method further includes:

[0040] Store the relationship category vector for use during the testing phase.

[0041] Optional, also includes:

[0042] Obtain the image of the person relationship category to be determined;

[0043] The image of the person relationship category to be determined is input into the base network of the final relationship detection model, and the feature data of the image of the person relationship category to be determined is output.

[0044] The feature data of the image whose relationship category to be determined is input into the region of interest pooling layer, and the feature data of the object and the feature data of the person are output.

[0045] Based on the stored relation category vectors and human feature data, the final target feature data of the human is obtained;

[0046] Input the object's feature data into the object detection branch to predict the object's category and bounding box coordinates;

[0047] Input the final target feature data of a person into the relation category branch to predict the person's behavior category and the person's bounding box coordinates;

[0048] The final relationship category is determined based on the set of object category scores and human behavior category scores.

[0049] Optionally, determining the final relationship category based on the set of object category scores and human behavior category scores includes:

[0050] For the set of category scores for the objects and category scores for the human behavior, calculate the product of each object category score and each behavior score, and the one with the highest score is the final relationship category.

[0051] This application also provides a model training apparatus that incorporates meta-learning, comprising:

[0052] The first-stage training unit is used to train the relationship detection model in the first stage of model training based on the relationship category data from the first-stage training. This enables the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0053] The second-stage training unit is used in the second stage of model training to continue training the first-stage relationship detection model based on the data of the second-stage training relationship categories, thus obtaining the final relationship detection model. The data of the second-stage training relationship categories is small sample data, and the number of relationship categories corresponding to the data of the first-stage training relationship categories should be greater than the number of relationship categories corresponding to the data of the second-stage training relationship categories. The sum of the number of relationship categories corresponding to the data of the two stages is the number of relationship categories that the relationship detection model can predict.

[0054] This application also provides an electronic device, including:

[0055] Processor; and

[0056] The memory stores the program for the model training method incorporating meta-learning. After the device is powered on and the program for the model training method incorporating meta-learning is run by the processor, the following steps are performed:

[0057] In the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0058] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0059] This application also provides a storage device storing a program for a model training method that incorporates meta-learning, which is executed by a processor to perform the following steps:

[0060] In the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0061] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0062] Compared with the prior art, this application has the following advantages:

[0063] This application provides a model training system and method incorporating meta-learning. In the relationship detection model training process, a two-stage training approach using meta-learning is employed. In the first stage, the relationship detection model is trained using data from the first stage's relationship category training, enabling it to learn to predict object categories and human behavior categories. In the second stage, the first stage's relationship detection model is further trained using data from the second stage's relationship category training (small sample data, scenarios such as people walking dogs, littering outside of designated time periods, etc., which are difficult to collect), resulting in the final relationship detection model. Because a large amount of training data was used in the first stage, the relationship detection model learns not only to predict human-object relationships in the first stage's relationship category training data, but also how to determine these relationships. For example, the model learns that features such as human posture and object category are key to determining human-object relationships. The second stage only requires training the relationship detection model with small sample data to obtain a high-performance model, avoiding the problem of severe long-tailed data distribution and significantly improving the model's recognition accuracy for few-sample relationship labels. Attached Figure Description

[0064] Figure 1a This is a scenario illustration provided in this application.

[0065] Figure 1 This is a flowchart of a model training method that incorporates meta-learning, as provided in the first embodiment of this application.

[0066] Figure 2 This is a schematic diagram of the training of a relationship detection model provided in the first embodiment of this application.

[0067] Figure 3 This is a schematic diagram of obtaining spatial feature data provided in the first embodiment of this application.

[0068] Figure 4 This is a schematic diagram of a relationship detection model test provided in the first embodiment of this application.

[0069] Figure 5 This is a schematic diagram of a model training device that incorporates meta-learning, as provided in the second embodiment of this application.

[0070] Figure 6 This is a schematic diagram of a model training system that incorporates meta-learning, as provided in the fifth embodiment of this application. Detailed Implementation

[0071] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0072] To enable those skilled in the art to better understand the solution of this application, a specific application scenario embodiment of this application will be described in detail first.

[0073] The model training method incorporating meta-learning provided in this application can be applied to scenarios involving client-server interaction, such as... Figure 1a When it is necessary to obtain the relationship categories between people and objects in an image to be classified, the client typically establishes a connection with the server first. After the connection is established, the client sends the image to be classified to the server. The server first obtains the image to be classified; then, it inputs the image to be classified into the base network of the final relationship detection model, outputting the feature data of the image to be classified; next, it inputs the feature data of the image to be classified into the region of interest pooling layer, outputting the feature data of objects and people; then, based on the stored relationship category vector and the feature data of people, the final target feature data of people is obtained; then, the feature data of objects is input into the object detection branch to predict the category of objects and the bounding box coordinates of objects; next, the final target feature data of people is input into the relationship category branch to predict the behavior category of people and the bounding box coordinates of people; finally, based on the set of object category scores and behavior category scores, the final relationship category is determined and sent to the client. The final relationship detection model is obtained through the following steps:

[0074] In the first stage of model training, the relationship detection model is trained based on the relationship category data from the first stage training. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, resulting in the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, while the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0075] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0076] The first embodiment of this application provides a model training method that incorporates meta-learning, which will be discussed below. Figure 1 , Figure 2 , Figure 3 , Figure 4 Please provide an explanation.

[0077] like Figure 1 As shown, in step S101, in the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This enables the relationship detection model to learn to predict the category of an object and the bounding box coordinates of the object, as well as the behavior category of a person and the bounding box coordinates of the person, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category of an object and the bounding box coordinates of the object, and the relationship category detection branch is used to predict the behavior category of a person and the bounding box coordinates of the person.

[0078] Meta-learning is a fast learning method, primarily designed for training with small sample sizes.

[0079] The data used for training the relation categories in the first stage is sufficient and easy to collect.

[0080] The category of human behavior refers to the type of human action. For example, riding or throwing.

[0081] Bounding-box coordinates: the coordinates of the smallest rectangular box containing the person or object.

[0082] In its specific implementation, this application can be based on the V-COCO dataset (containing 69 target categories and 29 relationships), dividing the relationship data into two parts: a Base class (data for training relationship categories in the first stage) and a Novel class (data for training relationship categories in the second stage). V-COCO is a subset of the COCO dataset, primarily a set of labeled human-object relationships. In this embodiment, the data for training relationship categories in the first stage consists of 24 categories, and the data for training relationship categories in the second stage consists of 5 categories.

[0083] The step of training a relationship detection model based on the data from the first-stage training of relationship categories, enabling the model to learn to predict object categories and bounding box coordinates, as well as human behavior categories and bounding box coordinates, results in the first-stage relationship detection model, including:

[0084] The data from the first-stage training of relation categories is input into the base network of the relation detection model to obtain the feature data of the first-stage image.

[0085] The feature data of the first stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the first stage are output.

[0086] Based on the feature data of objects and people in the first stage, the object detection branch and the relationship category detection branch are trained respectively, so that the relationship detection model learns to predict the category of objects and the bounding box coordinates of objects, as well as the behavior category of people and the bounding box coordinates of people, thus obtaining the first-stage relationship detection model.

[0087] like Figure 2 The first stage: Based on the Faster-RCNN detection model, after the Region of Interest Pooling (RoI Pooling) layer, a relation category detection branch is constructed. Figure 2 The algorithm consists of two branches: relation detection (Human) and object detection (Object detection). The Object branch has 68 categories (69-1=68), and the bounding-box branch regresses the bounding box coordinates of the objects. The Human branch has 29 categories, and the bounding-box branch regresses the bounding box coordinates of the Humans. In the first stage, only 24 base classes of data are input, and the dimensions of the vector are mapped to the actual number of categories (29) through a weight.

[0088] The specific training process in the first stage is as follows: First, the data of the relation categories trained in the first stage are input into the base network of the relation detection model (e.g., ...). Figure 2 The base network (backbone) obtains the feature data of the first-stage image; then, the feature data of the first-stage image is input into the region of interest pooling layer, which outputs the feature data of the first-stage object and the feature data of the person; then, the object detection branch (e.g., ...) is trained based on the feature data of the first-stage object. Figure 2 The object detection branch in the training algorithm); the relationship category detection branch trained based on human feature data (such as...). Figure 2 The first-stage relationship detection model is obtained by learning to predict the object's classification and bounding box coordinates, as well as the human behavior classification and bounding box coordinates.

[0089] like Figure 1 As shown, in step S102, in the second stage of model training, the first stage relationship detection model is trained based on the data of the second stage training relationship categories to obtain the final relationship detection model; wherein, the data of the second stage training relationship categories is small sample data, the number of relationship categories corresponding to the data of the first stage training relationship categories should be greater than the number of relationship categories corresponding to the data of the second stage training relationship categories, and the sum of the number of relationship categories corresponding to the data of the two stages is the number of relationship categories that the relationship detection model can predict.

[0090] The data used for training the relationship categories in the second stage is small sample data, which is difficult to collect. For example, scenarios such as people walking their dogs or littering outside of designated times are hard to collect. Furthermore, the number of relationship categories corresponding to the data from the first stage of relationship category training should be much greater than the number of relationship categories corresponding to the data from the second stage. Small sample data refers to data with a very small number of samples, such as 10 samples.

[0091] The step of continuing to train the first-stage relation detection model based on the data from the second-stage training of relation categories to obtain the final relation detection model includes:

[0092] The second set of training data for relation categories is input into the base network of the relation detection model, which outputs the feature data of the second-stage image.

[0093] The feature data of the second-stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the second stage are output.

[0094] The data from the second-stage training of relation categories is input into the meta-learning branch of the relation detection model to obtain relation category vectors;

[0095] Based on the relationship category vector and the second-stage human feature data, channel-level operations are performed to obtain the human target feature data.

[0096] Based on the feature data of objects in the second stage and the target feature data of people, the target detection branch and the relationship category detection branch in the first stage are trained respectively to obtain the final relationship detection model.

[0097] The step of performing channel-level operations based on the relation category vector and the second-stage human feature data to obtain the human target feature data includes:

[0098] The channel dimension convolution is performed on the relationship category vector and the human feature data of the second stage to obtain the human target feature data.

[0099] The second stage: The relationship detection model adds a meta-class vector prediction branch to predict the relationship class vector.

[0100] The second stage of training involves the following steps: First, the data for training the second set of relation categories is input into the base network of the relation detection model, outputting feature data for the second-stage image. Then, the feature data from the second-stage image is input into the region of interest (HOI) pooling layer, outputting feature data for the second-stage object and human. Next, the data for training the second-stage relation categories is input into the meta-learning branch of the relation detection model to obtain the relation category vector (HOI class vector). Finally, channel-level operations (such as...) are performed based on the relation category vector and the second-stage human feature data. Figure 2 The first stage object detection branch and the second stage relationship category detection branch are trained based on the second stage object feature data and the second stage object feature data and the second stage object feature data, respectively, to obtain the final relationship detection model. Among them, HOI: Human-Object Interaction, which is a human-centered model, identifies the relationship between a person and other objects (e.g., a car).

[0101] In the second stage of model training, a small amount of data was added to train the relationship detection model, so that the model could improve the Novel class (small number of samples) index as much as possible when data was insufficient.

[0102] When predicting human behavior categories and bounding box coordinates, spatial feature data between people and objects can be used to suppress invalid samples in the predicted candidate-object relationship set, thereby improving the model's prediction precision.

[0103] The process involves training the first-stage object detection branch and the relationship category detection branch based on the feature data of the object in the second stage and the target feature data of the person, respectively, to obtain the final relationship detection model, including:

[0104] The target detection branch is trained based on the feature data of the objects in the second stage, so that the first stage relationship detection model can learn to predict the object category and the bounding box coordinates of the object.

[0105] The relationship category detection branch is trained based on the target feature data of people in the second stage and the spatial feature data between people and objects. This enables the first-stage relationship detection model to learn to predict the behavior category of people and the bounding box coordinates of people, thus obtaining the final relationship detection model. Among them, the spatial feature data between people and objects is the data used to characterize the relative position between people and objects in space.

[0106] The first embodiment of this application may further include: obtaining spatial feature data between people and objects.

[0107] The acquisition of spatial feature data between people and objects includes:

[0108] Based on the human characteristic data from the second stage, obtain the bounding box coordinates of the human;

[0109] Obtain the bounding box coordinates of the object based on the object's feature data from the second stage;

[0110] Based on the bounding box coordinates of people and objects, spatial feature data between people and objects is obtained.

[0111] like Figure 3 As shown, 3-1 represents the characteristics of an object, 3-2 represents the characteristics of a person, and 3-3 represents the spatial characteristics between a person and an object.

[0112] In the first embodiment of this application, after training the first-stage relation detection model based on the target relation category vector to obtain the final relation detection model, the method may further include:

[0113] Store relationship category vectors for use during the testing phase.

[0114] For example, after the second phase of training is completed, the relation category vectors of the 29 relations are saved for use in the testing phase.

[0115] As one implementation method, embodiments of this application may further include:

[0116] Obtain the image of the person relationship category to be determined;

[0117] The image of the person relationship category to be determined is input into the base network of the final relationship detection model, and the feature data of the image of the person relationship category to be determined is output.

[0118] The feature data of the image whose relationship category to be determined is input into the region of interest pooling layer, and the feature data of the object and the feature data of the person are output.

[0119] Based on the stored relation category vectors and human feature data, the final target feature data of the human is obtained;

[0120] Input the object's feature data into the object detection branch to predict the object's category and bounding box coordinates;

[0121] Input the final target feature data of a person into the relation category branch to predict the person's behavior category and the person's bounding box coordinates;

[0122] The final relationship category is determined based on the set of object category scores and human behavior category scores.

[0123] The determination of the final relationship category based on the set of object category scores and human behavior category scores includes:

[0124] For the set of category scores for the objects and category scores for the human behavior, calculate the product of each object category score and each behavior score, and the one with the highest score is the final relationship category.

[0125] Figure 4 This is a schematic diagram of a relationship detection model test provided in the first embodiment of this application.

[0126] Corresponding to the model training method that introduces meta-learning provided in the first embodiment of this application, the second embodiment of this application provides a data processing device.

[0127] like Figure 5 As shown, the model training device incorporating meta-learning includes:

[0128] The first-stage training unit 501 is used to train a relation detection model in the first stage of model training based on the relation category data from the first-stage training. This enables the relation detection model to learn to predict the category and bounding box coordinates of objects, as well as the behavior category and bounding box coordinates of people, thus obtaining the first-stage relation detection model. The relation detection model includes an object detection branch and a relation category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relation category detection branch is used to predict the behavior category and bounding box coordinates of people.

[0129] The second-stage training unit 502 is used to continue training the first-stage relation detection model based on the second-stage training relation category data in the second stage of model training, so as to obtain the final relation detection model. The second-stage training relation category data is small sample data, the number of relation categories corresponding to the first-stage training relation category data should be greater than the number of relation categories corresponding to the second-stage training relation category data, and the sum of the number of relation categories corresponding to the data in the two stages is the number of relation categories that the relation detection model can predict.

[0130] As one implementation method, the first-stage training unit is specifically used for:

[0131] The data from the first-stage training of relation categories is input into the base network of the relation detection model to obtain the feature data of the first-stage image.

[0132] The feature data of the first stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the first stage are output.

[0133] Based on the feature data of objects and people in the first stage, the object detection branch and the relationship category detection branch are trained respectively, so that the relationship detection model learns to predict the category of objects and the bounding box coordinates of objects, as well as the behavior category of people and the bounding box coordinates of people, thus obtaining the first-stage relationship detection model.

[0134] As one implementation method, the second-stage training unit is specifically used for:

[0135] The second set of training data for relation categories is input into the base network of the relation detection model, which outputs the feature data of the second-stage image.

[0136] The feature data of the second-stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the second stage are output.

[0137] The data from the second-stage training of relation categories is input into the meta-learning branch of the relation detection model to obtain relation category vectors;

[0138] Based on the relationship category vector and the second-stage human feature data, channel-level operations are performed to obtain the human target feature data.

[0139] Based on the feature data of objects in the second stage and the target feature data of people, the target detection branch and the relationship category detection branch in the first stage are trained respectively to obtain the final relationship detection model.

[0140] As one implementation method, the second-stage training unit is specifically used for:

[0141] The channel dimension convolution is performed on the relationship category vector and the human feature data of the second stage to obtain the human target feature data.

[0142] As one implementation method, the second-stage training unit is specifically used for:

[0143] The target detection branch is trained based on the feature data of the objects in the second stage, so that the first stage relationship detection model can learn to predict the object category and the bounding box coordinates of the object.

[0144] The relationship category detection branch is trained based on the target feature data of people in the second stage and the spatial feature data between people and objects. This enables the first-stage relationship detection model to learn to predict the behavior category of people and the bounding box coordinates of people, thus obtaining the final relationship detection model. Among them, the spatial feature data between people and objects is the data used to characterize the relative position between people and objects in space.

[0145] In one embodiment, the device further includes a spatial feature data acquisition unit for acquiring spatial feature data between people and objects.

[0146] As one implementation, the spatial feature data acquisition unit includes:

[0147] Based on the human characteristic data from the second stage, obtain the bounding box coordinates of the human;

[0148] Obtain the bounding box coordinates of the object based on the object's feature data from the second stage;

[0149] Based on the bounding box coordinates of people and objects, spatial feature data between people and objects is obtained.

[0150] In one embodiment, the apparatus further includes a relation category vector storage unit, used to store the relation category vector after the second-stage training unit has worked, for use in the testing phase.

[0151] In one embodiment, the apparatus further includes a testing unit for:

[0152] Obtain the image of the person relationship category to be determined;

[0153] The image of the relationship category to be determined is input into the base network of the final relationship detection model, and the feature data of the image of the relationship category to be determined is output.

[0154] The feature data of the image whose relationship category to be determined is input into the region of interest pooling layer, and the feature data of the object and the feature data of the person are output.

[0155] Based on the stored relation category vectors and human feature data, the final target feature data of the human is obtained;

[0156] Input the object's feature data into the object detection branch to predict the object's category and bounding box coordinates;

[0157] Input the final target feature data of a person into the relation category branch to predict the person's behavior category and the person's bounding box coordinates;

[0158] The final relationship category is determined based on the set of object category scores and human behavior category scores.

[0159] Optionally, the test unit is specifically used for:

[0160] For the set of category scores for the objects and category scores for the human behavior, calculate the product of each object category score and each behavior score, and the one with the highest score is the final relationship category.

[0161] It should be noted that for a detailed description of the apparatus provided in the second embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0162] Corresponding to the model training method incorporating meta-learning provided in the first embodiment of this application, the third embodiment of this application provides an electronic device, including:

[0163] Processor; and

[0164] The memory stores the program for the model training method incorporating meta-learning. After the device is powered on and the program for the model training method incorporating meta-learning is run by the processor, the following steps are performed:

[0165] In the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0166] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0167] As one implementation method, the step of training a relationship detection model based on the data from the first-stage training of relationship categories, enabling the relationship detection model to learn to predict the category of objects and the bounding box coordinates of objects, as well as the category of human behavior and the bounding box coordinates of humans, to obtain the first-stage relationship detection model, includes:

[0168] The data from the first-stage training of relation categories is input into the base network of the relation detection model to obtain the feature data of the first-stage image.

[0169] The feature data of the first stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the first stage are output.

[0170] Based on the feature data of objects and people in the first stage, the object detection branch and the relationship category detection branch are trained respectively, so that the relationship detection model learns to predict the category of objects and the bounding box coordinates of objects, as well as the behavior category of people and the bounding box coordinates of people, thus obtaining the first-stage relationship detection model.

[0171] As one implementation method, the step of continuing to train the first-stage relation detection model based on the data from the second-stage training of relation categories to obtain the final relation detection model includes:

[0172] The second set of training data for relation categories is input into the base network of the relation detection model, which outputs the feature data of the second-stage image.

[0173] The feature data of the second-stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the second stage are output.

[0174] The data from the second-stage training of relation categories is input into the meta-learning branch of the relation detection model to obtain relation category vectors;

[0175] Based on the relationship category vector and the second-stage human feature data, channel-level operations are performed to obtain the human target feature data.

[0176] Based on the feature data of objects in the second stage and the target feature data of people, the target detection branch and the relationship category detection branch in the first stage are trained respectively to obtain the final relationship detection model.

[0177] As one implementation method, the step of performing channel-level operations based on the relation category vector and the second-stage human feature data to obtain the human target feature data includes:

[0178] The channel dimension convolution is performed on the relationship category vector and the human feature data of the second stage to obtain the human target feature data.

[0179] As one implementation method, the first-stage object detection branch and relation category detection branch are trained based on the feature data of the object in the second stage and the target feature data of the person, respectively, to obtain the final relation detection model, including:

[0180] The target detection branch is trained based on the feature data of the objects in the second stage, so that the first stage relationship detection model can learn to predict the object category and the bounding box coordinates of the object.

[0181] The relationship category detection branch is trained based on the target feature data of people in the second stage and the spatial feature data between people and objects. This enables the first-stage relationship detection model to learn to predict the behavior category of people and the bounding box coordinates of people, thus obtaining the final relationship detection model. Among them, the spatial feature data between people and objects is the data used to characterize the relative position between people and objects in space.

[0182] In one implementation, the electronic device also performs the following steps: obtaining spatial feature data between people and objects.

[0183] As one implementation method, obtaining spatial feature data between people and objects includes:

[0184] Based on the human characteristic data from the second stage, obtain the bounding box coordinates of the human;

[0185] Obtain the bounding box coordinates of the object based on the object's feature data from the second stage;

[0186] Based on the bounding box coordinates of people and objects, spatial feature data between people and objects is obtained.

[0187] As one implementation, after training the first-stage relation detection model based on the target relation category vector to obtain the final relation detection model, the method further includes:

[0188] Store the relationship category vector for use during the testing phase.

[0189] In one implementation, the electronic device further performs the following steps:

[0190] Obtain the image of the person relationship category to be determined;

[0191] The image of the person relationship category to be determined is input into the base network of the final relationship detection model, and the feature data of the image of the person relationship category to be determined is output.

[0192] The feature data of the image whose relationship category to be determined is input into the region of interest pooling layer, and the feature data of the object and the feature data of the person are output.

[0193] Based on the stored relation category vectors and human feature data, the final target feature data of the human is obtained;

[0194] Input the object's feature data into the object detection branch to predict the object's category and bounding box coordinates;

[0195] Input the final target feature data of a person into the relation category branch to predict the person's behavior category and the person's bounding box coordinates;

[0196] The final relationship category is determined based on the set of object category scores and human behavior category scores.

[0197] As one implementation method, determining the final relationship category based on the set of object category scores and human behavior category scores includes:

[0198] For the set of category scores for the objects and category scores for the human behavior, calculate the product of each object category score and each behavior score, and the one with the highest score is the final relationship category.

[0199] It should be noted that for a detailed description of the electronic device provided in the third embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0200] Corresponding to the model training method incorporating meta-learning provided in the first embodiment of this application, the fourth embodiment of this application provides a storage device storing a program for a model training method incorporating meta-learning. This program is executed by a processor to perform the following steps:

[0201] In the first stage of model training, a relationship detection model is trained based on the data of the relationship categories trained in the first stage. This allows the relationship detection model to learn to predict the category and bounding box coordinates of objects, as well as the category and bounding box coordinates of human behavior, thus obtaining the first-stage relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch is used to predict the category and bounding box coordinates of objects, and the relationship category detection branch is used to predict the category and bounding box coordinates of human behavior.

[0202] In the second stage of model training, the first-stage relation detection model is trained again based on the data of the second-stage training relation categories to obtain the final relation detection model. The data of the second-stage training relation categories is small sample data, and the number of relation categories corresponding to the data of the first-stage training relation categories should be greater than the number of relation categories corresponding to the data of the second-stage training relation categories. The sum of the number of relation categories corresponding to the data of the two stages is the number of relation categories that the relation detection model can predict.

[0203] It should be noted that for a detailed description of the storage device provided in the fourth embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0204] The fifth embodiment of this application provides a model training system that incorporates meta-learning.

[0205] like Figure 6 As shown, the model training system that incorporates meta-learning includes: a platform server 601 and a user terminal 602;

[0206] In the fifth embodiment of this application, the so-called platform server 601 refers to a computing device that provides services to the software platform or application platform installed on the user terminal 602 for executing the model training method of introducing meta-learning provided in this application. In specific implementations, it is generally a server or server cluster. The so-called user terminal 602 refers to a computing device that interacts with the platform server 601. The computing device has a software platform or application platform installed on it, and in specific implementations, it is generally a smartphone, tablet computer, personal computer, etc. After the user terminal 602 uploads the data of the first-stage training relation categories and the data of the second-stage training relation categories to the platform server 601 through the software platform or application platform installed on it, the platform server 601 obtains the final relation detection model using a SaaS (Software-as-a-Service) working mode and provides it to the user terminal 602.

[0207] The platform server 601 is used to obtain the first-stage training relation category data and the second-stage training relation category data sent by the user terminal; in the first stage of model training, a relation detection model is trained based on the first-stage training relation category data, enabling the relation detection model to learn to predict the category of objects and the category of human behavior, thus obtaining the first-stage relation detection model; in the second stage of model training, the first-stage relation detection model is further trained based on the second-stage training relation category data, thus obtaining the final relation detection model; wherein, the second-stage training relation category data is small sample data; and the final relation detection model is sent to the user terminal;

[0208] The user terminal 602 is used to send the data of the first stage training relationship categories and the data of the second stage training relationship categories to the platform server; and to obtain the final relationship detection model sent by the platform server.

[0209] The sixth embodiment of this application provides a method for identifying relationship categories, including:

[0210] Obtain the image of the person relationship category to be determined;

[0211] The image of the person-object relationship category to be determined is input into the final relationship detection model to obtain the relationship category of people and objects in the image of the person-object relationship category to be determined;

[0212] The final relationship detection model is obtained in the following manner:

[0213] In the first stage of model training, the relationship detection model is trained based on the data of the relationship categories trained in the first stage. This enables the relationship detection model to learn to predict the categories of objects and the categories of human behavior, thus obtaining the first-stage relationship detection model.

[0214] In the second stage of model training, the first-stage relationship detection model is trained again based on the data of the second-stage training relationship categories to obtain the final relationship detection model; the data of the second-stage training relationship categories is small sample data.

[0215] It should be noted that a detailed description of the relationship category identification method provided in the sixth embodiment of this application can be found in the relevant description of the first embodiment of this application, and will not be repeated here.

[0216] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0217] In a typical configuration, a computing device includes one or more processors (CPUs), memory-mapped input / output interfaces, network interfaces, and memory.

[0218] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0219] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0220] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A model training system incorporating meta-learning, characterized in that, include: Platform server and user terminal; The platform server is used to obtain data on the first-stage training relationship categories and the second-stage training relationship categories sent by the user terminal. In the first stage of model training, a relationship detection model is trained based on the data from the first stage training of relationship categories. This model learns to predict object categories and human behavior categories, resulting in a first-stage relationship detection model. This first-stage model is obtained by training the relationship detection model using the feature data of objects and humans from the first stage, which is obtained by inputting the data from the first stage training of relationship categories into the model. In the second stage of model training, the first-stage relationship detection model is further trained based on the data from the second stage training of relationship categories, resulting in a final relationship detection model. This final model is obtained by training the first-stage relationship detection model using the feature data of objects and human target features from the second stage. The feature data of objects in the second stage is obtained by inputting the data from the second stage training of relationship categories into the first-stage model. The target feature data of humans is obtained by performing calculations based on the relationship category vector and the second-stage human feature data. The relationship category vector is obtained by inputting the data from the second stage training of relationship categories into the meta-learning branch of the relationship detection model. The data from the second stage training of relationship categories consists of small sample data. The final relationship detection model is then sent to the user terminal. The user terminal is used to send the data of the first stage training relationship categories and the data of the second stage training relationship categories to the platform server; and to obtain the final relationship detection model sent by the platform server.

2. A method for identifying relation categories, characterized in that, include: Obtain the image of the person relationship category to be determined; The image of the person-object relationship category to be determined is input into the final relationship detection model to obtain the relationship category of people and objects in the image of the person-object relationship category to be determined; The final relationship detection model is obtained in the following manner: In the first stage of model training, a relationship detection model is trained based on the data of the first stage training relationship categories. This enables the relationship detection model to learn to predict the categories of objects and the categories of human behavior, resulting in the first-stage relationship detection model. The first-stage relationship detection model is obtained by training the relationship detection model based on the feature data of objects and the feature data of humans in the first stage. The feature data of objects and humans in the first stage are obtained by inputting the data of the first stage training relationship categories into the relationship detection model. In the second stage of model training, the first-stage relationship detection model is trained again based on the data from the second-stage training of relationship categories, resulting in the final relationship detection model. This final model is obtained by training the first-stage relationship detection model using the feature data of objects and the target feature data of people from the second stage. The feature data of objects in the second stage is obtained by inputting the data from the second-stage training of relationship categories into the first-stage relationship detection model. The target feature data of people is obtained by performing calculations based on the relationship category vector and the feature data of people from the second stage. The relationship category vector is obtained by inputting the data from the second-stage training of relationship categories into the meta-learning branch of the relationship detection model. The data from the second-stage training of relationship categories is small sample data.

3. A model training method incorporating meta-learning, characterized in that, include: In the first stage of model training, a relationship detection model is trained based on the relationship category data from the first stage training. This allows the relationship detection model to learn to predict the object category and bounding box coordinates, as well as the human behavior category and bounding box coordinates, resulting in the first-stage relationship detection model. This first-stage relationship detection model is obtained by training the relationship detection model using the object feature data and human feature data from the first stage training. The object feature data and human feature data from the first stage are obtained by inputting the relationship category training data from the first stage training into the relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch predicts the object category and bounding box coordinates, while the relationship category detection branch predicts the human behavior category and bounding box coordinates. In the second stage of model training, the first-stage relationship detection model is trained again based on the data from the second-stage training of relationship categories, resulting in the final relationship detection model. This final model is obtained by training the first-stage relationship detection model using the feature data of objects and the target feature data of people from the second stage. The feature data of objects in the second stage is obtained by inputting the data from the second-stage training of relationship categories into the first-stage relationship detection model. The target feature data of people is obtained by performing calculations based on the relationship category vector and the feature data of people from the second stage. The relationship category vector is obtained by inputting the data from the second-stage training of relationship categories into the meta-learning branch of the relationship detection model. The data from the second-stage training of relationship categories is small sample data. The number of relationship categories corresponding to the data from the first-stage training of relationship categories should be greater than the number of relationship categories corresponding to the data from the second-stage training of relationship categories. The sum of the number of relationship categories corresponding to the data from both stages is the number of relationship categories that the relationship detection model can predict.

4. The method according to claim 3, characterized in that, The step of training a relationship detection model based on the data from the first-stage training of relationship categories, enabling the model to learn to predict object categories and bounding box coordinates, as well as human behavior categories and bounding box coordinates, results in the first-stage relationship detection model, including: The data from the first-stage training of relation categories is input into the base network of the relation detection model to obtain the feature data of the first-stage image. The feature data of the first stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the first stage are output. Based on the feature data of objects and people in the first stage, the object detection branch and the relationship category detection branch are trained respectively, so that the relationship detection model learns to predict the category of objects and the bounding box coordinates of objects, as well as the behavior category of people and the bounding box coordinates of people, thus obtaining the first-stage relationship detection model.

5. The method according to claim 4, characterized in that, The step of continuing to train the first-stage relation detection model based on the data from the second-stage training of relation categories to obtain the final relation detection model includes: The second set of training data for relation categories is input into the base network of the relation detection model, which outputs the feature data of the second-stage image. The feature data of the second-stage image is input into the region of interest pooling layer, and the feature data of the object and the human in the second stage are output. The data from the second-stage training of relation categories is input into the meta-learning branch of the relation detection model to obtain relation category vectors; Based on the relationship category vector and the second-stage human feature data, channel-level operations are performed to obtain the human target feature data. Based on the feature data of objects in the second stage and the target feature data of people, the target detection branch and the relationship category detection branch in the first stage are trained respectively to obtain the final relationship detection model.

6. The method according to claim 5, characterized in that, The step of performing channel-level operations based on the relation category vector and the second-stage human feature data to obtain the human target feature data includes: The channel dimension convolution is performed on the relationship category vector and the human feature data of the second stage to obtain the human target feature data.

7. The method according to claim 5, characterized in that, Based on the feature data of objects in the second stage and the target feature data of people, the object detection branch and the relationship category detection branch in the first stage are trained respectively, resulting in the final relationship detection model, including: The target detection branch is trained based on the feature data of the objects in the second stage, so that the first stage relationship detection model can learn to predict the object category and the bounding box coordinates of the object. The relationship category detection branch is trained based on the target feature data of people in the second stage and the spatial feature data between people and objects. This enables the first-stage relationship detection model to learn to predict the behavior category of people and the bounding box coordinates of people, thus obtaining the final relationship detection model. Among them, the spatial feature data between people and objects is the data used to characterize the relative position between people and objects in space.

8. The method according to claim 7, characterized in that, Also includes: Obtain spatial feature data between people and objects.

9. The method according to claim 8, characterized in that, The acquisition of spatial feature data between people and objects includes: Based on the human characteristic data from the second stage, obtain the bounding box coordinates of the human; Obtain the bounding box coordinates of the object based on the object's feature data from the second stage; Based on the bounding box coordinates of people and objects, spatial feature data between people and objects is obtained.

10. The method according to claim 5, characterized in that, Based on the relation category vector, train the first... one After obtaining the final relationship detection model from the stage-based relationship detection model, the following steps are also included: Store the relationship category vector for use during the testing phase.

11. The method according to claim 10, characterized in that, Also includes: Obtain the image of the person relationship category to be determined; The image of the person relationship category to be determined is input into the base network of the final relationship detection model, and the feature data of the image of the person relationship category to be determined is output. The feature data of the image whose relationship category to be determined is input into the region of interest pooling layer, and the feature data of the object and the feature data of the person are output. Based on the stored relation category vectors and human feature data, the final target feature data of the human is obtained; Input the object's feature data into the object detection branch to predict the object's category and bounding box coordinates; Input the final target feature data of a person into the relation category branch to predict the person's behavior category and the person's bounding box coordinates; The final relationship category is determined based on the set of object category scores and human behavior category scores.

12. The method according to claim 10, characterized in that, The determination of the final relationship category based on the set of object category scores and human behavior category scores includes: For the set of category scores for the objects and category scores for the human behavior, calculate the product of each object category score and each behavior score, and the one with the highest score is the final relationship category.

13. A model training device incorporating meta-learning, characterized in that, include: The first-stage training unit is used in the first stage of model training to train a relationship detection model based on the relationship category data from the first-stage training. This enables the relationship detection model to learn and predict object categories and bounding box coordinates, as well as human behavior categories and bounding box coordinates, resulting in the first-stage relationship detection model. This first-stage relationship detection model is obtained by training the relationship detection model based on the feature data of objects and humans from the first stage. The feature data of objects and humans from the first stage are obtained by inputting the relationship category data from the first-stage training into the relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch predicts the object category and bounding box coordinates, while the relationship category detection branch predicts the human behavior category and bounding box coordinates. The second-stage training unit is used in the second stage of model training to continue training the first-stage relationship detection model based on the data of the second-stage training relationship categories, thus obtaining the final relationship detection model. The final relationship detection model is obtained by training the first-stage relationship detection model using the feature data of objects and the target feature data of people from the second stage. The feature data of objects in the second stage is obtained by inputting the data of the second-stage training relationship categories into the first-stage relationship detection model. The target feature data of people is obtained by performing calculations based on the relationship category vector and the second-stage human feature data. The relationship category vector is obtained by inputting the data of the second-stage training relationship categories into the meta-learning branch of the relationship detection model. The data of the second-stage training relationship categories is small sample data. The number of relationship categories corresponding to the data of the first-stage training relationship categories should be greater than the number of relationship categories corresponding to the data of the second-stage training relationship categories. The sum of the number of relationship categories corresponding to the data of the two stages is the number of relationship categories that the relationship detection model can predict.

14. An electronic device, characterized in that, include: processor; as well as The memory stores the program for the model training method incorporating meta-learning. After the device is powered on and the program for the model training method incorporating meta-learning is run by the processor, the following steps are performed: In the first stage of model training, a relationship detection model is trained based on the relationship category data from the first stage training. This allows the relationship detection model to learn to predict the object category and bounding box coordinates, as well as the human behavior category and bounding box coordinates, resulting in the first-stage relationship detection model. This first-stage relationship detection model is obtained by training the relationship detection model using the object feature data and human feature data from the first stage training. The object feature data and human feature data from the first stage are obtained by inputting the relationship category training data from the first stage training into the relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch predicts the object category and bounding box coordinates, while the relationship category detection branch predicts the human behavior category and bounding box coordinates. In the second stage of model training, the first-stage relationship detection model is trained again based on the data from the second-stage training of relationship categories, resulting in the final relationship detection model. This final model is obtained by training the first-stage relationship detection model using the feature data of objects and the target feature data of people from the second stage. The feature data of objects in the second stage is obtained by inputting the data from the second-stage training of relationship categories into the first-stage relationship detection model. The target feature data of people is obtained by performing calculations based on the relationship category vector and the feature data of people from the second stage. The relationship category vector is obtained by inputting the data from the second-stage training of relationship categories into the meta-learning branch of the relationship detection model. The data from the second-stage training of relationship categories is small sample data. The number of relationship categories corresponding to the data from the first-stage training of relationship categories should be greater than the number of relationship categories corresponding to the data from the second-stage training of relationship categories. The sum of the number of relationship categories corresponding to the data from both stages is the number of relationship categories that the relationship detection model can predict.

15. A storage device, characterized in that, A program containing a model training method that incorporates meta-learning is executed by the processor, performing the following steps: In the first stage of model training, a relationship detection model is trained based on the relationship category data from the first stage training. This allows the relationship detection model to learn to predict the object category and bounding box coordinates, as well as the human behavior category and bounding box coordinates, resulting in the first-stage relationship detection model. This first-stage relationship detection model is obtained by training the relationship detection model using the object feature data and human feature data from the first stage training. The object feature data and human feature data from the first stage are obtained by inputting the relationship category training data from the first stage training into the relationship detection model. The relationship detection model includes an object detection branch and a relationship category detection branch. The object detection branch predicts the object category and bounding box coordinates, while the relationship category detection branch predicts the human behavior category and bounding box coordinates. In the second stage of model training, the first-stage relationship detection model is trained again based on the data from the second-stage training of relationship categories, resulting in the final relationship detection model. This final model is obtained by training the first-stage relationship detection model using the feature data of objects and the target feature data of people from the second stage. The feature data of objects in the second stage is obtained by inputting the data from the second-stage training of relationship categories into the first-stage relationship detection model. The target feature data of people is obtained by performing calculations based on the relationship category vector and the feature data of people from the second stage. The relationship category vector is obtained by inputting the data from the second-stage training of relationship categories into the meta-learning branch of the relationship detection model. The data from the second-stage training of relationship categories is small sample data. The number of relationship categories corresponding to the data from the first-stage training of relationship categories should be greater than the number of relationship categories corresponding to the data from the second-stage training of relationship categories. The sum of the number of relationship categories corresponding to the data from both stages is the number of relationship categories that the relationship detection model can predict.

Images