Sustainable learning human interaction intelligent detection method, system and storage medium

Abstract

The present invention discloses a sustainable learning method, system and storable medium for intelligent detection of human interaction, which relates to the technical field of human interaction detection, including: obtaining an image data set and dividing it into different human interaction detection tasks; constructing a detection model for the human interaction detection task, training the detection model, and when the number of training iterations reaches a first preset parameter I, the pre-training ends, and then comparative learning is introduced to continue training the second preset parameter A times to obtain the final detection model, and using an interaction category classifier to predict the interaction category; after each human interaction detection task is completed, the knowledge of the new category is incrementally learned, and the preserved old category samples and feature fine-tuning model are used to obtain a sustainable learning model for human interaction detection, and complete the intelligent detection of human interaction. The technical solution in the present invention can enable the sustainable learning model for human interaction detection to have good detection capabilities for both new and old categories, thereby improving the detection effect.

Description

Technical Field

[0001] The present invention relates to the technical field of human interaction detection, and more specifically to a sustainable learning human interaction intelligent detection method, system and storable medium. Background Art

[0002] The problem of human interaction detection is one of the foundations of human-centric visual understanding tasks. It not only includes detecting people and objects, but also inferring the interaction relationship between pairs of people. Researching the problem of human interaction detection is of great help to downstream tasks (such as image generation description, image retrieval, visual question answering tasks, etc.). Current research methods are all based on strong assumptions, that is, all detection categories are known at the beginning of training. Models trained under such assumptions can usually achieve good results, but there are still two challenging scenarios: 1) When new categories are introduced, the model does not have the ability to detect new categories, so it is necessary to train on new categories. Considering the limitations of memory and computing resources, data privacy and other issues, it is obviously unrealistic to retrain a model every time; 2) As the number of new categories introduced increases, the model trained only on new categories will have catastrophic forgetting problems.

[0003] The number of objects and interaction categories annotated in the datasets commonly used for human interaction detection tasks is relatively small. For example, HICO-DET and V-COCO annotate 80 object categories, 117 and 29 action categories, respectively. Compared with the objects and interaction categories in the real world, the number of dozens or hundreds of categories is obviously not enough, and the data categories required for different tasks are not completely consistent. Therefore, in practical applications, it may be necessary to continuously introduce new categories for training. In recent years, research on incremental learning has gradually emerged, and researchers have also begun to use incremental learning to solve similar problems, which has achieved good results. However, how to select the weights of different parameters in parameter-based incremental learning methods and the memory burden problem in playback-based incremental learning methods are still not negligible. There is currently no research on the continuous learning problem of human interaction detection models.

[0004] Compared with the incremental learning problems in tasks such as target detection, target classification or behavior recognition, the incremental learning problems in person interaction detection have two main difficulties: first, the features are more complex. The model needs to process not only the features of individual people or objects, but also the joint features of people and objects; second, the memory burden problem. Person interaction detection consists of people, objects, and interactions. However, during the training process, the combination of people and objects is not fixed. Therefore, the model may encounter unknown person features and known object features, known person features and unknown object features, as well as known person and object features and unknown action category features. Based on such potential scenarios, the continuous learning model of person interaction detection needs to have the ability to detect people and objects at the same time and accurately infer the interaction relationship.

[0005] Therefore, how to make the sustainable learning model of human interaction detection have good detection capabilities for both new and old categories and improve the detection effect is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a method, system and storable medium for intelligent detection of human interaction with continuous learning, which solve the problems existing in the above background technologies.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A human interaction intelligent detection method with continuous learning includes the following steps:

[0009] Acquire an image dataset, and divide the image dataset into different human interaction detection tasks;

[0010] Constructing a detection model for the task of human interaction detection, training the detection model, and detecting people and objects in the image, and when the number of training iterations reaches a first preset parameter I, the pre-training ends, and a pre-trained detection model is obtained;

[0011] Using the pre-trained detection model, introducing contrastive learning to continue training the second preset parameter A times to obtain a final detection model, and using an interaction category classifier to predict the interaction category;

[0012] After each person interaction detection task is completed, the knowledge of new categories is incrementally learned, and the model is fine-tuned using the saved old category samples and features to obtain a sustainable learning model for person interaction detection and complete intelligent person interaction detection.

[0013] The technical effect achieved by the above technical solution is: the sustainable learning model of human interaction detection has good detection capabilities for both new and old categories, thereby improving the detection effect.

[0014] Optionally, build a detection model for the human interaction detection task, specifically:

[0015] Based on the structure of Faster-RCNN, the RPN network is replaced by HO-RPN, which uses human features to filter objects that do not interact.

[0016] Optionally, the obtaining of the final detection model is specifically as follows:

[0017] The image data set in the human interaction detection task is input into the detection model for training. When the number of training iterations is greater than the first preset parameter I, the model is continued to be trained using the self-supervised comparative clustering method. The number of iterations is the second preset parameter A times. At the same time, the human, object and joint interaction features are learned, the cluster feature vector is maintained, and the cluster feature vector is saved in the feature library.

[0018] Optionally, the features in the feature library are respectively category features of objects and category features of interactions, which are composed of human, object and action features.

[0019] Optionally, the comparative clustering method specifically includes: increasing the difference of heterogeneous features and reducing the difference of homogeneous features, so that different categories of features are represented by representations with greater differences in the feature space;

[0020] For each object category i∈K t and interaction category j∈H t , maintain a prototype vector β i and θ j , the contrast loss function is defined as:

[0021]

[0022] in,

[0023]

[0024] Where: f o Represents the characteristics of the object, β i represents the prototype vector of this type of object, and δ defines the similarity between similar and dissimilar items; Represents the interaction contrast loss, which is different from the object contrast loss The calculation method is the same as that of ; C represents the total number of object categories, M represents the total number of interaction categories, Dis(f o ,β i ) represents an arbitrary distance function.

[0025] Optionally, the predicted interaction category is specifically:

[0026] Detecting people and objects using the final detection model, combining people and objects in pairs, and determining whether they interact;

[0027] HO-RPN is used to filter objects without interaction, and the interaction category classifier is used to predict the interaction category.

[0028] Optionally, obtaining a sustainable learning model for human interaction detection specifically includes the following steps:

[0029] Design a feature replay network to obtain old category sample sets and high-dimensional clustering feature vectors as sample sets;

[0030] After each task is completed, the model is fine-tuned using the data in the sample set so that the fine-tuned model can extract significant features of the old category samples.

[0031] Optionally, the designed feature playback network includes two decoders, two classifiers and a backbone network; fine-tuning the model specifically includes the following steps:

[0032] Use the decoder to restore the object clustering feature F o and the interactive clustering feature F hoi , whose corresponding category is c o and c hoi ;

[0033] The object clustering features and interaction clustering features are restored to the real image size and sent to the backbone network to re-extract features, and new object clustering features F o ' and interactive clustering feature F hoi ';

[0034] Reconstruct based on the re-extracted features to obtain the restored image, and send the restored image to the object category classifier and the interaction category classifier respectively to obtain c o ' and c hoi ';

[0035] Fine-tune the model until F o and F o 'Consistent, F hoi and F hoi 'Consistent, c o =c o '、c hoi =c hoi ' and complete fine-tuning to obtain a sustainable learning model for human interaction detection.

[0036] The present invention also provides a sustainable learning character interaction intelligent detection system, comprising:

[0037] An acquisition module, used for acquiring an image data set and dividing the image data set into different human interaction detection tasks;

[0038] A construction module is used to build a detection model for human interaction detection tasks;

[0039] A training module is used to train the detection model to obtain a pre-trained detection model; then introduce contrast learning and continue training to obtain a final detection model;

[0040] A detection module detects people and objects in the image and predicts interaction categories by using the final detection model;

[0041] The sustainable learning module is used to incrementally learn new category knowledge after each character interaction detection task is completed, and use the saved old category samples and feature fine-tuning models to obtain a sustainable learning model for character interaction detection and complete intelligent character interaction detection.

[0042] The present invention also provides a computer storable medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned method for intelligent detection of human interaction are implemented.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method, system and storage medium for intelligent detection of human interaction with sustainable learning, which has the following beneficial effects:

[0044] (1) The present invention proposes a new character interaction detection task, namely, the character interaction detection continuous learning problem, which has good detection capabilities for both new and old categories;

[0045] (2) The present invention proposes a character interaction detection model based on incremental learning - feature playback network, which can solve the problem of continuous learning of character interaction detection;

[0046] (3) The present invention introduces a comparative clustering method based on human interaction into the model, which can be used to divide categories in the feature space to improve the detection effect, and can also be used in the feature playback network to alleviate the catastrophic forgetting problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 Flowchart of the intelligent detection method of human interaction for sustainable learning;

[0049] Figure 2It is a schematic diagram of the overall framework of the detection model;

[0050] Figure 3 Schematic diagram of the feature replay network structure. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example 1

[0053] Since it was first proposed, the problem of human interaction detection has played an important role in human-centered visual understanding tasks. Compared with other visual relationship understanding tasks, human interaction detection needs to process more fine-grained features. Accurate recognition of human interaction requires simultaneous consideration of human body part information, human spatial information, human posture information, etc. Current methods can be mainly divided into two types: sequential methods and parallel methods. Sequential methods first perform target detection, identify people and objects, combine people and objects in pairs, and determine whether they interact and the interaction category; parallel methods perform target detection and interactive action interpretation at the same time, usually using interaction points or interaction areas to combine target detection and action classification. Although parallel methods are faster than sequential methods, the definition of interaction points or interaction areas is rough, which is prone to errors.

[0054] This embodiment proposes a new character interaction detection task, namely, a continuous learning problem of character interaction detection. In order to enable the continuous learning model of character interaction detection to simultaneously have the ability to detect people and objects and accurately infer the interaction relationship, a continuous learning character interaction intelligent detection method is now disclosed. Figure 1-2 As shown in the figure, the following steps are included: (1) obtaining an image dataset and dividing the image dataset into different human interaction detection tasks; (2) constructing a detection model for the human interaction detection task and training the detection model to obtain a pre-trained detection model, then introducing contrastive learning and continuing training to obtain the final detection model; (3) using the final detection model to detect people and objects in the image and predict the interaction category; (4) incrementally learning the knowledge of new categories after each human interaction detection task, using the saved old category samples and feature fine-tuning model to obtain a sustainable learning model for human interaction detection and complete intelligent human interaction detection. In our proposed model, the interaction prediction score is composed of the human score, object score and action prediction score.

[0055] In traditional object detection models, if the IOU between a candidate box and the true label meets the threshold and has a high foreground probability, then the object will be given a high confidence. However, in the task of human interaction detection, the model should give high confidence to objects with higher interaction scores. Therefore, in order to better achieve this goal, based on the structure of Faster-RCNN, HO-RPN (human-object region proposal network) is used in this embodiment to replace the original RPN network, that is, human features are used to filter objects that do not interact.

[0056] The common idea of ​​contrastive learning is to construct positive and negative samples for each sample, and improve the model feature extraction capability by shortening the distance between positive sample features and extending the distance between negative sample features. Representing different category features with a large difference in the feature space can make the classifier better distinguish between the same and different categories. Therefore, in the model of this embodiment, the contrastive clustering method is used to increase the difference between different features and reduce the difference between the same features.

[0057] Furthermore, the final detection model is obtained, specifically: the image data set in the person interaction detection task is input into the detection model for training. When the number of training iterations is greater than the first preset parameter I, the model is continued to be trained using the self-supervised comparative clustering method. The number of iterations is the second preset parameter A times. At the same time, the human, object and joint interaction features are learned, the clustering feature vector is maintained, and the clustering feature vector is saved in the feature library.

[0058] For each object category i∈K t and interaction category j∈H t , maintain a prototype vector β i and θ j , to prevent the characteristics of extreme samples from affecting the results during training. In this model, the prototype vector is the average feature vector of the category. As the data changes during training, the feature vector of each category will also change, so the prototype vector must be continuously updated during training. Establish a feature library to save the changing features. The specific process algorithm can be seen in Table 1.

[0059] Table 1 Specific process algorithm

[0060]

[0061] The contrastive learning method in this technical solution is different from the previous methods in the following aspects: (1) The features in the feature library are the category features of objects and the category features of interactions, which are composed of human, object, and action features, and are therefore richer. The two features not only improve the accuracy of detection, but also promote each other and strengthen the differences between different categories; (2) The features stored in the feature library are not only used to expand the differences between classes, but also used to save the model's recognition ability for old categories in the feature playback network to alleviate the problem of catastrophic forgetting. The contrastive learning method in previous work is usually only used to improve the detection effect of the model.

[0062] The contrast loss function is defined as:

[0063]

[0064] in,

[0065]

[0066] Where: f o Represents the characteristics of the object, β i represents the prototype vector of this type of object, and δ defines the similarity between similar and dissimilar items; Represents the interaction contrast loss, which is different from the object contrast loss The calculation method is the same as that of ; C represents the total number of object categories, M represents the total number of interaction categories, Dis(f o ,β i ) represents an arbitrary distance function.

[0067] Furthermore, when performing the initial task, no increment is required. Therefore, the final detection model is first used to detect people and objects, and people and objects are combined in pairs to determine whether they interact. HO-RPN is used to filter objects without interaction, and the interaction category is predicted by the interaction category classifier.

[0068] Incremental learning aims to simultaneously learn new categories of knowledge and preserve the knowledge of old categories that have been learned. The main difficulty lies in alleviating the catastrophic forgetting problem and balancing the stability and generalization ability of the model. Based on deep learning, current incremental learning models can be simply divided into three categories: the first is the method based on the playback strategy, that is, saving a small part of the old samples in the memory, and then using these samples to fine-tune the model; the second is the method based on regularization, whose main idea is to expand the loss function to retain the weight parameters of the old categories; the last is the structure-based method, the core idea is to learn the relevant knowledge of the new category by changing the network structure, such as parameter masking, network pruning, etc.

[0069] In addition to good detection capabilities for new categories of people, objects, and actions, a necessary condition for a human interaction detection model with continuous learning capabilities is good recognition capabilities for new and old categories. Saving some old samples and training a new model together with new samples helps the model preserve the knowledge learned on old categories. This embodiment proposes a feature replay network (Feature Replay Network), and then obtains a sustainable learning model for human interaction detection, which specifically includes the following steps: design a feature replay network, obtain a set of old category samples and a high-dimensional clustering feature vector, and use it as a sample set to reproduce the effect of the model trained on all data; after each task, use the data in the sample set to fine-tune the model so that the fine-tuned model can extract the significant features of the old category samples, where the number of single-category samples of the old category saved in each task does not exceed N. ex indivual.

[0070] like Figure 3 As shown in the figure, the designed feature playback network includes two decoders, two classifiers and a backbone network; the model is fine-tuned, specifically including the following steps:

[0071] Use the decoder to restore the object clustering feature F o and the interactive clustering feature F hoi , whose corresponding category is c o and c hoi ;

[0072] The object clustering features and interaction clustering features are restored to the real image size and sent to the backbone network to re-extract features, and new object clustering features F o ' and interactive clustering feature F hoi ';

[0073] Reconstruct based on the re-extracted features to obtain the restored image, and send the restored image to the object category classifier and the interaction category classifier respectively to obtain c o ' and c hoi ';

[0074] Fine-tune the model until F o and F o 'Consistent, F hoi and F hoi 'Consistent, c o =c o '、c hoi =c hoi ' and complete fine-tuning to obtain a sustainable learning model for human interaction detection.

[0075] The loss function of the feature replay network is defined as follows:

[0076] LFRN =L o_cls +L hoi_cls +L o_feature +L hoi_feature ;

[0077] in,

[0078] L o_feature =D(F′ o ,F o );

[0079] L hoi_feature =D(F′ hoi ,F hoi );

[0080] Where D is the distance function. Minimize L FRN This allows the model to maintain good old category detection capabilities.

[0081] This embodiment also provides a computer storable medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for intelligent detection of human interaction are implemented.

[0082] Example 2

[0083] The embodiment of the present invention provides a human interaction intelligent detection system capable of continuous learning, including:

[0084] An acquisition module, used for acquiring an image dataset and dividing the image dataset into different human interaction detection tasks;

[0085] A construction module is used to build a detection model for human interaction detection tasks;

[0086] The training module is used to train the detection model to obtain a pre-trained detection model; then introduce contrastive learning and continue training to obtain the final detection model;

[0087] The detection module detects people and objects in the image and predicts the interaction category by using the final detection model;

[0088] The sustainable learning module is used to incrementally learn new category knowledge after each character interaction detection task is completed, and use the saved old category samples and feature fine-tuning models to obtain a sustainable learning model for character interaction detection and complete intelligent character interaction detection.

[0089] Next, in order to verify the method proposed in this invention, experiments are carried out on the HICO-DET and HOI-W datasets. The two datasets are divided into 4 tasks and 2 tasks respectively. The classification of new samples in each task is shown in Table 2 and Table 3. The task number is {T1, T2, …, T t}, when it comes to task T k When {T i : All categories in i<k} are old categories, T k The categories in are new categories.

[0090] Table 2 HICO-DET task composition

[0091]

[0092] Table 3 HOI-W task composition

[0093]

[0094] As for the evaluation index, we made some modifications to the traditional human interaction detection evaluation index mAP to better meet our task requirements. The new evaluation index is as follows:

[0095]

[0096] Among them, i represents the current task number, represents the raw mAP of the i-th task.

[0097] Table 4 shows the results of the continuous learning problem of human interaction detection for HICO-DET, and Table 5 shows the results of the continuous learning problem of human interaction detection for HOI-W.

[0098] Table 4 Results of continuous learning problem for character interaction detection of HICO-DET

[0099]

[0100] Table 5 Results of continuous learning problem for human interaction detection of HOI-W

[0101]

[0102] The above experimental results show the effectiveness of the model in the present invention in the continuous learning task of human interaction detection.

[0103] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0104] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sustainable learning method for intelligent detection of human interaction, characterized in that: The following steps are involved: Acquire an image dataset, and divide the image dataset into different human interaction detection tasks; Constructing a detection model for the task of human interaction detection, training the detection model, and detecting people and objects in the image, and when the number of training iterations reaches a first preset parameter I, the pre-training ends, and a pre-trained detection model is obtained; Using the pre-trained detection model, introducing contrastive learning to continue training the second preset parameter A times to obtain a final detection model, and using an interaction category classifier to predict the interaction category; After each person interaction detection task is completed, new category knowledge is incrementally learned, and the model is fine-tuned using the saved old category samples and features to obtain a sustainable learning model for person interaction detection and complete intelligent person interaction detection. Construct a detection model for the human interaction detection task, specifically: Based on the structure of Faster-RCNN, the RPN network is replaced with HO-RPN; The final detection model is obtained as follows: The image data set in the human interaction detection task is input into the detection model for training. When the number of training iterations is greater than the first preset parameter I, the model is continued to be trained using the self-supervised contrast clustering method, and the number of iterations is the second preset parameter A times. At the same time, the human, object and joint interaction features are learned, the cluster feature vector is maintained, and the cluster feature vector is saved in the feature library; The sustainable learning model for human interaction detection is obtained, which specifically includes the following steps: Design a feature replay network to obtain old category sample sets and high-dimensional clustering feature vectors as sample sets; After each task is completed, the model is fine-tuned using the data in the sample set so that the fine-tuned model can extract significant features of the old category samples; The designed feature playback network includes two decoders, two classifiers and a backbone network. The model is fine-tuned, specifically including the following steps: Use the decoder to restore the object clustering feature F ο and the interactive clustering feature F hoi , whose corresponding category is c ο and c hoi ; The object clustering features and interaction clustering features are restored to the real image size and sent to the backbone network to re-extract features, and new object clustering features F ο ' and interactive clustering feature F hoi '; Reconstruct based on the re-extracted features to obtain the restored image, and send the restored image to the object category classifier and the interaction category classifier respectively to obtain c ο ' and c hoi '; Fine-tune the model until F ο and F ο 'Consistent, F hoi and F hoi 'Consistent, c ο =c ο '、c hoi =c hoi ' and complete fine-tuning to obtain a sustainable learning model for human interaction detection.

2. The method for detecting human interaction with sustainable learning according to claim 1, characterized in that: The features in the feature library are respectively the category features of objects and the category features of interactions, and are composed of the features of people, objects, and actions.

3. The method for detecting human interaction with sustainable learning according to claim 1, characterized in that: The contrast clustering method specifically includes: increasing the difference of heterogeneous features and reducing the difference of homogeneous features, so that different categories of features are represented by representations with greater differences in the feature space; For each object category i∈K t and interaction category j∈H t , maintain a prototype vector β i and θ j , the contrast loss function is defined as: in, Where: f ο Represents the characteristics of the object, β i represents the prototype vector of this type of object, δ defines the similarity between similar and dissimilar items; l(f i ,θ j ) represents the interaction contrast loss, which is different from the object contrast loss l(f ο ,β i ) is calculated in the same way; C represents the total number of object categories, M represents the total number of interaction categories, Dis(f ο ,β i ) represents an arbitrary distance function.

4. The method for detecting human interaction with sustainable learning according to claim 1, characterized in that: The predicted interaction categories are specifically: Detecting people and objects using the final detection model, combining people and objects in pairs, and determining whether they interact; HO-RPN is used to filter objects without interaction, and the interaction category classifier is used to predict the interaction category.

5. A system for implementing the method for intelligent detection of human interaction for sustainable learning as claimed in any one of claims 1 to 4, characterized in that: include: An acquisition module, used for acquiring an image data set and dividing the image data set into different human interaction detection tasks; A construction module is used to build a detection model for human interaction detection tasks; A training module, used to train the detection model to obtain a pre-trained detection model; Then, contrastive learning is introduced and training is continued to obtain the final detection model; A detection module detects people and objects in the image and predicts interaction categories by using the final detection model; The sustainable learning module is used to incrementally learn new category knowledge after each character interaction detection task is completed, and use the saved old category samples and feature fine-tuning models to obtain a sustainable learning model for character interaction detection and complete intelligent character interaction detection.

6. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for intelligent detection of human interaction as described in any one of claims 1 to 4 are implemented.

Images