A method and device for recognizing human interactive behavior based on interpretable visual cues

Abstract

The present application provides a kind of based on explainable visual prompt human interaction behavior identification method and device, comprising: generating human interaction behavior identification dataset based on the multiple pictures containing human interaction behavior obtained;Image in data set is input into feature extraction network and gets whole map feature;Whole map feature and visual prompt added for image are input into human body object detector, and human body visual feature, object visual feature, human body visual prompt and object visual prompt are obtained;These features are input into human interaction behavior decoder and obtain interaction behavior feature;Image is detected based on pre-trained teacher network model, and human body posture feature, object feature and interaction area feature are obtained as teacher feature, human body visual prompt, object visual prompt and interaction behavior feature are used as student feature, the parameters of distillation identification network are distilled, and the human interaction behavior identification network after training is obtained.Based on the network, accurate human interaction behavior identification can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a person interaction behavior recognition method and device based on an interpretable visual cue. BACKGROUND

[0002] Person interaction behavior recognition is one of the hot research topics in computer vision and an important research direction of visual-based pattern recognition application. Due to the variety of person interaction behaviors, the data set can only cover some common interaction behaviors, so the person interaction behavior recognition model needs to have strong generalization ability. Many person interaction behavior recognition technologies use graphic-text pre-training models to help training, but the graphic-text pre-training model focuses on the whole image area, and usually a picture contains more than one interaction behavior, so whole image detection is not conducive to the recognition of multiple interaction behaviors, and if only the interaction area is recognized, the background information will be lost and the recognition accuracy will be reduced. SUMMARY

[0003] The present application provides a person interaction behavior recognition method and device based on an interpretable visual cue, to solve the defect of low recognition accuracy of existing person interaction behavior recognition technology, realize the use of visual cues of the target to be detected to guide the model to find specific targets, extract more critical picture features, and combine human, object and interaction behavior features to obtain more accurate recognition results. The technical solutions of the present application are as follows:

[0004] In a first aspect, the present application provides a person interaction behavior recognition method based on an interpretable visual cue, comprising:

[0005] Obtaining a plurality of pictures containing person interaction behaviors and a person interaction behavior recognition network, generating a person interaction behavior recognition data set based on the plurality of images containing person interaction behaviors; wherein the person interaction behavior recognition network comprises a feature extraction network, a human object detector and a person interaction behavior decoder;

[0006] Inputting the images in the person interaction behavior recognition data set into the feature extraction network to obtain whole image features;

[0007] Adding visual cues to the images, inputting the visual cues and the whole image features into the human object detector to obtain human visual features, object visual features, human visual cues and object visual cues;

[0008] Inputting the human visual features, the object visual features, the human visual cues and the object visual cues into the person interaction behavior decoder to obtain interaction behavior features;

[0009] Detecting the images based on a pre-trained teacher network model to obtain human pose features, object features and interaction region features;

[0010] distill parameters of the character interaction behavior recognition network, to obtain a trained character interaction behavior recognition network, by taking the human body posture feature, the object feature and the interaction region feature as teacher features, and taking the human body visual cue, the object visual cue and the interaction behavior feature as student features;

[0011] performing character interaction behavior recognition based on the trained character interaction behavior recognition network.

[0012] Optionally, the human body object detector comprises a visual encoder and an instance decoder.

[0013] the human body visual feature, the object visual feature, the human body visual cue and the object visual cue, comprises:

[0014] using position encoding to obtain position information of the human body and the object in the image;

[0015] obtaining a visual cue vector, fusing the global feature and the position information, and splicing the fused feature and the visual cue vector to obtain a fused input feature;

[0016] inputting the fused input feature into the visual encoder to obtain an encoded feature;

[0017] obtaining a randomly initialized human body query vector and an object query vector, inputting the encoded feature, the human body query vector and the object query vector into the instance decoder to obtain the human body visual feature, the object visual feature, the human body visual cue and the object visual cue.

[0018] Optionally, the encoded feature comprises a picture feature; inputting the human body visual feature, the object visual feature, the human body visual cue and the object visual cue into the character interaction behavior decoder to obtain an interaction behavior feature, comprises:

[0019] constructing an interaction query vector according to the human body visual feature and the object visual feature;

[0020] inputting the interaction query vector, the human body visual cue, the object visual cue and the picture feature into the character interaction behavior decoder to obtain an interaction behavior feature.

[0021] Optionally, the teacher network model comprises a human body key point detector, an object detector and an image encoder of a graph-text pre-training model.

[0022] the human body posture feature, the object feature and the interaction region feature, comprises:

[0023] segmenting the image to obtain a human region image, an object region image and an interaction region image;

[0024] inputting the human region image into the human key point detector to obtain a feature map about human key points;

[0025] inputting the feature map about human key points into a human conversion network to obtain a human pose feature;

[0026] inputting the object region image into the object detector to obtain an object feature;

[0027] inputting the interaction region image into the image encoder to obtain an interaction region feature.

[0028] Optionally, the character interaction behavior recognition network further comprises a text encoder; the human pose feature, the object feature and the interaction region feature are taken as teacher features, and the human visual cue, the object visual cue and the interaction behavior feature are taken as student features, parameters of the character interaction behavior recognition network are distilled to obtain a trained character interaction behavior recognition network, comprising:

[0029] the human pose feature is taken as a teacher feature, and the human visual cue is taken as a student feature, and a first distillation loss is calculated using KL divergence;

[0030] the object feature is taken as a teacher feature, and the object visual cue is taken as a student feature, and a second distillation loss is calculated using mean square error loss;

[0031] a third distillation loss is calculated using focal loss according to the interaction behavior feature and the interaction region feature;

[0032] parameters of the character interaction behavior recognition network are updated according to the first distillation loss, the second distillation loss and the third distillation loss to obtain a trained character interaction behavior recognition network.

[0033] Optionally, the character interaction behavior recognition network further comprises a plurality of classifiers; an output result of the character interaction behavior recognition network is determined by the following way:

[0034] the human visual cue and the human visual feature are spliced to obtain a feature, the object visual cue and the object visual feature are spliced to obtain a feature, and the interaction behavior feature is input into a corresponding classifier to obtain an action score, an object category score and an interaction behavior score respectively;

[0035] a supervised score of each category is calculated according to the action score, the object category score and the interaction behavior score;

[0036] The class with the highest supervision score is taken as the output result.

[0037] In a second aspect, the present invention further provides a device for identifying human interaction behaviors based on interpretable visual cues, comprising the following modules:

[0038] A data acquisition module is configured to acquire a plurality of images containing human interaction behaviors and a human interaction behavior recognition network, and generate a human interaction behavior recognition dataset based on the plurality of images containing human interaction behaviors; wherein the human interaction behavior recognition network includes a feature extraction network, a human object detector, and a human interaction behavior decoder;

[0039] A feature extraction module, configured to input the images in the human interaction behavior recognition dataset into a feature extraction network to obtain full-image features;

[0040] A first detection module is configured to add visual cues to the image, input the visual cues and the full image features into a human object detector, and obtain human visual features, object visual features, human visual cues, and object visual cues;

[0041] a behavior decoding module, configured to input the human visual features, the object visual features, the human visual cues, and the object visual cues into the human interaction behavior decoder to obtain interaction behavior features;

[0042] The second detection module is used to detect images based on the pre-trained teacher network model to obtain human posture features, object features and interaction area features;

[0043] a model training module, configured to use the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, to distill the parameters of the human interaction behavior recognition network to obtain a trained human interaction behavior recognition network;

[0044] The behavior recognition module is used to perform human interaction behavior recognition based on the trained human interaction behavior recognition network.

[0045] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying human interaction behaviors based on interpretable visual cues as described in the first aspect above is implemented.

[0046] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying human interaction behaviors based on interpretable visual cues as described in the first aspect above.

[0047] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the method for recognizing human interactive behavior based on interpretable visual cues according to the first aspect above.

[0048] Based on the above technical solution, the present application has the following beneficial effects compared with the prior art:

[0049] The method and device for recognizing human interactive behavior based on interpretable visual cues provided by the present application can learn different regions in the picture, i.e., the human body region and the object region, through the designed visual cues, thereby playing a feature enhancement role, and the provided prompt information is more accurate and effective. The training visual cues are mainly used, and the human body object detector and the human interactive behavior decoder are combined to recognize human interactive behavior. The picture features and the visual cues are spliced, the human body, the object, and the interactive region in the picture are recognized, the knowledge of the pre-trained teacher network model is learned in a distillation manner, and more accurate target information is obtained. The human interactive behavior recognition network trained by the method can use the visual cues of the target to be detected to guide the attention of the model to find a specific target, so as to extract more critical picture features, and then combine the human body, the object, and the interactive behavior features to obtain more accurate recognition results. At the same time, the visual cues can help the model to learn more general features, thereby improving the generalization ability of the model in different data sets or scenes.

[0050] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description, claims, and drawings.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 is a flowchart of the method for recognizing human interactive behavior based on interpretable visual cues provided by the present application.

[0054] Figure 2It is a structural diagram of the character interaction behavior recognition network provided by the present invention.

[0055] Figure 3 It is a structural schematic diagram of a device for identifying human interaction behavior based on interpretable visual cues provided by the present invention.

[0056] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] The following combination Figure 1-Figure 3 The present invention describes a method and device for identifying human interaction behaviors based on interpretable visual cues.

[0059] Reference Figure 1 As shown in Figure 1, this method for identifying interactive behaviors based on interpretable visual cues combines deep learning, feature extraction, and knowledge distillation techniques to improve the accuracy and interpretability of interactive behaviors. The method includes the following:

[0060] Step S110: Acquire multiple images containing human interaction behaviors and a human interaction behavior recognition network, and generate a human interaction behavior recognition dataset based on the multiple images containing human interaction behaviors.

[0061] Collect images of people interacting with each other from various sources (e.g., public datasets, self-taken photos, etc.). These images should cover different scenes, lighting conditions, angles, and types of interactions to ensure the diversity and richness of the dataset.

[0062] Collected images are annotated with bounding boxes for people, key points on people, bounding boxes for objects, bounding boxes for people's interactions, text descriptions of interactions, and interaction types. These annotations will be used to train and optimize the network for identifying people's interactions. Validation and test sets will also be constructed to evaluate the network's performance. Specifically, use rectangular or polygonal boxes to annotate the human body regions in the images. Ensure that the human body regions closely follow the human body contours and are neither too large nor too small. Similarly, use rectangular or polygonal boxes to annotate the object regions in the images. Ensure that the object regions accurately reflect the object's position and shape. Based on the interaction between the person and the object, use an appropriate annotation method (such as a rectangular box, polygonal box, or free-form box) to annotate the interaction region. This region should encompass the person and the object they are interacting with, and reflect their relative positions and relationships. Provide a concise and clear text description for each interaction in the image. The description should include the type of interaction (e.g., handshake, hug, kick a ball), the identity or category of the person and object (e.g., person, ball, chair), and the relationship between them (e.g., person holding a ball, person sitting on a chair).

[0063] The annotated images and their corresponding annotation information (human body boxes, object boxes, human interaction boxes, and textual descriptions of these interactions) are organized to form a structured dataset, the aforementioned human interaction behavior recognition dataset. The dataset is divided into training, validation, and test sets. The training set is used to train the model, the validation set is used to adjust model parameters and select the optimal model, and the test set is used to evaluate model performance. To improve the model's generalization, the dataset can be augmented with processing such as rotation, scaling, flipping, and cropping.

[0064] The aforementioned human interaction behavior recognition network primarily consists of the following components: a feature extraction network, a human object detector, a human interaction behavior decoder, a pre-trained image-text model, and multiple classifiers. The feature extraction network extracts global image features; the human object detector detects people and objects in the image and extracts their visual features; and the human interaction behavior decoder combines image features with visual cues of people and objects to infer interaction behaviors.

[0065] The feature extraction network described above is responsible for extracting useful feature information from the input image. These features include color, texture, shape, and higher-level semantic information. This feature extraction network can employ deep learning models such as convolutional neural networks (CNNs). These models, trained on large numbers of images, can automatically learn and extract key features from images.

[0066] The human interaction behavior decoder described above is responsible for inferring and identifying interactions between people and objects based on human and object detection information. This decoder can employ structures such as graph neural networks (GNNs), recurrent neural networks (RNNs), or Transformers to capture the spatiotemporal relationships and semantic connections between people and objects. The decoded output includes key information such as the type of interaction, human posture, and object attributes.

[0067] The text encoder described above processes textual information associated with the image, such as descriptions, tags, or annotations. By combining this textual information with image features, the network can more accurately understand and identify the interactions between people in the image. The text encoder can utilize techniques from natural language processing (NLP), such as word embeddings, LSTMs, or Transformers.

[0068] The aforementioned image-text pre-trained models are deep learning models that can process both image and text data. For example, they can be contrastive language-image pre-training (CLIP) models. They consist of an image encoder and a text encoder, which extract features from images and text. The text encoder converts the input text data into high-dimensional feature vectors that capture the semantic and syntactic information in the text.

[0069] After generating a dataset for identifying human interaction behaviors, the textual representations of these interactions are converted from triples of <person, object, interaction> into complete sentences. The text encoder of the pre-trained image-text model extracts textual features to help initialize the weight parameters of the human interaction behavior classifier. The triple of <person, object, interaction> is a structured textual representation, where "person" represents the subject participating in the interaction, "object" represents the object of the interaction, and "interaction" represents the behavioral relationship between the subject and the object. For a given triple of <person, object, interaction>, converting it into a complete sentence involves combining the three elements in natural language. For example, given the triple of <Xiaoming, mobile phone, making a phone call>, it can be converted into the complete sentence "Xiaoming is making a phone call on his mobile phone." Specifically, the converted complete sentence is input into the text encoder of the pre-trained image-text model. The text encoder encodes the input text and generates corresponding feature vectors. These feature vectors are then used in the subsequent human interaction behavior classification task.

[0070] Step S120: Input the image in the human interaction behavior recognition dataset into a feature extraction network to obtain full-image features.

[0071] Images from the human interaction behavior recognition dataset are fed into a feature extraction network. This network can be a deep convolutional neural network (CNN), which automatically learns feature representations in images. The output of the feature extraction network is full-image features, which contain global information about the image, such as color, texture, and shape.

[0072] Step S130 , adding visual cues to the image, inputting the visual cues and the full image features into a human object detector to obtain human visual features, object visual features, human visual cues, and object visual cues.

[0073] The visual cue module adds visual cues to the image. These cues can be keypoints, bounding boxes, or other visual elements that help identify behavior. These visual cues, along with full-image features, are fed into the human object detector. The human object detector receives the output of the feature extraction network and locates and identifies people and related objects in the image, generating human visual features, object visual features, human visual cues, and object visual cues. These features contain richer semantic and spatial structural information, facilitating subsequent decoding of human interaction behaviors.

[0074] Step S140: input the human visual features, the object visual features, the human visual cues, and the object visual cues into the human interaction behavior decoder to obtain interaction behavior features.

[0075] The human interaction behavior decoder uses these features to infer and identify interactions between people and objects. The human interaction behavior decoder can be a graph neural network (GNN), recurrent neural network (RNN), or Transformer architecture to capture the spatiotemporal relationships and semantic connections between people and objects. The human interaction behavior decoder outputs interaction behavior features, which contain key information about the interaction between people and objects, such as the interaction type.

[0076] Step S150: Detect the image based on the pre-trained teacher network model to obtain human posture features, object features, and interaction area features.

[0077] Use the pre-trained teacher network model to detect the image. The teacher network model is a more complex and accurate model that has been trained with a large amount of data and has high recognition performance.

[0078] The output of the teacher network model is human posture features, object features, and interaction area features. These features serve as teacher features with high accuracy and reliability and are used in the subsequent knowledge distillation process.

[0079] Step S160: Using the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, the parameters of the human interaction behavior recognition network are distilled to obtain a trained human interaction behavior recognition network.

[0080] Visual cues of people, objects, and interactive behaviors are used as student features. These features are extracted from the student network (the person interaction behavior recognition network being trained). Through knowledge distillation, the teacher features are used to guide the optimization of the student features. Knowledge distillation is a transfer learning technique that improves the student's performance by transferring knowledge from the teacher network to the student network. During the knowledge distillation process, one or more loss functions can be defined to measure the difference between the student and teacher features, and the backpropagation algorithm is used to optimize the student network's parameters. After multiple iterations of training, the performance of the student network will gradually improve, ultimately resulting in a trained person interaction behavior recognition network.

[0081] Step S170: performing human interaction behavior recognition based on the trained human interaction behavior recognition network.

[0082] The trained human interaction behavior recognition network is used to identify human interaction behaviors in new images. The new image is fed into the network, and after undergoing feature extraction, visual cue and feature fusion, and interaction behavior decoding, the network outputs a recognition result. The recognition result includes the type of interaction behavior (such as a handshake, hug, or kick a ball), human posture (such as joint location and posture angle), and object information (such as object category and location). This information can be used for subsequent applications and analysis.

[0083] Most existing human interaction recognition technologies focus on identifying object types and human interaction patterns in images. However, human posture plays a crucial role in identifying interactive behaviors. Furthermore, existing human interaction recognition technologies only utilize textual cues to provide additional semantic information to help the model better understand the context, but fail to leverage visual cues to enhance the model's understanding of images.

[0084] In response to the above-mentioned defects, the present invention enhances the network's perception of human interaction behaviors by introducing interpretable visual cues. The introduction of the teacher network model and the application of the knowledge distillation method further improve the network's recognition accuracy. The addition of visual cues enables the network to give clearer explanations when identifying behaviors, thereby improving the credibility of the results. By analyzing the visual cues and feature extraction process, we can gain an in-depth understanding of how the network makes recognition decisions. The trained human interaction behavior recognition network can adapt to different scenes and lighting conditions and has strong generalization capabilities. The application of this method helps to promote the development of human-computer interaction technology, especially in the fields of intelligent monitoring, smart home and intelligent transportation. By achieving more accurate and reliable human interaction behavior recognition, the intelligence level and user experience in these fields can be improved. Moreover, the present invention makes full use of the information of the pre-trained teacher network model to achieve more accurate human posture recognition and object category detection with a lighter detection module, thereby improving the model reasoning speed.

[0085] The aforementioned human interaction behavior recognition network includes multiple classifiers: a human pose classifier, an object classifier, and a human interaction behavior classifier. The human pose classifier is used to identify different human poses. Based on extracted visual features, it classifies human poses in an image into predefined pose categories. The object classifier is used to identify object categories in an image. Based on extracted visual features, it classifies objects in an input image into predefined object categories. The human interaction behavior classifier is a deep learning model used to identify and understand interactions between people and objects. The human interaction behavior classifier aims to identify and classify interactions between people or between people and objects by analyzing information in images or videos. It extracts key information from complex visual scenes, such as human poses, object attributes, and their spatial and temporal relationships, to determine the specific interaction behavior category. This classifier outputs a predicted score or probability distribution for the interaction behavior, thereby determining the most likely interaction behavior category.

[0086] Each of these classifiers receives the outputs of the instance decoder and the human interaction behavior decoder and uses a fully connected layer, softmax function, and other structures to make classification decisions. The human pose classifier and object classifier output predictions for human pose and object category, respectively, while the human interaction behavior classifier outputs a predicted score or probability distribution for interaction behavior. Based on the classifier's predictions, the network outputs recognition results for human pose, object category, and interaction behavior.

[0087] Before step S120, the weight parameters of the person interaction behavior classifier need to be initialized. The feature vector extracted by the text encoder of the image-text pre-training model can be used as a reference for initializing the weight parameters. Specifically, the weight parameters of the text encoder can be used as part of the initial weight of the classifier, or the knowledge of the text encoder can be transferred to the classifier in some way (such as transfer learning). The initialization process is: receiving a <person, object, interaction behavior> triple as input, converting the triple into a complete sentence. The text encoder of the image-text pre-training model is used to extract the feature vector of the converted sentence. The extracted feature vector is used to assist the person interaction behavior classifier to initialize the weight parameters.

[0088] The text feature is obtained by first using the text encoder of the image-text pre-training model to encode the image annotation sentence. The person interaction behavior classifier and the object classifier are initialized using the text feature. This not only makes full use of the prompt information carried by the text description, but also makes full use of the pre-training knowledge of the image-text pre-training model. The classifier contains a large amount of useful information at the beginning of training, which helps the model converge to the optimal solution faster during training.

[0089] The human object detector is responsible for locating and identifying human bodies and related objects in images. In an optional embodiment, the detector is further divided into a visual encoder and an instance decoder. The visual encoder in the human object detector is responsible for further extracting and encoding image features, while the instance decoder is responsible for decoding instance information of human bodies and objects from the encoded features.

[0090] Specifically, the visual encoder receives the feature map from the feature extraction network and further processes and encodes it. It can use deeper convolutional layers or recurrent neural networks (RNN) and other structures to capture the spatial and temporal information in the image. The encoded feature map will contain richer semantic information and spatial structure information. The instance decoder is responsible for decoding instance information of human bodies and objects from the encoded feature map. This includes the joint points of human bodies, the bounding boxes of objects, and their class labels, etc. The instance decoder can use techniques such as the region convolutional neural network (R-CNN) series, the YOLO series, or attention mechanism-based decoders.

[0091] The step S130 of inputting the visual cues and the full-image features into the human object detector to obtain human visual features, object visual features, human visual cues, and object visual cues includes:

[0092] S1301, obtaining the position information of human bodies and objects in the image using position encoding.

[0093] Position encoding is used to obtain the position information of people and objects in the image. Position encoding can be a method of converting spatial positions into learnable vector representations, allowing the model to perceive the positional relationship between different elements in the image.

[0094] S1302, obtain visual cue vector and integrate full image features With the location information , the fused features and visual cue vector Splicing to obtain fused input features .

[0095] The above visual cue module consists of two parts: human key point detection module and object detection modules Human key point detection module Focus on learning human key point detection model knowledge, object detection module It focuses on learning object detection model knowledge and using this module to add visual cues to images.

[0096] Human key point detection module Locate specific key points on the human body, such as joints (elbows, knees, shoulders, etc.) or facial landmarks (eyes, nose, mouth, etc.). Object Detection Module Identify objects in the image and mark their locations with boxes. Generate corresponding visual cues based on the output results of the human keypoint detection module and the object detection module. For human keypoint detection, visual cues can include information such as the location of keypoints. For object detection, visual cues can include information such as the object's bounding box, category label, or confidence level. The generated human keypoint detection visual cues and object detection visual cues are fused into the original image. Different colors, lines, or icons can be used to represent different types of visual cues so that users can clearly distinguish them. Visual cues can be used to assist in understanding image content and performing image analysis.

[0097] The visual cue module described above extracts visual cues from the image. These cues include the locations of key points on the human body, bounding boxes of objects, or other salient features. These visual cues are converted into a vector, known as a visual cue vector. The full-image features (i.e., the overall feature representation of the image) are fused with the position information. This can be achieved through simple concatenation, addition, or more complex attention mechanisms. The fused features contain both the overall image information and the position information.

[0098] S1303: Input the fused input features into the visual encoder to obtain the encoded features. .

[0099] The fused input features are input into the visual encoder to extract the high-dimensional feature representation of the image. Contains deep information about the human body, objects, and the relationships between them in the image. Encoded features Include image features .

[0100] S1304, obtain a randomly initialized human query vector and an object query vector, and convert the encoded features, the human query vector and the object query vector Input the instance decoder to get the final visual clue , human visual characteristics and object visual features Visual cues Including human visual cues and object visual cues .

[0101] Obtain randomly initialized person query vectors and object query vectors. These query vectors are used by the decoder to generate initial representations of person and object instances. Input the encoded features, person query vectors, and object query vectors into the instance decoder. The instance decoder, which can be a Transformer decoder or a similar attention mechanism, generates image features, person visual cues, and object visual cues based on the query vectors and encoded features. The generated image features are an abstract representation of the entire image content. Person visual features focus on the position and pose of people in the image, while object visual features focus on the position and attributes of objects in the image.

[0102] Specifically, fusion of location information With full-image features , and the visual cue vector are concatenated and fed into the visual encoder in the human object detector to obtain the encoded features, which are then combined with the randomly initialized human query vector and the object query vector are fed into the instance decoder together. Specifically, the fused input features Expressed as:

[0103]

[0104] in, For splicing operations, Represents the full image features, Indicates location information. represents the visual cue vector.

[0105]

[0106] in, is the encoded feature, that is, the output result of the visual encoder, Represents the operation of the visual encoder.

[0107] The decoded features (i.e., the features output by the instance decoder) are expressed as:

[0108]

[0109]

[0110] in, represents the operation of the instance decoder, Represents the visual characteristics of the human body, and the visual characteristics of objects represent the visual characteristics of objects. Indicates human visual cues, Indicates a visual cue, Indicates visual cues for objects.

[0111] In the above step S130, the image is subjected to a feature extraction network (such as a ResNet network) to obtain serialized image features, and absolute position encoding is used to obtain the position information of the human body and objects in the image. The position information helps the encoder find more accurate human body and object areas and areas where interactive behaviors occur, so as to obtain more effective regional features.

[0112] By combining visual cues and full-image features, the present invention enables the model to more accurately locate people and objects in the image, reducing false detections and missed detections. The introduction of position encoding and visual cue vectors enables the model to process image elements of different sizes and positions, improving the generalization ability of the model. The fused input features contain rich information, making the encoded features more expressive, which is helpful for subsequent classification, detection or segmentation tasks. This process can handle human detection and object detection tasks simultaneously, realize multi-task learning, and improve the efficiency and practicality of the model. By extracting and visualizing visual cue features, the decision-making process of the model can be more easily understood, promoting the interpretability of the model.

[0113] In an optional embodiment, the step S140 of inputting the human visual features, the object visual features, the human visual cues, and the object visual cues into the human interaction behavior decoder to obtain the interaction behavior features includes:

[0114] S1401, according to the human visual characteristics and the visual features of the object Construct an interaction query vector.

[0115] First, an interaction query vector is constructed based on the visual features of the person and the object. This vector aims to capture key information about the interaction between the person and the object, such as the person's pose, object attributes, and the spatial relationship between them. This construction can be achieved through steps such as feature fusion, feature transformation, or an attention mechanism to effectively combine the visual cues of the person and the object to form a query vector that represents the interaction behavior.

[0116] The interactive query vector in the present invention as follows:

[0117]

[0118] S1402: Combine the interactive query vector, the human visual prompt, the object visual prompt, and the image feature The character interaction behavior decoder is input to obtain interaction behavior features.

[0119] The constructed interactive query vector , visual cues (Visual cues Including human visual cues and object visual cues ) and image features The decoder generates interactive behavior features through a series of calculations and transformations based on the interactive query vector and image features. These features represent the interaction between humans and objects at a deep level, and contain rich semantic and syntactic information. These features can be used in subsequent classification and recognition tasks to accurately understand and judge human interaction behaviors.

[0120]

[0121] in, Represents the operation of the character interaction behavior decoder, Indicates interactive behavior characteristics.

[0122] The present invention constructs an interaction query vector by combining the visual features of the human body and the visual features of objects. The decoder can more accurately capture the interaction information between the human body and the object, thereby improving the accuracy of interactive behavior recognition. The construction process of the interaction query vector takes into account the multiple characteristics of the human body and the object, enabling the model to handle interactive behaviors in different scenarios and conditions, enhancing the model's generalization ability. By visualizing the interaction query vector and the interactive behavior features, the model's decision-making process can be more easily understood, promoting the model's interpretability. The obtained interactive behavior features can be used as input for subsequent tasks, providing strong support for these tasks.

[0123] In an optional embodiment, the teacher network model in the above step S150 includes a human key point detector, an object detector and an image encoder.

[0124] The image is detected based on the pre-trained teacher network model described in step S150 to obtain human posture features, object features, and interaction area features, including:

[0125] S1501 , segmenting the image to obtain a human body region image, an object region image, and an interaction region image.

[0126] First, the input image is segmented to produce images of the human body, object, and interaction regions. This step forms the basis for subsequent feature extraction, ensuring the precise location and extraction of the different elements in the image (human body, object, and interaction region).

[0127] S1502: Input the human body region image into the human body key point detector to obtain a feature map of multiple key points of the human body.

[0128] The human body region image is fed into a human keypoint detector. This teacher network has been extensively trained and can accurately identify multiple keypoints (such as joints) on the human body. The human keypoint detector generates feature maps of these keypoints. These feature maps contain detailed information about the human pose, such as the position and orientation of the keypoints.

[0129] S1503: Input the feature map of multiple key points of the human body into a human body conversion network to obtain human body posture features.

[0130] These feature maps are fed into the human body transformation network for further processing and transformation, ultimately yielding human body posture features. These features represent a deep level of human body posture and can be used for subsequent analysis and recognition tasks.

[0131] Reference Figure 2As shown in Figure 1, the human body transformation network consists of convolutional layers, pooling layers, and linear layers. The input to the human body transformation network is feature maps of multiple key points on the human body. These feature maps are generated by a human key point detector and contain key information such as the position and orientation of the key points. The convolutional layer convolves the input feature maps with a series of convolution kernels to extract local features. These kernels have different sizes and strides, allowing them to capture feature information at different scales. After the convolution operation, an activation function (such as ReLU) is applied to increase the nonlinearity of the network, enabling it to learn more complex feature representations. The pooling layer downsamples the feature maps output by the convolutional layer to reduce their size and computational complexity. Pooling allows the network to retain the most important features while reducing redundancy and noise, improving the model's generalization ability. The linear layer (also known as a fully connected layer) receives the feature vector output by the pooling layer and performs a linear transformation using a weight matrix and a bias vector. This step maps the feature vector to the output space to generate the final human pose feature. After processing by the linear layer, the network outputs the human pose feature. These features are a deep representation of human posture and can be used for subsequent tasks of identifying and classifying human interaction behavior types.

[0132] S1504: Input the object region image into the object detector to obtain object features.

[0133] The object region image is fed into the object detector. This teacher network has also undergone extensive training and can accurately identify objects and their attributes in the image. The object detector processes the image to obtain object features. These features contain information such as the object's category, location, and size, and serve as the basis for subsequent object analysis and recognition.

[0134] S1505: Input the interaction area image into the image encoder to obtain interaction area features.

[0135] The interaction region image is fed into an image encoder. This deep learning model extracts visual features from an image and converts them into understandable representations. The image encoder processes the image to generate interaction region features. These features contain information about the interaction between the human body and the object, such as the type of interaction, and are crucial for subsequent analysis and identification of interaction behavior.

[0136] The present invention uses pre-trained teacher network models for feature extraction, which can make full use of the knowledge and experience learned by these models on large amounts of data, thereby improving the accuracy of feature extraction.

[0137] Because teacher network models have been trained on large-scale datasets, they possess strong generalization capabilities. This means they can handle different types of images and scenes and extract useful features. This process simultaneously handles multiple tasks, such as human pose estimation, object detection, and interaction behavior recognition, achieving multi-task learning. This helps the model better understand and process complex information in images. By extracting human pose features, object features, and interaction area features, the model's decision-making process is more easily understood. This helps explain the model's behavior and output results, improving its transparency and credibility. The extracted human pose features, object features, and interaction area features can be used as input for subsequent tasks, providing strong support for these tasks.

[0138] Specifically, the network described in the paper "ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation" can be used as a human key point detector. The human region image is input into the ViTPose model to obtain a feature map (Heat Map) of 17 key points of the human body. The human body transformation network then outputs the human pose features, which are specifically expressed as follows:

[0139]

[0140]

[0141] in, Represents a feature map of multiple key points on the human body, Represents the operation of the ViTPose model, represents the human body region image, Represents the human body posture characteristics, Represents the operation of the pooling layer, Represents the operation of the convolutional layer.

[0142] Human keypoint detection aims to locate key parts of the human body, such as the head, shoulders, elbows, and knees, in images or videos. The ViTPose method is a human keypoint detection technology based on the Vision Transformer (ViT) architecture. It leverages the Transformer's self-attention mechanism to capture long-range dependencies in images, thereby more accurately identifying the locations of keypoints on the human body. Compared to traditional convolutional neural network-based methods, ViTPose processes global information from the entire image sequence, better handling occlusions and complex pose changes. This improves the accuracy and robustness of keypoint detection, significantly contributing to human pose estimation.

[0143] The network described in the paper "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection" can be used as an object detector. The object area image is input into the GroundingDINO model, and the object features are output, which are specifically represented as follows:

[0144]

[0145] in, Represents the characteristics of an object, Represents the operation of the GroundingDINO model, Represents the object area image.

[0146] GroundingDINO is a self-supervised learning method for scene understanding in images. It combines visual perception and semantic reasoning to learn semantic representations of objects in images without labels. By introducing contrastive learning of visual and semantic embeddings, the method enables the model to align objects in images with their corresponding semantic categories, thereby achieving semantic understanding and object localization in image scenes. GroundingDINO can learn representations rich in semantic information from large-scale unlabeled image data, and has shown promising results in tasks such as object detection, scene understanding, and image classification.

[0147] The interaction region image can be fed into the image encoder described in the paper “Contrastive Language-Image Pre-Training” to obtain the following interaction region features:

[0148]

[0149] in, represents the interaction area characteristics, represents the operation of the image encoder, Represents the interactive area image.

[0150] In an optional embodiment, knowledge distillation technology is combined to improve the performance of the human interaction behavior recognition network through comparative learning of teacher features and student features, and joint optimization of multiple loss functions. The above step S160 uses the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features to distill the parameters of the human interaction behavior recognition network to obtain the trained human interaction behavior recognition network, including:

[0151] S1601: Use the human posture features as teacher features, use the human visual cues as student features, and use KL divergence to calculate the first distillation loss.

[0152] Human pose features extracted from the pre-trained teacher network model serve as high-quality teacher features, containing the precise location and pose information of key points of the human body. Meanwhile, human visual cues extracted from the human interaction behavior recognition network to be trained serve as student features to be improved.

[0153] KL divergence is an asymmetric measure of the difference between two probability distributions. In this step, the human posture features (considered as teacher distribution) and human visual cues (considered as a student distribution) is probabilized (e.g., by a softmax function), and then the KL divergence between them is calculated. This divergence value reflects the degree of difference between the student characteristics and the teacher characteristics, that is, the inadequacy of the student characteristics in imitating the teacher characteristics.

[0154] The calculated KL divergence is used as the first distillation loss. This loss function will guide the network to gradually reduce the difference between student features and teacher features during training, thereby improving the accuracy of student features. as follows:

[0155]

[0156] in, represents the KL divergence, 、 Represents the activation function.

[0157] S1602, the object features As a teacher feature, the object visual cue As the student feature, the second distillation loss is calculated using the mean squared error loss.

[0158] Similarly, the object features extracted from the teacher network model As teacher features, they contain the precise location and category information of objects in the image. Visual cues of objects extracted from the network to be trained serve as student features.

[0159] Mean squared error loss is a measure of the difference between two values. In this step, the object features are directly calculated (Teacher characteristics) and object visual cues The second distillation loss is obtained by calculating the mean squared error between the student features and the teacher features. This loss value reflects the accuracy gap between the student features and the teacher features.

[0160] The calculated mean square error is used as the second distillation loss. This loss function will guide the network to gradually reduce the numerical difference between the student features and the teacher features during the training process, thereby improving the accuracy of the student features. as follows:

[0161]

[0162] in, represents the mean square error, 、 Represents the activation function.

[0163] S1603: Calculate a third distillation loss using a focal loss according to the interactive behavior feature and the interactive area feature.

[0164] Focal loss is a loss function used to deal with class imbalance. It can pay more attention to samples that are difficult to classify during training, thereby improving the model's ability to recognize minority classes. The focal loss function is used to calculate the focal loss based on the interaction behavior characteristics and interaction area characteristics. The calculated focal loss is used as the third distillation loss. This loss function will guide the network to gradually reduce the misjudgment of difficult-to-classify samples during training, thereby improving the overall performance of the model. Third distillation loss The specific expressions are as follows:

[0165]

[0166] in, represents the third distillation loss, represents the focal loss function, Indicates the interactive behavior characteristics, Represents the characteristics of the interaction area.

[0167] The present invention also inputs the text representation of the image into a text encoder to obtain text features, which contain the semantic information of the interactive behavior in the image. The similarity is calculated with the text features output by the text encoder to obtain the interaction behavior category score, which is then used to supervise the person interaction behavior detection network so that the pre-trained CLIP model visual features and text features are in the same feature space, thereby allowing the visual features output by the person interaction behavior decoder to be pulled into the same feature space.

[0168]

[0169]

[0170] in, Represents text features, represents the operation of a text encoder, Represents a textual representation, Indicates the similarity between the interactive behavior features and the text features, represents the cosine similarity function, Indicates interactive behavior characteristics.

[0171] S1604: Update the parameters of the human interaction behavior recognition network according to the first distillation loss, the second distillation loss, and the third distillation loss to obtain a trained human interaction behavior recognition network.

[0172] The backpropagation algorithm is used to calculate the gradients of the network parameters based on the first, second, and third distillation losses. The parameters of the person interaction behavior recognition network are updated based on the calculated gradients. This process is completed through multiple iterations of training, with each iteration adjusting the network parameters based on the current loss value to gradually optimize network performance. As training progresses, the network loss value gradually decreases, and the model performance gradually improves. When the loss value reaches a preset threshold or stops decreasing significantly, the model is considered to have converged and the training process is complete.

[0173] The present invention uses the human visual cues extracted from step S140 as human posture student features, and the object visual cues extracted from step S140 as object category student features, to help the human object detector learn the rich knowledge and experience of the teacher model, and achieve detection accuracy close to that of the teacher model with fewer training parameters.

[0174] Using the CLIP model as a supervisory model, the textual description of the interactive behavior is fed into a text encoder to obtain text features. The similarity between the interactive behavior features and the text features is calculated to obtain the interactive behavior category score. Focal loss is used to supervise the recognition of interactive behaviors, helping the interactive behavior recognition module to better understand the correspondence between images and text. At the same time, the CLIP model, which performs self-supervised learning on a large amount of unlabeled data, has strong generalization capabilities. This generalization capability can be transferred to the supervised human interactive behavior recognition network, helping the interactive behavior recognition module to handle a wider range of and more complex input situations.

[0175] The present invention uses knowledge distillation technology to transfer the high-quality features of the teacher network to the student network, which can significantly improve the performance of the student network. The joint optimization of multiple loss functions enables the model to learn richer information during the training process, thereby enhancing the generalization ability of the model. The use of focal loss helps the model to be more effective in dealing with category imbalance problems and improves the model's recognition ability for minority categories. By combining visual and textual information, the model can more comprehensively understand the interactive behavior in the image and improve the accuracy and robustness of recognition. The introduction of text features enables the model to provide richer contextual information when identifying interactive behaviors, which helps to explain the model's decision-making process.

[0176] Moreover, the method of the present invention mainly trains visual cues, and combines human key point detectors, object detectors, and image-text pre-training models to perform human interaction behavior recognition. By splicing image features and visual cues, the human body, objects, and interaction areas in the image are recognized in conjunction with position encoding and query vectors. The knowledge of the large model (i.e., the above-mentioned human key point detector, object detector, and image-text pre-training model) is learned using a distillation method to obtain more accurate target information. The human interaction behavior recognition network trained by this method can use the visual cues of the target to be detected to guide the model's attention to find specific targets, so as to extract more critical image features, and then combine the human body, object, and interaction behavior features to obtain more accurate recognition results. At the same time, visual cues can help the model learn more general features, thereby improving the model's generalization ability in different data sets or scenarios.

[0177] During the training of the human interaction behavior recognition network, this paper introduces distillation losses (including the first, second, and third distillation losses described above) to update network parameters. During training, the distillation losses can be combined to form a total loss function. The total loss function is expressed as:

[0178] Total loss = classification loss + weight 1 × first distillation loss + weight 2 × second distillation loss + weight 3 × third distillation loss.

[0179] Among them, weight 1, weight 2, weight 3, etc. are hyperparameters determined according to specific tasks and network architecture, which are used to balance the contributions between different loss items.

[0180] Using the chain rule, the partial derivatives of the total loss function with respect to the weights and biases of each neuron in each layer of the network are calculated. The weights and biases of each layer of the neural network are updated using the gradient descent algorithm (or its variants, such as Adam and RMSprop). After each update, the forward error calculation process is performed to evaluate the model performance under the current network parameters. By repeatedly executing the forward error calculation and backward parameter update process, the neural network's loss function is gradually reduced. Parameter adjustment ceases when the loss function reaches convergence (i.e., the change is less than a preset threshold) or when the specified number of iterations is reached. At this point, the model parameters are considered to fully fit the current data, and the model performance is optimal.

[0181] In an optional embodiment, in order to more accurately identify the human posture, object category, and interaction behavior between them in an image, the network is designed to include multiple classifiers, namely the human posture classifier, object classifier, and human interaction behavior classifier, corresponding to human posture prediction, object category prediction, and interaction behavior recognition, respectively. The human interaction behavior recognition network performs recognition to obtain output results in the following manner:

[0182] S210, the human body visual prompt and the human visual characteristics The features obtained after splicing and the visual cues of the object Visual features of the object The concatenated features and the interactive behavior features are input into corresponding classifiers to obtain action scores, object category scores, and interactive behavior scores, respectively.

[0183] From the input image, the network first extracts human visual features, object visual features, and interactive behavior features. These features reflect the posture information of the human body in the image, the category information of the object, and the information of the interaction area between the human and the object.

[0184] The extracted human visual features Visual cues with the human body After splicing, it is input into the human posture classifier, which is trained to recognize different human postures and output the human posture prediction value. , including the human body frame and action score .

[0185]

[0186] in, represents the predicted value of human posture, represents the operation of the human pose classifier, Represents a splicing operation.

[0187] The visual features of the object Visual cues with objects After splicing, it is input into the object classifier, which can identify the object category in the image and output the object category prediction value , including object boxes and object category scores .

[0188]

[0189] in, represents the object category prediction value, represents the operation of the object classifier, Represents a splicing operation.

[0190] The interactive behavior features extracted from the network to be trained are used as the input of the interactive behavior classifier. These features contain the visual information of the interactive behavior between people and objects in the image. Input into the interaction behavior classifier, which can identify the interaction behavior between people and objects and output the interaction behavior score ,Right now Figure 2 in , specifically expressed as follows:

[0191]

[0192] in, represents the interaction behavior score, () represents the operation of the character interaction behavior classifier.

[0193] S220: Calculate the supervision score of each category based on the action score, object category score, and interaction behavior score.

[0194] Based on the action score, object category score, and interaction score, a supervision score is calculated for each category. This score comprehensively considers information about human pose, object category, and interaction behavior, and is used to assess the likelihood of interaction for each category in the image. Supervision scores can be calculated using weighted sums, products, or other combinations, depending on the network design and training objectives.

[0195]

[0196] in, represents the supervision score.

[0197] S230: The category with the highest supervision score is output as the result. This means that the network believes that this category best matches the interactive behavior of the people in the image.

[0198] The present application improves the accuracy of recognition by introducing multiple classifiers, which can respectively identify human poses, object categories and interaction behaviors. Each classifier focuses on its specific task and can capture more detailed information in the image. The use of multiple classifiers makes the network more robust to noise and changes in the image. Even if the prediction of a certain classifier deviates, the information of other classifiers can still be supplemented to improve the stability of the overall recognition. In addition to the final interaction behavior category, the network can also output human pose prediction values and object category prediction values. These information is very useful for understanding the specific context and details in the image and can provide users with more rich visual feedback.

[0199] The following describes the human interaction behavior recognition device based on interpretable visual cues provided by the present application. The human interaction behavior recognition device based on interpretable visual cues described below can be mutually corresponding with the human interaction behavior recognition method based on interpretable visual cues described above.

[0200] The human interaction behavior recognition device based on interpretable visual cues provided by the present application, as shown in Figure 3 includes:

[0201] The data acquisition module 310 is configured to acquire a plurality of pictures containing human interaction behaviors and a human interaction behavior recognition network, generate a human interaction behavior recognition dataset based on the plurality of pictures containing human interaction behaviors; wherein the human interaction behavior recognition network includes a feature extraction network, a human object detector and a human interaction behavior decoder.

[0202] The feature extraction module 320 is configured to input the images in the human interaction behavior recognition dataset into the feature extraction network to obtain full-image features.

[0203] The first detection module 330 is configured to add visual cues to the images, input the visual cues and the full-image features into the human object detector to obtain human visual features, object visual features, human visual cues and object visual cues.

[0204] The behavior decoding module 340 is configured to input the human visual features, the object visual features, the human visual cues and the object visual cues into the human interaction behavior decoder to obtain interaction behavior features.

[0205] The second detection module 350 is configured to detect the images based on a pre-trained teacher network model to obtain human pose features, object features and interaction behavior features.

[0206] A model training module 360 ​​is configured to use the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, to distill the parameters of the human interaction behavior recognition network to obtain a trained human interaction behavior recognition network;

[0207] The behavior recognition module 370 is used to perform human interaction behavior recognition based on the trained human interaction behavior recognition network.

[0208] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to execute the method for identifying human interaction behaviors based on interpretable visual cues.

[0209] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0210] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the human interaction behavior recognition method based on interpretable visual cues provided by the above methods.

[0211] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for identifying human interaction behaviors based on interpretable visual cues provided by the above methods.

[0212] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0213] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying human interaction behaviors based on interpretable visual cues, characterized in that: include: Acquire multiple images containing human interaction behaviors and a human interaction behavior recognition network, and generate a human interaction behavior recognition dataset based on the multiple images containing human interaction behaviors; wherein the human interaction behavior recognition network includes a feature extraction network, a human object detector, and a human interaction behavior decoder; Inputting the image in the human interaction behavior recognition dataset into a feature extraction network to obtain full image features; Adding visual cues to the image, inputting the visual cues and the full image features into a human object detector to obtain human visual features, object visual features, human visual cues, and object visual cues; Inputting the human visual features, the object visual features, the human visual cues, and the object visual cues into the character interaction behavior decoder to obtain interaction behavior features; The image is detected based on the pre-trained teacher network model to obtain human posture features, object features and interaction area features; Using the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, distilling the parameters of the human interaction behavior recognition network to obtain a trained human interaction behavior recognition network; Human interaction behavior recognition is performed based on the trained human interaction behavior recognition network.

2. The method for identifying human interaction behaviors based on interpretable visual cues according to claim 1, characterized in that: The human object detector includes a visual encoder and an instance decoder; The step of inputting the visual prompt and the full image features into a human object detector to obtain human visual features, object visual features, human visual prompts, and object visual prompts includes: Use position coding to obtain the position information of human bodies and objects in the image; Obtaining a visual cue vector, fusing the full image feature with the position information, and concatenating the fused feature with the visual cue vector to obtain a fused input feature; Inputting the fused input features into the visual encoder to obtain encoded features; Obtain a randomly initialized human query vector and an object query vector, input the encoded features, the human query vector, and the object query vector into the instance decoder, and obtain human visual features, object visual features, human visual cues, and object visual cues.

3. The method for identifying human interaction behaviors based on interpretable visual cues according to claim 2, characterized in that: The encoded features include picture features; Inputting the human visual features, the object visual features, the human visual cues, and the object visual cues into the character interaction behavior decoder to obtain interaction behavior features, including: constructing an interactive query vector according to the human visual features and the object visual features; The interaction query vector, the human visual prompt, the object visual prompt and the image feature are input into the human interaction behavior decoder to obtain interaction behavior features.

4. The method for identifying human interaction behaviors based on interpretable visual cues according to claim 1, characterized in that: The teacher network model includes a human key point detector, an object detector, and an image encoder of a picture-text pre-training model; The pre-trained teacher network model is used to detect images to obtain human posture features, object features, and interaction area features, including: Segment the image to obtain a human body region image, an object region image, and an interaction region image; Inputting the human body region image into the human body key point detector to obtain a feature map of multiple key points of the human body; Inputting the feature map of multiple key points of the human body into a human body conversion network to obtain human body posture features; Inputting the object region image into the object detector to obtain object features; The interaction area image is input into the image encoder to obtain interaction area features.

5. The method for identifying human interaction behaviors based on interpretable visual cues according to claim 1, characterized in that: The human interaction behavior recognition network further includes a text encoder; using the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, distilling the parameters of the human interaction behavior recognition network to obtain a trained human interaction behavior recognition network, including: The human posture feature is used as the teacher feature, the human visual clue is used as the student feature, and the KL divergence is used to calculate the first distillation loss; The object features are used as teacher features, the object visual cues are used as student features, and the mean square error loss is used to calculate the second distillation loss; Calculating a third distillation loss using a focal loss according to the interaction behavior feature and the interaction region feature; The parameters of the human interaction behavior recognition network are updated according to the first distillation loss, the second distillation loss, and the third distillation loss to obtain a trained human interaction behavior recognition network.

6. The method for identifying human interaction behaviors based on interpretable visual cues according to claim 1, characterized in that: The character interaction behavior recognition network further includes a plurality of classifiers; the output result of the character interaction behavior recognition network is determined by the following method: Input the features obtained by concatenating the human visual cue and the human visual features, the features obtained by concatenating the object visual cue and the object visual features, and the interactive behavior features into corresponding classifiers to respectively obtain an action score, an object category score, and an interactive behavior score; Calculate the supervision score of each category based on the action score, object category score and interaction behavior score; The class with the highest supervision score is taken as the output result.

7. A device for identifying interactive behaviors of people based on interpretable visual cues, characterized in that: include: A data acquisition module is configured to acquire a plurality of images containing human interaction behaviors and a human interaction behavior recognition network, and generate a human interaction behavior recognition dataset based on the plurality of images containing human interaction behaviors; wherein the human interaction behavior recognition network includes a feature extraction network, a human object detector, and a human interaction behavior decoder; A feature extraction module, configured to input the images in the human interaction behavior recognition dataset into a feature extraction network to obtain full-image features; A first detection module is configured to add visual cues to the image, input the visual cues and the full image features into a human object detector, and obtain human visual features, object visual features, human visual cues, and object visual cues; a behavior decoding module, configured to input the human visual features, the object visual features, the human visual cues, and the object visual cues into the human interaction behavior decoder to obtain interaction behavior features; The second detection module is used to detect images based on the pre-trained teacher network model to obtain human posture features, object features and interaction area features; a model training module, configured to use the human posture features, the object features, and the interaction area features as teacher features, and the human visual cues, the object visual cues, and the interaction behavior features as student features, to distill the parameters of the human interaction behavior recognition network to obtain a trained human interaction behavior recognition network; The behavior recognition module is used to perform human interaction behavior recognition based on the trained human interaction behavior recognition network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for identifying human interaction behaviors based on interpretable visual cues as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying human interaction behaviors based on interpretable visual cues as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying human interaction behaviors based on interpretable visual cues as described in any one of claims 1 to 6 is implemented.

Images