Method and device for identifying interactive characters based on multi-feature fusion of video data

Abstract

The present invention discloses a method and device for multi-feature fusion character interaction recognition based on video data, the method comprising: obtaining a key point set of an entity in the video data to obtain an initial geometric feature; inputting the initial geometric feature into a geometric feature encoder to obtain a target geometric feature; obtaining the relative distance and relative angle of each character pair in the entity to obtain an initial global feature; inputting the initial global feature into a global feature encoder to obtain a target global feature; obtaining a detection frame image set in the video data to obtain a visual feature; performing feature fusion processing on the target geometric feature, the target global feature and the visual feature to obtain a character interaction recognition result. The present invention can effectively improve the accuracy and robustness of character interaction recognition and can be widely used in the field of image recognition technology.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method and device for multi-feature fusion character interaction recognition based on video data. Background Art

[0002] The rise and development of HOI recognition related research is closely related to the widespread application of deep learning in the field of computer vision (CV). Existing video-based HOI recognition methods usually only use visual features, extract features from each frame of the video through a deep learning model, obtain interactive visual information, and then combine the temporal information of the image sequence to segment the HOI of the video using models such as Recurrent Neural Network (RNN) to finally obtain the recognition result. The single modality easily causes information loss and reduces the accuracy of recognition. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a method and device for character interaction recognition based on multi-feature fusion of video data, so as to improve the accuracy of character interaction recognition.

[0004] To achieve the above object, an embodiment of the present invention provides a method for identifying interactive characters based on multi-feature fusion of video data, the method comprising:

[0005] Obtain a set of key points of entities in the video data to obtain initial geometric features;

[0006] Inputting the initial geometric features into a geometric feature encoder to obtain target geometric features;

[0007] Obtaining the relative distance and relative angle of each pair of characters in the entity to obtain an initial global feature;

[0008] Inputting the initial global feature into a global feature encoder to obtain a target global feature;

[0009] Obtaining a set of detection frame images in the video data to obtain visual features;

[0010] The target geometric features, the target global features and the visual features are subjected to feature fusion processing to obtain a person interaction recognition result.

[0011] In some embodiments, the step of acquiring a set of key points of an entity in the video data to obtain initial geometric features includes:

[0012] Acquire key points of human skeleton and key points of objects in the video data;

[0013] According to the key points of the human skeleton and the key points of the object, a key point set of the entity is obtained to obtain the initial geometric features;

[0014] Wherein, the entity includes the human skeleton and the object.

[0015] In some embodiments, inputting the initial geometric features into a geometric feature encoder to obtain target geometric features includes:

[0016] Performing an embedding operation on the initial geometric features through a fully connected layer, and outputting a first intermediate variable;

[0017] According to the first intermediate variable, obtaining the dot product similarity between each key point;

[0018] Position encoding the dot product similarity to obtain an intermediate feature;

[0019] Through the Transformer encoder, multi-head attention calculation is performed on the intermediate features to obtain a second intermediate variable;

[0020] Residual connection and layer normalization operations are performed on the second intermediate variable to obtain target geometric features.

[0021] In some embodiments, the obtaining of the relative distance and relative angle of each pair of characters in the entity to obtain the initial global features includes:

[0022] Get the center point of the entity;

[0023] According to the center point, relative distances and relative angles of each pair of characters in the entity are obtained to obtain initial global features;

[0024] Wherein, the expression of the set of initial global features is:

[0025]

[0026] In the formula, f glob represents the set of initial global features; h1,…,h H Represents each human body; o1,…,o O Represents individual objects; Represents the initial global features of each person pair.

[0027] In some embodiments, inputting the initial global feature into a global feature encoder to obtain a target global feature comprises:

[0028] Inputting the initial global feature into a convolutional neural network for convolution operation to obtain a third intermediate variable;

[0029] The third intermediate variable is input into a multilayer perceptron to obtain the target global feature.

[0030] In some embodiments, the step of fusing the target geometric features, the target global features, and the visual features to obtain a person interaction recognition result includes:

[0031] Generate each instance according to the target geometric features, the target global features and the visual features;

[0032] Generate a frame-level state of the instance through a bidirectional recurrent neural network;

[0033] According to the frame-level state, obtaining frame-level inter-class interaction information and frame-level intra-class interaction information;

[0034] Segmenting the video data according to the frame-level inter-class interaction information and the frame-level intra-class interaction information to obtain target sub-actions;

[0035] Obtain segment-level inter-class interaction information and segment-level intra-class interaction information;

[0036] concatenating the frame-level inter-class interaction information, the frame-level intra-class interaction information, the segment-level inter-class interaction information, and the segment-level intra-class interaction information to obtain a fourth intermediate variable;

[0037] Inputting the fourth intermediate variable into the bidirectional recurrent neural network to update the segment-level state;

[0038] The target sub-action is identified through the segment-level state to obtain a character interaction identification result.

[0039] To achieve the above object, another aspect of an embodiment of the present invention provides a multi-feature fusion person interaction recognition device based on video data, the device comprising:

[0040] The first module is used to obtain a set of key points of entities in the video data and obtain initial geometric features;

[0041] The second module is used to input the initial geometric features into a geometric feature encoder to obtain target geometric features;

[0042] The third module is used to obtain the relative distance and relative angle of each person pair in the entity to obtain the initial global feature;

[0043] A fourth module is used to input the initial global feature into a global feature encoder to obtain a target global feature;

[0044] A fifth module is used to obtain a set of detection frame images in the video data to obtain visual features;

[0045] The sixth module is used to perform feature fusion processing on the target geometric features, the target global features and the visual features to obtain a person interaction recognition result.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned multi-feature fusion method for character interaction recognition based on video data.

[0047] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-feature fusion character interaction recognition method based on video data.

[0048] To achieve the above object, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned multi-feature fusion person interaction recognition method based on video data.

[0049] The embodiments of the present invention include at least the following beneficial effects: The present application provides a method and device for character interaction recognition based on multi-feature fusion of video data, the scheme obtains a set of key points of entities in the video data to obtain initial geometric features; inputs the initial geometric features into a geometric feature encoder to obtain target geometric features; obtains the relative distance and relative angle of each character pair in the entity to obtain initial global features; inputs the initial global features into a global feature encoder to obtain target global features; obtains a set of detection frame images in the video data to obtain visual features; performs feature fusion processing on the target geometric features, the target global features and the visual features to obtain character interaction recognition results. The present invention fuses multiple modal information through target geometric features, target global features and visual features, avoids information loss caused by a single modality, and can effectively improve the accuracy and robustness of character interaction recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 is a flow chart of a method for character interaction recognition based on multi-feature fusion of video data provided by an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of the structure of a geometric feature encoder provided in an embodiment of the present application;

[0053] Figure 3 is a schematic diagram of the structure of a global feature encoder provided in an embodiment of the present application;

[0054] Figure 4 It is a structural diagram of a feature fusion module provided in an embodiment of the present application;

[0055] Figure 5 It is a structural schematic diagram of a multi-feature fusion character interaction recognition device based on video data provided by an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0058] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0062] Human-Object Interaction (HOI) recognition is an important research direction in the field of computer vision, which aims to detect and understand the interaction between people and surrounding objects in images or videos.

[0063] HOI technology can help robots recognize and understand human behavioral intentions when interacting with objects, and enable robots to gain an overall perception of complex scenes, thereby providing necessary information for the robot's behavioral decision-making and planning, which is the basis for achieving natural human-computer interaction. However, existing HOI recognition methods mainly perform HOI recognition on a single RGB image. To apply HOI recognition to human-computer interaction, it is necessary to be able to perform HOI recognition on videos. Due to the diversity of interactive relationships in practical applications, the complexity of visual data, and the lack of background knowledge in videos, existing HOI recognition methods based on a single RGB image only obtain interactive visual information through extraction. The modality is single and prone to information loss. It is not enough to support real-time human-computer interaction, which reduces the accuracy of human interaction recognition.

[0064] In view of this, if Figure 1 As shown, the embodiment of the present invention provides a method for identifying interactive characters based on multi-feature fusion of video data. Figure 1 The method may include but is not limited to steps S100 to S600.

[0065] Step S100, obtaining a set of key points of an entity in video data to obtain initial geometric features;

[0066] Step S200, inputting the initial geometric features into a geometric feature encoder to obtain target geometric features;

[0067] Step S300, obtaining the relative distance and relative angle of each person pair in the entity to obtain an initial global feature;

[0068] Step S400, inputting the initial global feature into a global feature encoder to obtain a target global feature;

[0069] Step S500, obtaining a set of detection frame images in the video data to obtain visual features;

[0070] Step S600, performing feature fusion processing on the target geometric features, the target global features and the visual features to obtain a person interaction recognition result.

[0071] In steps S100 to S600 shown in the embodiment of the present invention, an encoder is introduced to regard the key point sequence of the entity as a batch of long sequence data for extracting geometric features, and global features are obtained through the relative positions of the center points of people and objects (including relative distances and relative angles). Combined with visual features, the three features are fused to obtain the result of character interaction recognition, which integrates multiple modal information such as visual and structural data, and can improve the accuracy and robustness of recognition.

[0072] In some embodiments, step S100 may include but is not limited to steps S110 to S120:

[0073] Step S110, obtaining key points of human skeleton and key points of objects in the video data;

[0074] Step S120, acquiring a key point set of the entity according to the key points of the human skeleton and the key points of the object, and obtaining the initial geometric features;

[0075] Specifically, the entity includes the human skeleton and the object. The human body can be represented by an abstract graph structure, each point represents a joint of the human skeleton, and the lines between the points represent the skeleton. Any action that the human body can make can be uniquely represented by a set of graph structures in space. The spatial position of each point and the angle between the lines can represent the action structure of the human body, and the movement speed and angle change of the point can represent the movement state of the human body. For an object, the key points of the entity can be represented by obtaining the corner points of the detection box in the video data.

[0076] For example, among the 2D key points of each person pair, the i-th point of the h-th person in the t-th frame image is the key point of the human skeleton. Then the expression of the key point set of all human bones in the video data is: Among them, T represents the frame image set of video data, H represents the total number of people, and I represents the key point set of a person's human skeleton. Each key point of the human skeleton includes information in four dimensions, and the expression is as follows:

[0077]

[0078] In the formula, represents the i-th human skeleton key point of the h-th person in the t-th frame image, Represents the horizontal coordinate of the i-th human skeleton key point of the h-th person in the t-th frame image, Represents the vertical coordinate of the i-th human skeleton key point of the h-th person in the t-th frame image, represents the horizontal axis velocity of the i-th human skeleton key point of the h-th person in the t-th frame image, represents the vertical axis velocity of the i-th human skeleton key point of the h-th person in the t-th frame image, and R represents a real number set.

[0079] In addition, the jth point of the oth object in the tth frame image is the key point of the object Then the expression of the key point set of all objects in the video data is: Among them, O represents the total number of objects. The key points of each object include information in four dimensions, and the expression is as follows:

[0080]

[0081] In the formula, represents the jth object key point of the oth object in the tth frame image, Represents the horizontal coordinate of the jth object key point of the oth object in the tth frame image, Represents the ordinate of the jth object key point of the oth object in the tth frame image, Represents the horizontal axis velocity of the jth object key point of the oth object in the tth frame image, Represents the vertical axis velocity of the j-th object key point of the o-th object in the t-th frame image.

[0082] Among them, the speed of the key points of the human skeleton is represented by the corresponding one-step forward difference, then In the formula, Represents the horizontal coordinate of the i-th human skeleton key point of the h-th person in the t+1-th frame image, represents the vertical coordinate of the i-th human skeleton key point of the h-th person in the t+1-th frame image. The velocity of the object key point is also represented by the corresponding one-step forward difference, so we have In the formula, Represents the horizontal coordinate of the jth object key point of the oth object in the t+1th frame image, Represents the ordinate of the j-th object key point of the o-th object in the t+1-th frame image.

[0083] According to the key points of each human skeleton and each object, the key point set of all entities in the video data can be obtained, and the expression of the initial geometric features is:

[0084]

[0085] In the formula, P represents the key point set of the entity, which represents the initial geometric features; Represents the key points of each human skeleton; Represents the key points of each object. Among them, the number of all key points in each frame image is N, and N = I × H + 2 × O, then P ∈ R N×T×4 .

[0086] In some embodiments, step S200 may include but is not limited to steps S210 to S250:

[0087] Step S210, embedding the initial geometric features through a fully connected layer to output a first intermediate variable;

[0088] Step S220, obtaining the dot product similarity between each key point according to the first intermediate variable;

[0089] Step S230, position encoding the dot product similarity to obtain an intermediate feature;

[0090] Step S240, performing multi-head attention calculation on the intermediate features through a Transformer encoder to obtain a second intermediate variable;

[0091] Step S250, performing residual connection and layer normalization operations on the second intermediate variable to obtain target geometric features.

[0092] Specifically, Figure 2 As shown in the figure, the geometric features encoder is used to process the initial geometric features. First, the initial geometric features are embedded through two fully connected layers (FC layers), and the expression of the first intermediate variable is obtained as follows:

[0093]

[0094] In the formula, f geo represents the first intermediate variable output after embedding the initial geometric features through the fully connected layer, σ(·) represents the ReLU activation function, W1 represents the first weight matrix, W1∈R 64×4 , W2 represents the second weight matrix, b1 represents the first bias vector, b2 represents the second bias vector, and C1 represents the first output dimension of the fully connected layer.

[0095] According to the first intermediate variable, the dot product similarity between each key point is calculated, and the expression of the dot product similarity between each key point is obtained as follows:

[0096]

[0097] In the formula, Y represents the dot product similarity, A represents the adjacency matrix, and W geo represents the transformation matrix, C2 represents the second dimension.

[0098] The dot product similarity is sent to the Transformer encoder. First, the features are aligned for position encoding, and then added to the dot product similarity to obtain the expression of the intermediate feature:

[0099] Y embed =Y+posEmbed(Y)

[0100] Where Y embed represents the intermediate feature, and posEmbed(·) represents the position encoding operation.

[0101] Through the Transformer encoder, multi-head attention is calculated for the intermediate features. Specifically, each intermediate feature is passed to h different self-attention modules, and num_h output matrices z are calculated. n , n∈1,2,…num_h. Among them, num_h is the number of subspaces in the multi-head attention mechanism, that is, the number of Head in Multi-Head Attention.

[0102] Each output matrix is ​​concatenated and passed through a linear layer to obtain the second intermediate variable. The expression of the output matrix is:

[0103]

[0104] The expression of the second intermediate variable is:

[0105] z=σ(W z cat(z1,z2,…,z h )+b z )

[0106] In the formula, Q Y Represents the intermediate feature Y embed The query matrix, K Y Represents the intermediate feature Y embed The key value matrix, V Y Represents the intermediate feature Y embed The value matrix of , (·) T represents the transpose operation, d k represents the vector dimension, softmax(·) represents the softmax function, z represents the second intermediate variable, W zrepresents the third weight matrix, cat(·) represents the concatenation operation, b z represents the third bias vector, z1,z2,…,z h Represents each output matrix.

[0107] The second intermediate variable is subjected to residual connection calculation and layer normalization operations, and then connected to an embedded multi-layer perceptron (MLP) after passing through multiple Transformer Encoder blocks to obtain the target geometric features.

[0108] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0109] Step S310, obtaining the center point of the entity;

[0110] Step S320, obtaining the relative distance and relative angle of each pair of characters in the entity according to the center point, and obtaining an initial global feature.

[0111] Specifically, the center point of each entity is obtained by averaging all the key points of each entity. For each person pair, the relative distance between the center point of the person and the center point of the object is calculated. and relative angle The initial global feature is obtained, and the expression of the initial global feature is:

[0112]

[0113] The expression of the set of initial global features is:

[0114]

[0115] In the formula, f glob represents the set of initial global features; h1,…,h H Represents each human body; o1,…,o O Represents individual objects; Represents the initial global features of each character pair; Representative figures (h i ,o j )’s relative distance between the center points; Representative figures (h i ,o j )’s center points; Relative distance between the center points of the representative character pair (h1, o1); Relative angle between the center points of the pair (h1,o1).

[0116] In some embodiments, step S400 may include but is not limited to steps S410 to S420:

[0117] Step S410, inputting the initial global feature into a convolutional neural network for convolution operation to obtain a third intermediate variable;

[0118] Step S420: input the third intermediate variable into a multilayer perceptron to obtain the target global feature.

[0119] Specifically, Figure 3 As shown in the figure, the global feature encoder is mainly composed of a one-dimensional convolutional neural network (1dCNN). The initial global feature is fed into the one-dimensional convolutional neural network, and ReLU is used as the activation function of the one-dimensional convolutional neural network to perform a convolution operation on the initial global feature, and the expression of the third intermediate variable is obtained as follows:

[0120] r = Conv1d(Conv1d(f glob ,D1),D2)

[0121] Where r represents the third intermediate variable of the one-dimensional convolutional neural network output, Conv1d(·) represents the convolution operation, and D1 and D2 represent the number of channels of the convolutional layer.

[0122] The third intermediate variable is sent to a multi-layer perceptron to obtain the target global feature.

[0123] In step S500 of some embodiments, a bounding box of each entity in each frame of the video data is obtained, and information of the bounding box is used as a representation of the visual feature.

[0124] In some embodiments, step S600 may include but is not limited to steps S610 to S680:

[0125] Step S610, generating each instance according to the target geometric features, the target global features and the visual features;

[0126] Step S620, generating a frame-level state of the instance through a bidirectional recurrent neural network;

[0127] Step S630, acquiring frame-level inter-class interaction information and frame-level intra-class interaction information according to the frame-level state;

[0128] Step S640, segmenting the video data according to the frame-level inter-class interaction information and the frame-level intra-class interaction information to obtain target sub-actions;

[0129] Step S650, obtaining segment-level inter-class interaction information and segment-level intra-class interaction information;

[0130] Step S660, concatenating the frame-level inter-class interaction information, the frame-level intra-class interaction information, the segment-level inter-class interaction information, and the segment-level intra-class interaction information to obtain a fourth intermediate variable;

[0131] Step S670, inputting the fourth intermediate variable into the bidirectional recurrent neural network to update the segment level state;

[0132] Step S680, identifying the target sub-action through the segment-level state to obtain a character interaction recognition result.

[0133] Specifically, Figure 4 As shown, construct Figure 4 The feature fusion module shown in the figure is used to fuse the target geometric features, target global features and visual features to obtain the result of human interaction recognition. The feature fusion module adopts the improved ASSIGN (Asynchronous-Sparse Interaction Graph Network) network model. ASSIGN is a recursive graph network that can automatically detect the HOI structure associated with asynchronous and sparse entities in the video. In the improved ASSIGN network model, the target geometric features and target global features are respectively regarded as independent instances of a category, and each object is regarded as an independent instance of a category to obtain an entity-human module, and each human body is regarded as an independent instance of a category to obtain an entity-object module, wherein the features of the entity-human module are the visual features of the corresponding human body, and the features of the entity-object module are the visual features of the corresponding object. Messages can be passed between each instance to achieve fusion between multiple features and improve the accuracy and robustness of recognition.

[0134] For example, in the improved ASSIGN network model, each instance e can be represented as {x e ,c e} e∈E , where E represents the total number of instances and x e Represents the characteristics of the instance, c e is the category of the instance. e When is the target geometric feature, the category c of the instance e is a geometric class; when the instance feature x e When is the target global feature, the category c of the instance e is a global class; when the instance feature x e is the visual feature of the object, then the category c of the instance eis the object class; when the feature x of the instance e When is the visual feature of the human body, the category c of the instance e For the human body.

[0135] Taking the bidirectional recurrent neural network (BiRNN) as the time edge, the frame level state of instance e at time t is generated, and the following expression is obtained:

[0136]

[0137] In the formula, Represents the frame-level state of instance e output by the bidirectional recurrent neural network at time t; Represents the characteristics of instance e at time t; Represents the state of the frame-level forward recurrent neural network; represents the state of the frame-level backward recurrent neural network; f represents the frame-level information.

[0138] According to the frame-level state, we can obtain the frame-level inter-class interaction information and frame-level intra-class interaction information of instance e at time t. Among them, the frame-level intra-class interaction information is calculated on the entity set of the same category. Then the expression of the frame-level inter-class interaction information is:

[0139]

[0140] The expression of frame-level intra-class interaction information is:

[0141]

[0142] In the formula, represents the frame-level inter-class interaction information from time t to instance e; represents the frame-level intra-class interaction information from time t to instance e; Represents the characteristics of instance e at time t; Represents the features of instance k at time t; Represents the frame-level state of instance e output by the bidirectional recurrent neural network at time t; represents the frame-level state of instance k output by the bidirectional recurrent neural network at time t; c k represents the category of instance k; c e represents the category of instance e; d represents the feature dimension.

[0143] According to the frame-level inter-class interaction information and frame-level intra-class interaction information, combined with the frame-level state, the video frame is segmented to obtain the target sub-action. Specifically, a multi-layer perceptron (MLP) and a differentiable discrete value estimator can be used to segment the video frame to determine whether the segmented video frame at the current moment is the last frame of the current video segment. The differentiable discrete value estimator uses the Gumbel-Softmax (GSM) operator, and the following expression is obtained:

[0144]

[0145] In the formula, Represents binary output. When , the video frame segmented at the current moment is the last frame of the current video segment; when When , the video frame segmented at the current moment is not the last frame of the current video segment. After the video frame segmentation of each video segment is completed, the target sub-action of each video segment can be obtained.

[0146] Only when When , messages are passed between each video segment to update the segment-level status. The process of message passing between the segmentation level is similar to the process of message passing between the frame level. The expression of the segment-level inter-class interaction information is:

[0147]

[0148] The expression of segment-level intra-class interaction information is:

[0149]

[0150] In the formula, represents the segment-level inter-class interaction information from time t to instance e; represents the segment-level intra-class interaction information from time t to instance e; Represents the features of instance e at time t-1; Represents the features of instance k at time t-1; Represents the segment-level state of instance e at time t-1; Represents the segment-level state of instance k at time t-1.

[0151] The fourth intermediate variable is obtained by concatenating the frame-level inter-class interaction information, the frame-level intra-class interaction information, the segment-level inter-class interaction information, and the segment-level intra-class interaction information. The expression is:

[0152]

[0153] The fourth intermediate variable is sent into the bidirectional recurrent neural network to update the segment-level state. The expression of the updated segment-level state is:

[0154]

[0155] In the formula, Represents the segment-level state of instance e at time t; Represents the state of the segment-level forward recurrent neural network; Represents the state of the segment-level backward recurrent neural network; Se represents segment-level information.

[0156] Use the updated segment-level state to identify the target sub-action of the current video segment and obtain the classification label of the current video segment. Then we have the classification label of the current video segment. The expression is:

[0157]

[0158] Among them, the classification label represents the result of character interaction recognition.

[0159] like Figure 5 As shown, the embodiment of the present invention further provides a multi-feature fusion character interaction recognition device 700 based on video data, which can implement the multi-feature fusion character interaction recognition method based on video data, and the device includes:

[0160] The first module 701 is used to obtain a set of key points of an entity in the video data to obtain initial geometric features;

[0161] The second module 702 is used to input the initial geometric features into a geometric feature encoder to obtain target geometric features;

[0162] The third module 703 is used to obtain the relative distance and relative angle of each person pair in the entity to obtain the initial global feature;

[0163] The fourth module 704 is used to input the initial global feature into a global feature encoder to obtain a target global feature;

[0164] The fifth module 705 is used to obtain a set of detection frame images in the video data to obtain visual features;

[0165] The sixth module 706 is used to perform feature fusion processing on the target geometric features, the target global features and the visual features to obtain a person interaction recognition result.

[0166] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0167] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned multi-feature fusion person interaction recognition method based on video data is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0168] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] refer to Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the multi-feature fusion character interaction recognition method based on video data in the embodiment of this application;

[0172] Input / output interface 803, used to implement information input and output;

[0173] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0174] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0175] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0176] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for multi-feature fusion character interaction recognition based on video data is implemented.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned multi-feature fusion person interaction recognition method based on video data.

[0179] In summary, the method and device for identifying interactive characters based on multi-feature fusion of video data according to the embodiment of the present invention have the following advantages:

[0180] 1. The embodiment of the present invention extracts geometric features through a geometric feature encoder, so that the model focuses on the details of human-object interaction actions and can model and understand long-term continuous video data streams. It also extracts global features through a global feature encoder, so that the model focuses on the overall motion momentum between people and objects, thereby improving recognition accuracy.

[0181] 2. The embodiment of the present invention constructs an improved ASSIGN network model. The entire network model takes visual features, geometric features and global features as inputs, uses multiple modalities for feature extraction, and realizes information fusion between different modalities. It can avoid information loss caused by a single modality, help provide more refined semantic understanding, and enhance the reasoning ability of the network model.

[0182] 3. The improved ASSIGN network model constructed in the embodiment of the present invention can be applied to human-computer interaction practices in different scenarios after being trained with various interactive action data sets.

[0183] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0184] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0185] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0186] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0187] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0188] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0189] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0190] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0191] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A multi-feature fusion person interaction recognition method based on video data, characterized in that: Obtain a set of key points of entities in the video data to obtain initial geometric features; Inputting the initial geometric features into a geometric feature encoder to obtain target geometric features; Obtaining the relative distance and relative angle of each pair of characters in the entity to obtain an initial global feature; Inputting the initial global feature into a global feature encoder to obtain a target global feature; Obtaining a set of detection frame images in the video data to obtain visual features; Performing feature fusion processing on the target geometric features, the target global features and the visual features to obtain a person interaction recognition result; The step of inputting the initial geometric features into a geometric feature encoder to obtain target geometric features includes: Performing an embedding operation on the initial geometric features through a fully connected layer, and outputting a first intermediate variable; According to the first intermediate variable, obtaining the dot product similarity between each key point; Position encoding the dot product similarity to obtain an intermediate feature; Through the Transformer encoder, multi-head attention calculation is performed on the intermediate features to obtain a second intermediate variable; Performing residual connection and layer normalization operations on the second intermediate variable to obtain target geometric features; The step of obtaining the relative distance and relative angle of each pair of characters in the entity to obtain the initial global features includes: Get the center point of the entity; According to the central point, the relative distance and relative angle of each pair of characters in the entity are obtained to obtain an initial global feature.

2. The method for identifying interactive characters based on multi-feature fusion of video data according to claim 1, characterized in that: The step of obtaining a set of key points of an entity in the video data to obtain initial geometric features includes: Acquire key points of human skeleton and key points of objects in the video data; According to the key points of the human skeleton and the key points of the object, a key point set of the entity is acquired to obtain the initial geometric features; Wherein, the entity includes the human skeleton and the object.

3. The method for identifying interactive characters based on multi-feature fusion of video data according to claim 1 is characterized in that: The step of obtaining the relative distance and relative angle of each pair of characters in the entity to obtain the initial global features includes: The expression of the set of initial global features is: In the formula, f glob represents the set of initial global features; h1,…,h H Represents each human body; o1,…,o O Represents individual objects; Represents the initial global features of each person pair.

4. The method for identifying interactive characters based on multi-feature fusion of video data according to claim 1, characterized in that: The step of inputting the initial global feature into a global feature encoder to obtain a target global feature comprises: Inputting the initial global feature into a convolutional neural network for convolution operation to obtain a third intermediate variable; The third intermediate variable is input into a multilayer perceptron to obtain the target global feature.

5. The method for identifying interactive characters based on multi-feature fusion of video data according to claim 1, characterized in that: The step of fusing the target geometric features, the target global features and the visual features to obtain a result of person interaction recognition includes: Generate each instance according to the target geometric features, the target global features and the visual features; Generate a frame-level state of the instance through a bidirectional recurrent neural network; According to the frame-level state, obtaining frame-level inter-class interaction information and frame-level intra-class interaction information; Segmenting the video data according to the frame-level inter-class interaction information and the frame-level intra-class interaction information to obtain target sub-actions; Obtain segment-level inter-class interaction information and segment-level intra-class interaction information; concatenating the frame-level inter-class interaction information, the frame-level intra-class interaction information, the segment-level inter-class interaction information, and the segment-level intra-class interaction information to obtain a fourth intermediate variable; Inputting the fourth intermediate variable into the bidirectional recurrent neural network to update the segment-level state; The target sub-action is identified through the segment-level state to obtain a character interaction identification result.

6. A multi-feature fusion person interaction recognition device based on video data, characterized in that: include: The first module is used to obtain a set of key points of entities in the video data and obtain initial geometric features; The second module is used to input the initial geometric features into a geometric feature encoder to obtain target geometric features; The third module is used to obtain the relative distance and relative angle of each person pair in the entity to obtain the initial global feature; A fourth module is used to input the initial global feature into a global feature encoder to obtain a target global feature; A fifth module is used to obtain a set of detection frame images in the video data to obtain visual features; The sixth module is used to perform feature fusion processing on the target geometric features, the target global features and the visual features to obtain a person interaction recognition result; Wherein, the second module is specifically used for: Performing an embedding operation on the initial geometric features through a fully connected layer, and outputting a first intermediate variable; According to the first intermediate variable, obtaining the dot product similarity between each key point; Position encoding the dot product similarity to obtain an intermediate feature; Through the Transformer encoder, multi-head attention calculation is performed on the intermediate features to obtain a second intermediate variable; Performing residual connection and layer normalization operations on the second intermediate variable to obtain target geometric features; The third module is specifically used for: Get the center point of the entity; According to the central point, the relative distance and relative angle of each pair of characters in the entity are obtained to obtain an initial global feature.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images