A method for detecting human interaction using textual identity and visual cues

Abstract

The present invention provides a method for detecting human interaction by coordinating text identity and visual clues, designs a dual-branch multimodal human interaction detection network consisting of a visual modality branch and a text modality branch; uses text features to encode the identity of objects in a human pair, fully explores the label distribution knowledge in a human interaction detection data set, explicitly constructs a constraint relationship between the object identity and the interactive action category, and clearly brings into play the role of the object identity in interactive reasoning; for the feature difference between the visual modality and the text modality, solves the multimodal feature fusion problem in the task interaction detection task by transferring the feature knowledge in a visual language model, and on this basis, designs a multimodal feature fusion module, takes the two modal representations of the object identity as a guide, aligns and fuses the visual and text modality representations of the same human pair to reason about the interactive behavior between people and objects.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for detecting human interaction by coordinating text identity and visual clues. Background Art

[0002] Human interaction detection is an important topic in the field of computer vision. Its purpose is to locate human and object instances in images and identify the interaction relationship between these instances. In this task, the interaction between humans and surrounding scenes is abstracted into <human, object, action> triples, where human and object instances are described by bounding boxes with identity labels, and the interactions between them are classified into corresponding action categories. Human interaction detection enables artificial algorithms to no longer simply view images as a combination of various object bounding boxes, but also to clarify the relationship between the bounding boxes of each instance. Therefore, this task plays an important role in promoting computers to approach the human level of understanding of the real world.

[0003] The task of human interaction detection can be summarized as the process of detecting instances and classifying the interactions between these instances. Early research on human interaction detection used traditional methods to extract visual features from images. These methods require manual feature construction and lack generalization capabilities compared to deep learning-based methods. In the past decade, object detection methods using deep learning techniques have made significant progress and have greatly reshaped the research work in the field of human interaction detection based on object detection. Unlike object detection that focuses on isolated instances, the essence of human interaction detection is to learn an effective representation of human-object relationships in human-centered scenes, which involves multiple explanatory factors such as instance identity, spatial relationships, and scene context, and requires a logical representation to express human interactions. In current research work, it is mainly through the extraction and processing of visual features. This counter-intuitive processing method does not work well for complex understanding tasks.

[0004] Since the interaction between people and the surrounding environment is displayed in the image, the visual signal encapsulates various clues related to the interaction. Research work in the field of human interaction detection mainly focuses on designing different methods to extract various types of features from images and construct representations of human interaction behaviors based on the detected features to predict interaction behaviors. Human interaction detection methods based on deep learning require an object detection network to extract local candidate regions representing objects (including humans) in the input image, and classify interaction behaviors by combining local candidate regions. According to the different stages required to process local candidate regions, early human interaction detection methods based on convolutional models can be divided into two categories: two-stage methods and one-stage methods. The two-stage method uses a pre-trained object detector to identify individuals and objects in the image. Subsequently, these methods enumerate and pair instances and widely use four visual features in the local image regions describing the instances to learn the interaction between people and objects, namely appearance, spatial relationship, human posture and human body parts, and classify the interaction behaviors of human pairs by fusing the features filtered by different branches. With the emergence of anchor-based one-stage object detection methods, one-stage human interaction detection methods with simpler model structures have also been proposed compared to two-stage methods. In the one-stage human interaction detection method, instance detection and interaction classification are performed simultaneously. The interaction behavior is described in a joint region that can cover both human and object instances. The visual features of humans and objects are extracted from a larger image region to locate instances and classify interactions. In recent years, the trend of using Transformer models for computer vision tasks has also influenced the research on human interaction detection. Transformer-based human interaction detection methods inherit the processing paradigm of convolutional network-based human interaction detection methods, and are also studied according to the two-stage and one-stage paradigms. Compared with convolutional network-based human interaction detection methods, Transformer-based human interaction detection methods have achieved superior performance, making the research on Transformer-based human interaction methods a new trend. The key value of the Transformer-based method is self-attention. The self-attention mechanism enables the model to learn to pay attention to important areas of the image through training. Compared with visual algorithms based on convolutional models, self-attention can use soft mask matrices to describe the importance of each pixel on the image. This extensive contextual information is considered to be very effective in reasoning about human interaction behaviors.

[0005] Although the development of the field of object detection has greatly promoted the research work on visual signals in the field of human interaction detection, compared with object detection, human interaction detection involves multiple explanatory factors such as instance identity, spatial relationship and scene context, each of which plays an important role in interaction detection. Due to the significant intra-identity differences of visual features, this leads to unsatisfactory expression of the combination of person-pair instance identities. Therefore, in order to enhance the clarity of the expression of instance identities in person pairs and further improve the accuracy of human interaction detection, some research works use text features extracted from instance categories to express instance identities, and merge these text features with visual features to determine whether there is interaction between instances. Summary of the invention

[0006] The embodiment of the present invention provides a method for detecting human interaction by coordinating text identity and visual clues, which is used to solve the technical problems existing in the prior art.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme.

[0008] A method for detecting person interactions by coordinating text identity and visual cues, comprising:

[0009] S1 performs target detection on the original image through the target detection model in the feature extraction network of the network model to obtain the detector backbone network feature map, instance appearance features, instance category and bounding box coordinates;

[0010] S2 performs an integration operation based on the appearance features and bounding box coordinates of the detector backbone network feature map instance through the visual cue integration branch of the interactive classification network of the network model to obtain the first interactive visual representation; it also processes the original image through the visual encoder in the feature extraction network to obtain a global semantic feature map;

[0011] S3 performs semantic enhancement operation on the first interactive visual representation based on the global semantic feature graph through the visual cue integration branch of the interactive classification network to obtain a second interactive visual representation corresponding to the text modality feature;

[0012] S4 traverses the person-object pairs of all images of the data set to which the original image belongs based on the instance category, obtains person interaction triples having the instance category of the object in the person-object pair, and performs a text format conversion operation on the person interaction triples;

[0013] S5 performs a feature vector conversion operation on the execution result of step S4 to obtain the interactive text representations of all person-object pairs, and merges the interactive text representations of the person-object pairs to obtain the interactive text representation;

[0014] S6 performs a cross-fusion operation on the second interactive visual representation and the interactive text representation through an interactive classification network to obtain an interactive representation of the synergistic effect of instance identity and visual clues;

[0015] S7 repeatedly executes steps S1 to S6 for multiple rounds, updates the weights of the interaction classification network, and obtains a person interaction detection model;

[0016] S8 uses the human interaction detection model to perform human interaction detection on the target image.

[0017] Preferably, step S2 comprises:

[0018] Pass-through

[0019] I=SelfAttn(I) (1)

[0020] I pair =F contac t(Ih , I o ) (2)

[0021] The appearance features of the human body in the original image I h and the appearance characteristics of the object I o Connect them to obtain the combined person-object pair appearance feature I pair Where, F contact (I h ,I o )) is a connection operation, SelfAttn() represents the self-attention operation of the self-attention mechanism, the parameters in () act as query, key and value in the self-attention operation, and I represents the instance appearance feature;

[0022] Pass-through

[0023] S=F contact (MLP(F encode (b)) (3)

[0024] P=MLP(F contact (I pair ,S)) (4)

[0025] The appearance feature I pair and the spatial relationship feature S to obtain the feature P of the person-object pair; where S is the spatial encoding feature of the person-object pair, P is the updated feature of the person-object pair, and F ecncode Represents the process of calculating various spatial information based on the bounding box coordinates b;

[0026] Pass-through

[0027] V = CrossAttn(P, G) (5) Calculate and obtain the first interactive visual representation V related to the interactive behavior; where CrossAttn(P, G) represents the cross-attention operation, P in () acts as a query in the cross-attention operation, and G is the global feature extracted by the backbone network of the target detection model, acting as a key and value in ();

[0028] Pass-through

[0029] Vs=VisEnc(Reshape(x)) (6)

[0030] The global semantic feature map Vs is obtained by calculation; where Reshape(x) represents image size adjustment, and x in () represents the original image;

[0031] Step S3 includes:

[0032] Pass-through

[0033] V=Proj(SelfAttn(V)) (7)

[0034] y v =CrossAttn(V,Vs) (8)

[0035] Calculate the second interactive visual representation y v ; Where Proj represents the linear projection operation, which transforms the dimension of the feature.

[0036] Preferably, step S4 comprises:

[0037] Pass-through

[0038] T ij =TextEnc(HOI(i,j)),i∈N q ,j∈K (9)

[0039] Perform text format conversion on the character interaction triplet; where HOI(i,j) represents the text description of human-object interaction, i and j represent the object and action categories respectively, K represents the number of action categories corresponding to each person in the dataset, and T ij It is the text feature corresponding to each character’s description of the interactive text;

[0040] Step S5 includes:

[0041] Pass-through

[0042] T i =L2Norm(∑ j∈K T ij ) (10)

[0043] Calculate the text feature T corresponding to each character's description of the interactive texti The average value of; where L2Norm is the L2 normalization operation;

[0044] Pass-through

[0045] y t =F contact (T1,T2,…,T n ) (11)

[0046] Merge the interactive text representations of the person-object pair to obtain the interactive text representation y t .

[0047] Preferably, step S6 comprises:

[0048] Pass-through

[0049] y v =Proj(y v ) (12)

[0050] y=CrossAttn(y v ,y t ) (13)

[0051] y′=SeldAttn(y) (14)

[0052] The second interactive visual representation and the interactive text representation are merged to obtain the interactive representation of the instance identity and the synergistic effect of visual cues; where y is the fused character interaction representation, and y′ is the further improved and refined character interaction representation obtained after the self-attention operation.

[0053] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention provides a method for detecting human interaction by coordinating text identity and visual clues, and designs a dual-branch multimodal human interaction detection network consisting of a visual modality branch and a text modality branch; uses text features to encode the object identity in the human pair, and fully explores the label distribution knowledge in the human interaction detection data set, explicitly constructs the constraint relationship between the object identity and the interactive action category, and clearly plays the role of the object identity in interactive reasoning; for the feature difference between the visual modality and the text modality, the multimodal feature fusion problem in the task interaction detection task is solved by transferring the feature knowledge in the visual language model, and on this basis, a multimodal feature fusion module is designed, which uses the two modal representations of the object identity as a guide, aligns and fuses the visual and text modality representations of the same human pair to reason about the interactive behavior between people and objects. The method provided by the present invention solves the problem of feature differences between visual and text modalities, realizes the synergy between object identity and visual clues, thereby fully integrating the explanatory factors involved in human interaction behavior, and greatly improving the reliability of human interaction detection.

[0054] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0056] Figure 1 A processing flow chart of a method for detecting character interaction by coordinating text identity and visual clues provided by the present invention;

[0057] Figure 2 An architectural diagram of a detection model for a method of detecting human interaction by coordinating text identity and visual clues provided by the present invention;

[0058] Figure 3 A flowchart of a preferred embodiment of a method for detecting human interaction by coordinating text identity and visual cues provided by the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0062] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0063] The present invention provides a method for detecting human interaction by coordinating text identity and visual clues, so as to solve the following problems existing in the prior art.

[0064] For the task of human interaction detection involving multiple explanatory factors, it is relatively direct and effective to use multi-modal features to discover and integrate explanatory factors. In the relatively mature visual signal extraction scheme for human interaction detection, there is currently work that uses text modality to express instance identity to strengthen the expression of structural relationships between instances in images. The human interaction representations learned by these methods are more powerful than those of pure visual methods. However, there are still the following shortcomings:

[0065] 1. The method of combining visual features and text features is relatively simple, ignoring the inherent differences between features of different modalities. As a result, it is difficult for the instance identity expressed by the text features to achieve sufficient semantic association with the information in the visual features at the interactive level. The synergy between text features and visual features is not positive, and even the feature information in different modalities will interfere with each other, reducing the classification effect of interactive behavior.

[0066] 2. The model that uses text modal features to express the identity of characters to instances relies on the word embedding vector distribution provided by the pre-trained text sequence model to implicitly measure the proximity between instances and between instances and actions. It lacks clear directionality between instance identity and action category, and cannot guarantee the effective role of instance identity in the interactive behavior reasoning process.

[0067] The method provided by the present invention is inspired by the process of human perception activities during the experiment, extracts interaction-related detail clues in the visual modality, realizes the role of instance identity in the image in the text modality, and uses the prior feature knowledge in the visual language large model to fill the semantic gap between different modalities, thereby effectively integrating the feature information between multiple modalities. Thus, a method for improving character interaction detection by coordinating text modality and visual modality is designed.

[0068] See also Figure 1The present invention provides a method for detecting character interaction by coordinating text identity and visual clues, comprising the following steps:

[0069] S1 performs target detection on the original image through the target detection model in the feature extraction network to obtain the detector backbone network feature map, instance appearance features, instance category and bounding box coordinates;

[0070] S2 obtains the first interactive visual representation by integrating the detector backbone network feature map, instance appearance features, and bounding box coordinates through the visual cue integration branch of the interactive classification network. It also processes the original image through the visual encoder in the feature extraction network to obtain a global semantic feature map.

[0071] S3 performs semantic enhancement operation on the first interactive visual representation based on the global semantic feature graph through the visual cue integration branch of the interactive classification network to obtain a second interactive visual representation corresponding to the text modality feature;

[0072] S4, based on the instance category, traverses the person-object pairs of all images of the data set to which the original image belongs, obtains person interaction triples having the instance category of the object in the person-object pair, and performs a text format conversion operation on the person interaction triples;

[0073] S5 performs a feature vector conversion operation on the execution result of step S4 to obtain the interactive text representations of all person-object pairs, and merges the interactive text representations of the person-object pairs to obtain the interactive text representation;

[0074] S6 performs a cross-fusion operation on the second interactive visual representation and the interactive text representation through an interactive classification network to obtain an interactive representation of the synergistic effect of instance identity and visual clues;

[0075] S7 repeats steps S1 to S6 for multiple rounds to update the weight of the interaction classification network to improve the accuracy of the system for character interaction classification and recognition, and obtains a trained character interaction detection model;

[0076] S8 performs human interaction detection on the target image using the human interaction detection model.

[0077] According to some existing technical solutions, the perceptual process of human recognition of human interaction behavior is summarized as follows: 1) Detecting and identifying human and object instances in the scene; 2) Using the spatial relationship between human and object and the identity of the object, judging whether there is interaction between human and object and the possible interaction action category; 3) Determining the interaction action category between human and object pairs based on the appearance details of the instance and the scene context. As the technology in the field of target detection is becoming increasingly mature, the model of the present invention adopts a two-stage paradigm, using a pre-trained target detection model to complete the instance detection and positioning in the image in the first stage, and focusing on the design of the interaction classification network in the second stage, designing a logical representation that can fully extract and integrate various interaction-related explanatory factors, thereby effectively expressing the interaction behavior.

[0078] The method of character interaction detection of the present invention provides an improved interaction detection model, and the overall structure of the model is composed of a feature extraction network and an interaction classification network. The feature extraction network is composed of two parts: a pre-trained target detection model and a visual language model. The function of the feature extraction network is first to complete the positioning detection of instances in the image, and secondly to extract various feature information for use by the interaction behavior classification network. The interaction classification network is composed of three parts: a visual modality branch, a text modality branch, and a multimodal interaction feature fusion module. The interaction classification network integrates the feature information extracted by the feature extraction network from the perspectives of visual modality and text modality, and coordinates the role of the feature information represented by the two modalities in the multimodal feature fusion module to infer the interaction behavior category. Assume that the target detection model uses instance category c and bounding box coordinates b to represent the location of instances in the image. In addition, the feature information extracted from the feature extraction network also includes instance appearance features I extracted by the target detection model, global features G extracted by the backbone network of the target detection model, and image-level visual semantic features Vs extracted by the visual language model. The visual modality branch in the interactive classification network is represented by Visual, the text modality branch is represented by Text, and the multimodal feature fusion module is represented by Multi. Then, the model of the present invention processes the input image data, and the interactive visual representation y v and interactive text representation y t The process of collaboratively generating the character interaction representation y is shown in formula (I)(II)(III):

[0079] y v =Visual(b,I,G,Vs) (I)

[0080] y t =Text(c) (II)

[0081] y=Multi(y v ,y t ) (III)

[0082] The entire model structure is shown in the figure Figure 2 shown.

[0083] The improved detection model provided by the present invention adopts a two-stage human interaction detection process. In the first stage, the object detection network is only responsible for detecting (i.e., locating and classifying) instances in the image; in the second stage, the interaction classification network is responsible for classifying the interactions between the detected instances.

[0084] The training focus of this detection model is to improve the classification accuracy of the interaction classification network for the interactions between detected instances. In order to avoid optimizing the weights of both target detection and interaction classification at the same time, which would result in suboptimal training results for both subtasks, the target detection network uses the DETR network pre-trained with the target detection dataset and freezes the weights to retain its good target detection capabilities, and only updates the weights of the interaction classification network.

[0085] In the preferred embodiment provided by the present invention, the configuration of each component of the detection model is specifically as follows.

[0086] 1. Feature Extraction Network

[0087] Through the pre-trained target detection model and visual language model, the detection and positioning of instances in the image are completed, and the features related to the explanatory factors required for human interaction detection are extracted. DETR is used as the target detection model in the designed model structure, which can solve the instance positioning task in human interaction detection. DETR outputs appearance features I containing instance details according to the input image data x, and uses instance category c and bounding box coordinates b to describe the detection information of instances in the image. For the visual language model, the CLIP model trained with large-scale visual-text pair data has a strong cross-modal alignment capability, and the distribution of its visual and text features has an inherent correlation. Therefore, the feature knowledge in the CLIP model is transferred and the features are processed in the subsequent interactive classification network, which is conducive to the alignment and fusion of multimodal features to achieve the synergy of multiple explanatory elements. CLIP extracts bimodal features through visual encoders and text encoders. The visual encoder VisEnc processes the input image data x to obtain global visual features v, and the text encoder TextEnc extracts text representation t for each category label c. The calculation process of the two modal encoders of CLIP is shown in formula (IV):

[0088]

[0089] 2. Interactive Classification Network

[0090] Based on the results output by the target detection model, the people and objects detected in the image are first connected in the interactive classification network to obtain all possible people-object pairs in the image. Then, the relevant explanatory factors of these people-object pairs are extracted from the perspectives of both visual and textual modalities. In the visual branch, the instance appearance features, bounding box coordinates and global features in the DETR model are used to generate a visual representation with spatial position information and interactive context clues, and then the global semantic features extracted from the image by the CLIP visual encoder are migrated to enhance the visual representation. In the text branch, the potential action categories corresponding to each object identity are obtained in combination with the label distribution of the data set and constraints are established, and the CLIP text encoder is used to generate a text representation that can express such constraints. For the two modal representations, a multimodal feature fusion module is designed in the present invention, and the instance identity is used as the explanatory factor that guides the two modal representations to fuse the expressions in the two modalities at the interactive level.

[0091] (1) Visual modality branch

[0092] High-quality visual representation is crucial for accurate reasoning of interactive behavior. Images encapsulate multiple explanatory factors related to interaction, including the spatial relationship between people and objects, instance appearance features, and scene context related to interaction. Therefore, it is necessary to extract these visual explanatory factors to prepare for interactive reasoning. The present invention effectively combines the positioning function of the detection model and the semantic understanding function of the visual language model to construct interactive visual representation.

[0093] The above step S2 includes the following process: using the instance appearance feature to merge the spatial relationship between the person-object pair and extract the scene context information. Initially, the instance appearance feature I is updated through the self-attention mechanism, and the feature dimension is The updated instance features are divided into the appearance features of the human body I h and the appearance characteristics of the object I o , and connect them:

[0094] I=SelfAttn(I) (1)

[0095] I pair =F contact (I h ,I o ) (2)

[0096] Among them, I pair isI h and I o The merged appearance features of the person-object pairs are the visual representations of all possible person-object pairs, F contact(α, β) is a connection operation, SelfAttn(α) represents the self-attention operation, and α acts as a query, key, and value in the self-attention operation. In order to represent the positioning information of the person-object pair and the spatial relationship between the person and the object, the spatial features of the person-object pair are calculated from the bounding box, including the center, width, and height of the instance box, as well as the intersection and union of the person box and the object box, the relative area, and the direction. Then, a multilayer perceptron (MLP) is used to encode the position of the person-object pair and obtain the spatial relationship S of the person-object pair. Then, the appearance feature I is converted into pair Fused with the spatial relationship feature S, we get the feature P of the person-object pair:

[0097] S=F contact (MLP(F encode (b)) (3)

[0098] P=MLP(F contact (I pair ,S)) (4)

[0099] Among them, S is the spatial encoding feature of the person-object pair, P is the updated feature of the person-object pair, and F encode Represents the process of calculating various spatial information based on bounding box coordinates. Compared with general object recognition, interactive reasoning needs to consider more scene context information between people and objects, rather than just focusing on the appearance features of people and objects. The present invention adopts a cross-attention mechanism to retrieve the context related to the interaction from the global feature map G extracted by the detector DETR backbone network, thereby supplementing the context area information related to the interactive behavior:

[0100] V=CrossAttn(P,G) (5)

[0101] The resulting interactive visual representation V is the first interactive visual representation, which contains the local appearance details, spatial structure information and related contextual scene information of the person-object pair, and has sufficient extraction of interactive explanatory factors in the image. CrossAttn(α,β) represents the cross-attention mechanism, where α is the query and β is the key and value.

[0102] In order to better realize the combination of visual representation and textual representation, the feature knowledge in the transfer visual language model is further enhanced to enhance the interactive visual representation V. First, the visual language model is used to extract features from the image, and the input image x is adjusted to the required size without cropping to fully retain the feature extraction ability of the CLP model. Subsequently, the CLIP visual encoder is used to generate an image-level semantic feature map:

[0103] Vs=VisEnc(Reshape(x)) (6)

[0104] Among them, Vs is the semantic feature map, Reshape(x) represents image size adjustment, and x in () represents the original image.

[0105] Next (i.e., step S3 above), the semantic feature map of CLIP is used to enhance the interactive visual representation V. After updating the interactive visual representation V through the self-attention operation, a cross-attention operation is performed on the updated interactive visual representation V and the semantic feature map Vs:

[0106] V=Proj(SelfAttn(V)) (7)

[0107] y v =CrossAttn(V,Vs) (8)

[0108] Among them, y v It is the HOI visual representation, that is, the second interactive visual representation, which inherits the good semantic correspondence between CLIP visual features and language patterns, and is more robust in feature expression, thus being able to more effectively promote the alignment and fusion between the two interactive modal representations.

[0109] (2) Text mode branch

[0110] In the process of human perception of interactive behavior, object identity plays a role in judging whether there may be interaction between people and objects, and what potential interactive behavior categories are. In the process of reasoning about interactive categories, object identity can reduce the range of action types, and determine the correct action category in combination with detail clues in the corresponding action range. Therefore, to construct the role of instance identity in interactive behavior reasoning, it is necessary to clearly express the instance identity in the person-object pair and link the instance identity with the potential behavior category. In the text identity generation branch, the present invention combines the label distribution of the data set to build the constraint range of instance identity and behavior category, and uses the CLIP text encoder to convert the text description into a feature vector, that is, each verb category in the data set is converted into a sentence, and then the corresponding text features and visual features are generated for similarity matching to act as a classifier. First, all person interaction behavior labels containing object categories in person-object pairs are retrieved from the person interaction detection benchmark data sets HICO-DET and V-COCO. For example, taking 'chair' as an example, the corresponding action categories in the data set include 'hold', 'carry', 'lie on', 'siton', 'stand on' and 'no interaction'. These retrieved person interaction labels are converted into corresponding person interaction text descriptions in the format of 'A photo of a person[Verb-ing]a / an[Object]', where "no interaction" is replaced by "and". Then, the CLIP text encoder is used to generate text features for the interaction text description of the person-object pair:

[0111] T ij =TextEnc(HOI(i,j)),i∈N q ,j∈K (9)

[0112] Among them, HOI(i,j) represents the HOI text description, i and j represent the object and action categories respectively. ij is the text feature corresponding to each character’s interactive text description, where K represents the number of action categories corresponding to each person in the dataset, and C t is the text feature dimension. The text feature of each person-object pair is the sum of all corresponding text features T ij The average value of:

[0113] T i =L2Norm(∑ j∈K T ij ) (10)

[0114] Among them, L2Norm is the L2 normalization operation, T i is the calculated text representation of the interaction between people and objects. Then, the text representation of each pair of people and objects is merged. iGet interactive text representations for all pairs of people in the image:

[0115] y t =F contact (T1,T2,…,T n ) (11)

[0116] By normalizing the textual features of each person-object pair, each person-object pair can maintain an appropriate semantic distance with the corresponding action category, thus helping the visual signal to obtain accurate action prediction within the corresponding action range.

[0117] (3) Multimodal feature fusion module

[0118] In order to address the modal heterogeneity between vision and text, the present invention incorporates the knowledge of CLIP into the construction of interactive visual and textual representations, thereby minimizing the feature gap between the two modalities. On this basis, in order to better effectively combine the roles of explanatory factors in the two modalities, the present invention integrates the rich contextual clues in the visual representation and the action constraints of the text representation by establishing a multimodal feature fusion module using cross-attention and self-attention mechanisms. First, with the help of CLIP's powerful image-text matching capabilities, the cross-attention mechanism uses the instance category expression in the text representation and the instance appearance features in the visual representation for semantic alignment, embedding the visual features into the text features, thereby promoting the combination of multimodal information between each character pair:

[0119] y v =Proj(y v ) (12)

[0120] y=CrossAttn(y v ,y t ) (13)

[0121] Among them, y is the fused character interaction representation, which is further improved and strengthened through the self-attention mechanism:

[0122] y′=SelfAttn(y) (14)

[0123] The multimodal feature fusion module uses instance identity as guidance to unify action prior knowledge in textual representations with contextual clues in visual representations, thereby facilitating accurate interaction inference using visual cues within the range of actions relevant to human-object pairings.

[0124] The present invention also provides an embodiment for demonstrating the effect of implementing the method provided by the present invention.

[0125] like Figure 3 As shown, the implementation process of this embodiment is as follows:

[0126] Step 1: Prepare input data for training the model. The datasets used in this invention are HICO-DET and V-COCO.

[0127] Step 2: Prepare the pre-trained model required for the feature extraction network. The present invention uses DETR as the target detection model and CLIP of ViT-32 / B version as the feature enhancement model.

[0128] Step 3: For the input image data x, the DETR model is used for target detection, and the detector backbone network feature map G, instance appearance features I, and instance category c representing the detection result and the corresponding bounding box coordinates b are output.

[0129] Step 4: The backbone network feature map, instance appearance features, and bounding box coordinates output by the DETR model in the third step are input into the visual cue integration branch of the interactive classification network for integration to obtain the interactive visual representation V. At the same time, as in the third step, for the input image data x, the size is changed to 224×224 and then input into the CLIP visual encoder to output the global semantic feature map Vs of the image.

[0130] Step 5: Input the global semantic feature map Vs output by the CLIP model in the third step into the visual cue integration branch of the interactive classification network to semantically enhance the interactive visual representation V and obtain the interactive visual representation y that has a good correspondence with the text modality features. v .

[0131] Step 6: Search the dataset according to the instance category output by the DETR model in step 3. For each person-object pair, all person interaction triplets containing the object instance category can be obtained.<human,object,action> , converting them one by one into text descriptions in the format of 'A photo of a person[Verb-ing]a / an[Object]'.

[0132] Step 7: The interactive text description of each person pair is converted into a feature vector through the CLIP text encoder. The feature vectors of multiple interactive text descriptions corresponding to each person-person pair are processed by L2 normalization to obtain the interactive text representation of each person pair. The interactive text representations of all person pairs in the image are merged to obtain the interactive text representation y corresponding to the input data t .

[0133] Step 8: The multimodal feature fusion module in the interactive classification network fuses the interactive representations output by the visual and text branches to obtain an interactive representation of the synergy of instance identity and visual cues.

[0134] Step 9: Steps 3 to 8 are the forward propagation process of the model. The model is trained for 30 rounds using the dataset, where the pre-trained models DETR and CLIP in the feature extraction network are frozen and weights are not trained, and the weights of the interaction classification network are updated. A highly accurate human interaction detection model can be obtained.

[0135] Step 10: End the process.

[0136] In terms of visual signals, instance appearance features, spatial information and scene context are effectively extracted from images; in terms of text identity, text features are used to clearly express the identity of objects in human-object pairs, and the correspondence between the identity of objects in human-object pairs and potential action categories is constructed with the help of the label distribution of the data set. In the process of constructing the model, the interactive representation data of the two modal features is strengthened by migrating the features in the CLIP model, reducing the feature differences between the modalities. On this basis, the multimodal feature fusion module can effectively align the visual and text modal data belonging to the same person pair, realize the instance identity to constrain the action range while using the context clues in the visual signal for interactive classification. The test sets of the HICO-DET and V-COCO benchmark data sets were selected for testing. The experimental results are shown in Table 1. The detection performance of the present invention is very superior.

[0137] Table 1 Ours represents the method of the present invention, and the table shows the comparison of the detection performance of each model.

[0138]

[0139]

[0140] In summary, the present invention provides a method for detecting human interaction by coordinating text identity and visual clues, studies the logical representation of human interaction behavior, and uses multiple modalities to extract explanatory factors related to interaction behavior. Text features are used to clearly express the identity of the object in the person-object pair, avoiding the intra-class difference problem of the expression of instance identity by visual features, and combined with the distribution of behavior labels in the data set of human interaction detection, the behavior category range corresponding to each instance identity is established, and the constraint conditions between instance identity and behavior category are vectorized, so that the role of object identity can be expressed in model reasoning.

[0141] The present invention also introduces the visual language model CLIP to solve the problem of feature differences between multimodal signals in the field of human interaction detection, so that the feature information between different modalities can be effectively combined. On this basis, the multimodal feature fusion module explores the fusion method of visual modality representation and text modality representation in human interaction detection based on the attention mechanism, effectively integrating the action constraints of object identity and the interaction detail clues in visual features to perform reasoning on human interaction detection, thereby improving the detection accuracy of the model, which also has a driving effect on the representation learning research of human interaction.

[0142] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0143] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0144] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0145] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for detecting person interaction by coordinating text identity and visual cues, characterized in that: include: S1 performs target detection on the original image through the target detection model in the feature extraction network of the network model to obtain the detector backbone network feature map, instance appearance features, instance category and bounding box coordinates; S2 performs an integration operation based on the appearance features and bounding box coordinates of the detector backbone network feature map instance through the visual cue integration branch of the interactive classification network of the network model to obtain a first interactive visual representation; and also processes the original image through the visual encoder in the feature extraction network to obtain a global semantic feature map; S3 performs a semantic enhancement operation on the first interactive visual representation based on the global semantic feature graph through a visual cue integration branch of an interactive classification network to obtain a second interactive visual representation having a corresponding relationship with a text modality feature; S4, based on the instance category, traverses the person-object pairs of all images of the data set to which the original image belongs, obtains person interaction triples having the instance category of the object in the person-object pair, and performs a text format conversion operation on the person interaction triples; S5 performs a feature vector conversion operation on the execution result of step S4 to obtain the interactive text representations of all person-object pairs, and merges the interactive text representations of the person-object pairs to obtain the interactive text representation; S6 performs a cross-fusion operation on the second interactive visual representation and the interactive text representation through an interactive classification network to obtain an interactive representation of the instance identity and the visual clues acting in synergy; S7 repeatedly executes steps S1 to S6 for multiple rounds, updates the weights of the interaction classification network, and obtains a person interaction detection model; S8 performs human interaction detection on the target image using the human interaction detection model.

2. The method according to claim 1, characterized in that Step S2 includes: Pass-through I=SelfAttn(I) (1) I pair =F contact (I h ,I o ) (2) The appearance features of the human body in the original image I h and the appearance characteristics of the object I o Connect them to obtain the combined person-object pair appearance feature I pair Where, F contact (I h ,I o )) is a connection operation, SelfAttn() represents the self-attention operation of the self-attention mechanism, the parameters in () act as query, key and value in the self-attention operation, and I represents the instance appearance feature; Pass-through S=F contact (MLP(F encode (b)) (3) P=MLP(F contact (I pair ,S)) (4) The appearance feature I pair and the spatial relationship feature S to obtain the feature P of the person-object pair; where S is the spatial encoding feature of the person-object pair, P is the updated feature of the person-object pair, and F encode Represents the process of calculating various spatial information based on the bounding box coordinates b; Pass-through V = CrossAttn(P, G) (5) Calculate and obtain the first interactive visual representation V related to the interactive behavior; where GrossAttn(P, G) represents the cross-attention operation, P in () acts as a query in the cross-attention operation, and G is the global feature extracted by the backbone network of the target detection model, acting as a key and value in (); Pass-through Vs=VisEnc(Reshape(x)) (6) The global semantic feature map Vs is obtained by calculation; where Reshape(x) represents image size adjustment, and x in () represents the original image; Step S3 includes: Pass-through V=Proj(SelfAttn(V)) (7) y v =GrossAttn(V,Vs) (8) Calculate and obtain the second interactive visual representation y v ; Where Proj represents the linear projection operation, which transforms the dimension of the feature.

3. The method according to claim 2, characterized in that Step S4 includes: Pass-through T ij =TextEnc(HOI(i,j)),i∈N q ,j∈K (9) The character interaction triplet is converted into a text format; where HOI(i,j) represents the text description of the human-object interaction, i and j represent the object and action categories respectively, K represents the number of action categories corresponding to each person-object pair in the dataset, and T ij It is the text features corresponding to the text description of each person-character interaction; Step S5 includes: Pass-through T i =L2Norm(∑ j∈K T ij ) (10) Calculate the text feature T corresponding to each character's description of the interactive text i The average value of; where L2Norm is the L2 normalization operation; Pass-through y t =F contact (T1,T2,…,T n ) (11) The interactive text representations of the person-object pair are merged to obtain the interactive text representation y t .

4. The method according to claim 3, characterized in that Step S6 includes: Pass-through and v =Proj(and v ) (12) y=CrossAttn(y v ,and t ) (13) y′=SelfAttn(y) (14) The second interactive visual representation and the interactive text representation are merged to obtain the interactive representation of the instance identity and the visual clues in synergy; wherein y is the fused character interaction representation, and y′ is the further improved and refined character interaction representation obtained after the self-attention operation.

Images