Semantic detection method, system and equipment for character interaction behaviors and medium

Abstract

The invention discloses a semantic detection method, system, equipment and medium for character interaction behaviors, and relates to the technical field of computer vision analysis, and the method comprises the steps: obtaining the type, posture and state information of a background, an intelligent agent and an object in a scene; detecting an intelligent body movement track, an object position and a character interaction behavior in a character interaction process in the scene by adopting a character interaction detection module to obtain a semantic description result of the character interaction behavior; and the character interaction detection module comprises interaction body detection, action primitive detection and behavior semantic description. According to the character interaction behavior detection method provided by the invention, through flexible combination of the character interaction detection modules, semantic description of character interaction behaviors in a complex scene from a space-time dimension is realized, and good interpretability and generalization are achieved; the accuracy and robustness of character interaction behavior detection are improved by defining the encapsulation and de-encapsulation operation of the interaction body.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision analysis, and in particular to a semantic detection method, system, device and medium for human interaction behavior. Background Art

[0002] Human-object interaction (HOI) refers to human-object interaction detection, aiming to locate people, objects, and the interactions between them in a scene. It has wide applications in scene understanding tasks such as visual analysis, scene question answering, and human-computer interaction, and is a crucial component of visual scene understanding and behavioral relationship detection within artificial intelligence. Human-object interaction relationships can typically be represented as a triplet: <person, interaction behavior, object>. For example, in a picture depicting a person riding a horse, the human-object interaction relationship can be represented as <person, riding, horse>. Here, we briefly explain the necessity of semantic description. Generally speaking, information is a combination of meaning and symbols. Intrinsic meaning can only be expressed through specific external forms (such as actions, expressions, text, audio, images, and other symbols). Specifically, in the field of visual analysis, a scene and its interactions can be expressed in various forms (external forms, symbols) such as text, images, audio, and video, which can be stored as data in various formats. The abstract descriptions (internal forms, meanings) of these different representations are semantics, which can be stored as structured representations of data in various formats. The space corresponding to these structured representations is the semantic space. For example, in common cognition, cats of different breeds, colors, and sizes (multi-category features) and cats described in text, images, and language (diverse forms of expression) are all cats (semantics).

[0003] Traditional HOI detection methods manually extract certain features (such as color, appearance, and shape) from people and objects in a scene, use Bayesian models to infer the type of interaction behavior, and leverage contextual information from images to improve the accuracy of behavior analysis. However, manually designed regions of interest are often difficult to accurately and promptly associate with interactive behaviors. Deep learning-based HOI detection methods typically utilize end-to-end trainable attention modules to detect and identify the actions of objects and agents in a scene and analyze the interactions between them. However, in complex scenarios, due to the large disparity in the distribution of training data for interactive behaviors, analysis and processing are easily affected by interference and influence from different features, resulting in poor robustness and accuracy. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a method, system, device and medium for semantic detection of human interaction behavior to solve the problems in the prior art.

[0005] The present invention specifically provides the following technical solutions: A semantic detection method for human interaction behavior, comprising: Collecting scene information and analyzing it to obtain information about the background, agents, and types, locations, and states of objects in the scene; agents are defined as humans and robots with human-like intelligence; At the beginning of human interaction detection, the actions and motion trajectories of the agent are tracked based on the types, positions, and states of objects in the scene. When the interaction conditions between the agent and the object or between the objects are met, the agent is encapsulated. When the agent and the object or between the objects separate or the interaction disappears, the agent is decapsulated. The action primitives between the start time and the moment of encapsulation or decapsulation are obtained. The action primitives are specifically the basic semantic units of the agent's movement. For the detected action primitives, the semantics of the action primitives are extracted by matching them with the action primitive library, and the semantics of all action primitives and the character interaction relationships in the character interaction process are analyzed. The semantics of the action primitives and the character interaction relationships are matched with the behavior knowledge base to output the semantic description results of the character interaction behavior.

[0006] Preferably, the interactive body is encapsulated when the interaction conditions between the intelligent agent and the object or between the objects are met, and the interactive body is decapsulated when the intelligent agent and the object or between the objects are separated or the interaction disappears, specifically: When encapsulating the interactive body, the local intelligent body and the interacted object are regarded as a whole, and no distinction is made during detection, but the relative independence of the two is still maintained within the encapsulated overall structure; when decapsulating the interactive body, in the interactive body formed by human-object-object interaction, the last interacted object is separated from the interactive body, and in the interactive body formed by human-object interaction, the interacted object is separated from the local intelligent body, and the decapsulated intelligent body or object is analyzed separately; the interactive body includes: the human-object interactive body formed by contact or interaction between the intelligent body and the object, and the human-object-object interactive body formed by contact or interaction between the interactive body and the object.

[0007] Preferably, the action primitive semantics refers to the semantic information corresponding to a certain action primitive described in the semantic space; the action primitive library specifically comprises: using relevant information processing and visual analysis technology to extract intelligent action primitives and describe their semantic information in pictures or videos of character interaction scenes, and constructing a multi-modal feature database for describing the semantics of various intelligent action primitives, wherein the multi-modal features describe the diverse features and data structures of the action information, including but not limited to skeleton trajectory features, optical flow features and color features.

[0008] Preferably, when the character interaction detection starts, it also includes the detection of empty nodes, which means that the intelligent agent and objects in the scene at that moment are in the initial state, and serve as an indication of the moment when the detection starts during the character interaction process.

[0009] Preferably, the interaction condition is specifically: the distance between the agent and the object meets the threshold set in the current scene, and the agent behavior and the object attributes meet the semantic constraints.

[0010] Preferably, before matching the semantics of action primitives, character interaction relationships and the behavior knowledge base, the rationality of the character interaction relationships is judged through the object attribute library, and matching is performed when it is reasonable; wherein the object attribute library is a database that describes the relationship between the object categories in the scene and their own attributes or usage functions.

[0011] Preferably, the behavior knowledge base is defined as: using technologies such as knowledge graphs to construct a hierarchical semantic relationship structure for describing character interaction behaviors, which is composed of action primitives based on rational constraints between intelligent agent behaviors and object attributes.

[0012] The present invention provides a semantic detection system for character interaction behavior, comprising: An information acquisition module, which is used to collect scene information and analyze it to obtain information about the background, intelligent agents, and the types, locations, and states of objects in the scene; intelligent agents are defined as humans and robots with human-like intelligence; The human interaction detection module is used to track the actions and motion trajectories of the agent based on the types, positions, and states of objects in the scene at the start of human interaction detection, encapsulate the agent when the interaction conditions between the agent and the object, or between the objects, are met, and decapsulate the agent when the agent and the object, or between the objects, separate or the interaction disappears, to obtain action primitives between the start time and the moment of encapsulation or decapsulation of the agent; the action primitives are specifically the basic semantic units of the agent's movement; The semantic extraction module is used to extract the semantics of the detected action primitives by matching them with the action primitive library, analyze the semantics of all action primitives and the character interaction relationships during the character interaction process, and match the semantics of the action primitives and the character interaction relationships with the behavior knowledge base to output the semantic description results of the character interaction behavior.

[0013] The present invention provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned semantic detection method for human interaction behavior.

[0014] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned semantic detection method for human interaction behavior are realized.

[0015] Compared with the prior art, the present invention has the following significant advantages: The method for detecting the semantics of interactive behaviors of characters proposed in the present invention tracks the actions and motion trajectories of intelligent bodies based on the types, positions and states of objects in the scene, and performs encapsulation and decapsulation of interactive bodies when conditions are met, thereby obtaining action primitives. By defining the encapsulation and decapsulation operations of interactive bodies, the accuracy and robustness of the detection of interactive behaviors of characters are improved. Based on the action primitives, the method is matched with the action primitive library, the semantics of the action primitives are extracted, the semantics of the action primitives and the interactive relationships of characters are matched with the behavioral knowledge base, and the semantic description results of the interactive behaviors of characters are output, thus realizing the semantic description of the interactive behaviors of characters in complex scenes from the spatiotemporal dimensions, which can effectively avoid the interference and adverse effects brought by the diverse expressions or certain characteristics of people and objects, effectively improving the accuracy and robustness of the detection of complex behaviors of characters in a small number of samples and multiple scenes, and solving the shortcomings of the existing behavior analysis methods that the performance is heavily dependent on training data and is easily interfered with. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The semantic model of character interaction behavior proposed by the present invention; Figure 2 Schematic diagram of the action primitive library, object attribute library and behavior knowledge library in the present invention; Figure 3 "Opening the door" for an application example using the present invention; Figure 4 The application example "eating an apple" of the present invention is used; DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0018] Example 1: A semantic detection method for character interaction behavior in this embodiment, such as Figure 1 As shown, the following steps are included: Step S1: Initialization.

[0019] An optical image acquisition device is used to acquire scene information, in this case a video or image sequence of the scene. Target detection technology is then used to analyze the scene information to obtain information such as the background, intelligent agents, and the type, posture, and state of objects in the scene. Intelligent agents are defined as humans and robots with human-like intelligence.

[0020] Scene modeling technology can be used to reconstruct three-dimensional information from two-dimensional plane information, thereby improving the accuracy of target detection and subsequent processing.

[0021] Step S2: Detect the motion trajectory of the intelligent body, the position of objects, and the interactive behavior of the characters in the scene to obtain the semantic description results of the interactive behavior of the characters.

[0022] Among them, step S21: when the character interaction detection starts, it also includes empty node detection.

[0023] The empty node is used to describe that the agent and objects in the scene maintain their initial state at that moment, which serves as an indicator of the moment when detection starts during the character interaction process.

[0024] Step S22: character interaction detection.

[0025] The HOI detection module is used to output the semantic description results of the interactive behavior of the characters in the scene.

[0026] Step S221: State detection, interaction body detection, and action primitive detection. Using target detection, human pose estimation, and behavior recognition, the motion trajectory of the agent and the position of objects during human interaction in the scene are acquired. At the start of human interaction detection, the action and motion trajectory of the agent are tracked based on the type, position, and state of the objects in the scene. When the interaction conditions are met between the agent and the object, or between objects, the interaction body is encapsulated. The local agent and the interacted object are considered as a whole and no longer distinguished during detection. However, the relative independence of the two is maintained within the encapsulated overall structure. If the interaction conditions are not met, the interaction body is decapsulated. That is, when the agent and the object, or between objects, are separated or the interaction disappears, the interaction body is decapsulated. The local agent is separated from the interacted object, and the decapsulated agent or object is analyzed separately to obtain the action primitives between the start time and the interaction body encapsulation or the interaction body decapsulation time.

[0027] When the interaction conditions are met, it also includes: the agent and the object not only meet the scope constraint, that is, the distance between the agent and the object should meet the reasonable threshold set by the current scene, but also meet the rationality constraint of the interaction behavior, that is, the agent behavior and the object attributes should meet reasonable semantic constraints.

[0028] An action primitive is defined as an action between the start of detection and the occurrence of human-object interaction; further, an action primitive can also be defined as an action between the formation of a human-object interaction body and the occurrence of a human-object-object interaction; further, an action primitive can also be defined as an action between the decapsulation of the interaction body.

[0029] Step S222: Extract action primitive semantics.

[0030] The detected action primitives are matched against an action primitive library to extract their semantics. This library uses relevant information processing and visual analysis techniques to extract agent action primitives from images or videos of human interaction scenes and describe their semantics. This library then constructs a multi-modal feature database to describe the semantics of various agent action primitives. These multi-modal features describe the diverse characteristics and data structures of action information, including but not limited to skeletal trajectory features, optical flow features, and color features.

[0031] Step S223: behavioral semantic extraction.

[0032] Analyze the semantics of all action primitives and character interaction relationships during the character interaction process, judge the rationality of the character interaction relationship based on the object attribute library, and when it is reasonable, match the semantics of the action primitives, character interaction relationships and the behavior knowledge library to output the semantic description results of the character interaction behavior.

[0033] The object attribute library is a database that describes the relationships between object categories in the scene and their own attributes or usage functions. The behavior knowledge base is a hierarchical semantic relationship structure that describes character interaction behaviors. It is composed of action primitives based on the rationality constraints between intelligent agent behaviors and object attributes.

[0034] Step S224: The detection process ends.

[0035] Empty nodes are used to describe the state of the agent and objects in the scene at that moment, and are used to indicate the moment when the detection ends during the character interaction process.

[0036] The above steps S221 and S222 can be flexibly combined and used according to the specific circumstances of the scene.

[0037] Example 2: In order to illustrate the human interaction semantic model proposed in the present invention, the human interaction detection dataset CAD-120 is taken as an example to further describe the relationship and composition structure among the action primitive library, object attribute library and behavior knowledge library in the present invention. Figure 2 shown.

[0038] The CAD-120 dataset is an RGB-D video dataset focusing on high-level human activities and object interactions, primarily used to detect and recognize 10 categories of human indoor behaviors (such as preparing food and taking medicine). The CAD-120 dataset contains 120 video sequences, each of which is annotated with 10 activity labels (such as move, reach, and open), and each object is annotated with 12 attribute labels (such as moveable, reachable, and openable). Based on the CAD-120 dataset, the action primitive library in this paper contains 10 activity labels, the object attribute library contains multiple objects and 10 attribute labels, and the behavior knowledge base contains 10 categories of human interaction behaviors.

[0039] Specifically, using the character interaction semantic model proposed in the present invention, for a character interaction behavior in a scene, such as "opening the door", the semantic description of the character interaction behavior is as follows: "Open the door" = "Open (action primitive combination)" + "Interaction (hand and door handle)" "Open" = "Reach (Action Primitive)" + "Move (Action Primitive)" "Interaction body (hand and door handle)" = "hand" + "door handle (object attribute: openable)" In the above semantic description, the "hand" as the doer of the behavior forms an interactive body (hand and door handle) with the object "door handle"; the "door handle" as the carrier of the behavior has the attribute of "openable"; the two conform to the mutual constraints of the intelligent agent behavior and the object attributes. When the interactive body moves, its semantic description result is "opening the door".

[0040] Example 3: Application example "door opening" of the character interaction detection method proposed by the present invention. Figure 3 As shown, the following steps are included: Step KM1: Initialization.

[0041] Use the Kinect depth camera to obtain a video or image sequence of the scene, and use the YOLOv8 detection algorithm to obtain the position of people in the scene and the position and status of doors.

[0042] The initial state of the scene is that the door is closed and the human hand does not touch the door handle.

[0043] Step KM2: Detect the motion trajectory of the intelligent body, the position of objects, and the interactive behavior of the characters in the scene to obtain the semantic description results of the interactive behavior of the characters.

[0044] Step KM21: Empty node detection.

[0045] The empty node is used to describe the moment when the detection starts during the character interaction process, indicating that no character interaction behavior occurs in the scene at that moment.

[0046] Step KM22: Human interaction detection.

[0047] In the first case, step KM221 (a): state detection, interaction body detection and action primitive detection.

[0048] Use the YOLOv8 detection algorithm, the BlazePose human pose estimation algorithm, and the ST-GCN behavior recognition algorithm to obtain the motion trajectories of people (including hands) and the positions of doors (including door handles) in the scene at consecutive moments.

[0049] When it is detected that the human hand and the door handle meet the interaction conditions, the human hand and the door handle are regarded as a whole "hand + door handle" and encapsulated as an interactive body. No distinction is made during detection, but the relative independence of the two is still maintained internally.

[0050] Preferably, the interaction condition can be set as: the center distance between the human hand and the door handle is less than a set threshold, or the overlap ratio between the human hand area and the door handle area is greater than a set threshold.

[0051] The action between the start of detection and the formation of the interaction body "hand + door handle" is defined as an action primitive.

[0052] Step KM222(a): Action primitive semantic extraction.

[0053] The detected action primitive is matched with the action primitive library and the semantics of the action primitive detected in step KM221 (a) is identified as "reach", that is, the human hand touches the door handle.

[0054] In the second case, step KM221(b): state detection, interaction body detection and action primitive detection.

[0055] After the interactive body is encapsulated, the YOLOv8 detection algorithm, BlazePose human pose estimation algorithm, and ST-GCN behavior recognition algorithm are used to obtain the motion trajectory and position of the interactive body "hand + door handle" in the scene at continuous moments.

[0056] When it is detected that the interaction object does not meet the interaction conditions, the interaction object "hand + door handle" is unpacked, the human hand is out of contact with the door handle, and the interaction behavior ends.

[0057] Preferably, the interaction condition can be set as: the center distance between the human hand and the door handle is greater than a set threshold, or the overlap ratio between the human hand area and the door handle area is less than a set threshold.

[0058] The action between the start moment of the interaction object "hand + door handle" movement and the end moment of the interaction behavior is defined as a new action primitive.

[0059] Step KM222(b): Action primitive semantic extraction.

[0060] For the detected new action primitive, by matching it with the action primitive library, it is identified that the semantics of the action primitive detected in step KM221 (b) is "movement", that is, the hand and the door handle move as a whole.

[0061] Step KM223: behavioral semantic extraction.

[0062] The extracted action primitive semantics of "reach" and "move" are analyzed. According to the attribute of the object "door" in the object attribute library, which is "openable", the semantic description of the character interaction behavior in the scene is output as "opening the door" by matching with the behavior knowledge base.

[0063] Step KM224: The detection process ends.

[0064] The moment when the interactive body "hand + door handle" is unpacked is defined as a new empty node, indicating that no character interaction behavior occurs in the scene at that moment.

[0065] Example 4: Application example of the character interaction detection method proposed by the present invention, "eating an apple", such as Figure 4 As shown, the following steps are included: Step CPG1: Initialization.

[0066] Use the Kinect depth camera to obtain a video or image sequence of the scene, and use the YOLOv8 detection algorithm to obtain the position of people, tables, and apples in the scene.

[0067] The initial state of the scene is that the apple is on the table and the human hand does not touch the apple.

[0068] Step CPG2: Detect the motion trajectory of the intelligent body, the position of objects, and the interactive behavior of the characters in the scene to obtain the semantic description results of the interactive behavior of the characters.

[0069] Step CPG21: Empty node detection.

[0070] The empty node is used to describe the moment when the detection starts during the character interaction process, indicating that no character interaction behavior occurs in the scene at that moment.

[0071] Step CPG22: character interaction detection.

[0072] In the first case: Step CPG221 (a): state detection, interactor detection and action primitive detection.

[0073] Use the YOLOv8 detection algorithm, the BlazePose human pose estimation algorithm, and the ST-GCN behavior recognition algorithm to obtain the motion trajectory of people (including hands) and the position of apples in the scene at consecutive moments.

[0074] When it is detected that the human hand and the apple meet the interaction conditions, the human hand and the apple are regarded as a whole "hand + apple" and encapsulated as an interactive body. No distinction is made during detection, but the relative independence of the two is still maintained internally.

[0075] Preferably, the interaction condition can be set as: the center distance between the human hand and the apple is less than a set threshold, or the overlap ratio between the human hand area and the apple area is greater than a set threshold.

[0076] The action between the start of detection and the formation of the interactive body "hand + apple" is defined as an action primitive.

[0077] Step CPG222(a): Action primitive semantic extraction.

[0078] For the detected action primitive, by matching it with the action primitive library, the semantics of the action primitive detected in step CPG221 (a) is identified as "reach", that is, the human hand touches the apple.

[0079] In the second case, step CPG221 (b): state detection, interactor detection and action primitive detection.

[0080] After the interactive body is encapsulated, the YOLOv8 detection algorithm, BlazePose human pose estimation algorithm, and ST-GCN behavior recognition algorithm are used to obtain the motion trajectory and position of the interactive body "hand + apple" in the scene at continuous moments.

[0081] When it is detected that the interactive body "hand + apple" meets the interaction conditions with the mouth, a new interaction process occurs. The human hand, apple, and mouth are regarded as a whole "hand + apple + mouth" and a new interactive body is encapsulated. No distinction is made during detection, but the relative independence of the two is still maintained internally.

[0082] Preferably, the interaction condition can be set as: the center distance between the "hand + apple" and the mouth is less than a set threshold, or the overlap ratio between the "hand + apple" area and the mouth area is greater than a set threshold.

[0083] The action between the start of the movement of the interactive body "hand + apple" and the moment of obtaining the new interactive body "hand + apple + mouth" is defined as a new action primitive.

[0084] Step CPG222(b): Action primitive semantic extraction.

[0085] For the detected new action primitive, by matching with the action primitive library, the semantics of the new action primitive detected in step CPG221 (b) is identified as "movement", that is, the hand and the apple are placed near the mouth as a whole.

[0086] In the third case, step CPG221 (c): state detection, interaction body detection and action primitive detection.

[0087] After the new interactive body is encapsulated, the YOLOv8 detection algorithm, BlazePose human pose estimation algorithm, and ST-GCN behavior recognition algorithm are continued to be used to obtain the motion trajectory and position of the interactive body "hand + apple + mouth" in the scene at continuous moments.

[0088] When it is detected that the interaction body "hand + apple + mouth" does not meet the interaction conditions, the interaction body is unpacked, and the interaction body "hand + apple + mouth" is unpacked, and the "hand + apple" is out of contact with the mouth.

[0089] Preferably, the interaction condition can be set as: the center distance between the "hand + apple" and the mouth is greater than a set threshold, or the overlap ratio between the "hand + apple" area and the mouth area is less than a set threshold.

[0090] The action between the start moment of the interaction body "hand + apple + mouth" movement and the decapsulation moment of the interaction body "hand + apple + mouth" is defined as a new action primitive.

[0091] Step CPG222: action primitive semantic extraction.

[0092] For the detected new action primitive, by matching it with the action primitive library, the semantics of the new action primitive detected in step CPG221 (c) is identified as "eating", that is, the hand, apple and mouth move continuously as a whole.

[0093] In the fourth case, step CPG221 (d): state detection, interaction body detection and action primitive detection.

[0094] After the new interactive body is encapsulated, the YOLOv8 detection algorithm, BlazePose human pose estimation algorithm, and ST-GCN behavior recognition algorithm are continued to be used to obtain the motion trajectory and position of the interactive body "hand + apple" in the scene at continuous moments.

[0095] When it is detected that the interactive body "hand + apple" does not meet the interaction conditions, the interactive body is unpacked, the interactive body "hand + apple" is unpacked, the hand and apple are out of contact, and the interaction behavior ends.

[0096] Preferably, the interaction condition can be set as: the center distance between the hand and the apple is greater than a set threshold, or the overlap ratio between the hand area and the apple area is less than a set threshold.

[0097] The action between the start moment of the interaction body "hand + apple" movement and the decapsulation moment of the interaction body "hand + apple" is defined as a new action primitive.

[0098] Step CPG222(d): Action primitive semantic extraction.

[0099] For the detected new action primitive, by matching with the action primitive library, it is identified that the semantics of the new action primitive detected in step CPG221 (d) is "put down", that is, there is no apple in the hand.

[0100] Step CPG223: behavioral semantic extraction.

[0101] The extracted action primitive semantics of "reach", "move", "eat" and "put down" in each case are analyzed. According to the attributes of the object "apple" in the object attribute library, which are "movable and edible", the semantic description of the character's interactive behavior in the scene is output as "eating apple" by matching with the behavior knowledge base.

[0102] Step CPG224: The detection process ends.

[0103] The moment when the interactive body "hand + apple" is unpacked is defined as a new empty node, indicating that no character interaction occurs in the scene at that moment.

[0104] The present invention proposes a semantic detection system for human interaction behavior, comprising an information acquisition module and a state detection module.

[0105] Among them, the information acquisition module is used to collect scene information and analyze the scene information to obtain the type, position and status information of the background, intelligent agent and objects in the scene; the intelligent agent is defined as a human and a robot with human-like intelligence; the detection module is used to detect the motion trajectory of the intelligent agent, the position of objects, and the interactive behavior of the characters during the interaction between the characters in the scene, and obtain the semantic description results of the interactive behavior of the characters.

[0106] The state detection module detects the motion trajectory of the intelligent body, the position of objects, and the interactive behavior of the characters in the process of human interaction in the scene. It specifically includes: a human interaction detection module and a semantic extraction module, wherein: the human interaction detection module is used to track the action and motion trajectory of the intelligent body based on the type, position and state of the objects in the scene at the beginning of human interaction detection, encapsulate the interactive body when the interaction conditions are met between the intelligent body and the object or between objects, and decapsulate the interactive body when the intelligent body and the object or between objects are separated or the interaction disappears, and obtain the action primitives between the start time and the interactive body encapsulation or decapsulation time; the semantic extraction module is used to extract the action primitive semantics by matching the detected action primitives with the action primitive library, analyze the semantics of all action primitives in the process of human interaction, judge the rationality of the interaction relationship according to the object attribute library, and output the semantic description results of the human interaction behavior by matching with the behavior knowledge base.

[0107] The present invention also provides a computer device, including a memory and a processor. The memory stores a program, and when the program is executed by the processor, the processor executes the steps of a method for detecting human interaction behavior.

[0108] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device that enables a computing device to communicate with one or more other computing devices (e.g., routers, modems, etc.).

[0109] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. For those skilled in the art to which the present invention belongs, several simple deductions or replacements can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A semantic detection method for character interaction behavior, characterized in that: include: Collect scene information and analyze it to obtain the background, agents, and type, location, and status information of objects in the scene; The intelligent agent is defined as a human and a robot with human-like intelligence; At the beginning of human interaction detection, the actions and motion trajectories of the agent are tracked based on the types, positions, and states of objects in the scene. When the interaction conditions between the agent and the object or between the objects are met, the agent is encapsulated. When the agent and the object or between the objects separate or the interaction disappears, the agent is decapsulated. The action primitives between the start time and the moment of encapsulation or decapsulation are obtained. The action primitives are specifically the basic semantic units of the agent's movement. For the detected action primitives, the semantics of the action primitives are extracted by matching them with the action primitive library, and the semantics of all action primitives and the character interaction relationships in the character interaction process are analyzed. The semantics of the action primitives and the character interaction relationships are matched with the behavior knowledge base to output the semantic description results of the character interaction behavior.

2. The semantic detection method of character interaction behavior according to claim 1, characterized in that: The interactive body is encapsulated when the interaction conditions between the intelligent agent and the object or between the objects are met, and the interactive body is decapsulated when the intelligent agent and the object or between the objects are separated or the interaction disappears, specifically: When encapsulating the interactive body, the local intelligent agent and the object being interacted with are considered as a whole and no longer distinguished during detection. However, the relative independence of the two is maintained within the encapsulated overall structure. When decapsulating the interactive body, in the interactive body formed by human-object-object interaction, the object being interacted with is separated from the interactive body. In the interactive body formed by human-object interaction, the object being interacted with is separated from the local intelligent agent. The decapsulated intelligent agent or object is analyzed separately. The interactive bodies include: human-object interactive bodies formed by contact or interaction between intelligent bodies and objects, and human-object-object interactive bodies formed by contact or interaction between interactive bodies and objects.

3. The semantic detection method of a person interaction behavior according to claim 1, characterized in that: The action primitive semantics refers to the semantic information corresponding to a certain action primitive described in the semantic space; the action primitive library specifically uses relevant information processing and visual analysis technology to extract intelligent action primitives and describe their semantic information in pictures or videos of human interaction scenes, and constructs a multi-modal feature database for describing the semantics of various intelligent action primitives, wherein the multi-modal features describe the diverse features and data structures of action information, including but not limited to skeleton trajectory features, optical flow features and color features.

4. The semantic detection method of character interaction behavior according to claim 1, characterized in that: When the character interaction detection starts, it also includes the detection of empty nodes. The empty nodes indicate that the intelligent body and objects in the scene at that moment remain in the initial state, which serves as an indication of the moment when the detection starts during the character interaction process.

5. The semantic detection method of character interaction behavior according to claim 1, characterized in that: The interaction conditions are specifically: the distance between the agent and the object meets the threshold set in the current scene, and the agent behavior and object attributes meet the semantic constraints.

6. The semantic detection method of character interaction behavior according to claim 1, characterized in that: Before matching the semantics of action primitives, character interaction relationships and the behavior knowledge base, the rationality of the character interaction relationships is also judged through the object attribute library, and matching is performed when it is reasonable; the object attribute library is a database that describes the relationship between the object categories in the scene and their own attributes or usage functions.

7. The semantic detection method of character interaction behavior according to claim 1, characterized in that: The behavior knowledge base is a hierarchical semantic relationship structure that describes the interactive behavior of characters, and is composed of action primitives based on the rationality constraints between the behavior of intelligent agents and the attributes of objects.

8. A semantic detection system for character interaction behavior, characterized in that: include: The information acquisition module is used to collect scene information and analyze the scene information to obtain the background, intelligent agents, and the type, location and status information of objects in the scene; The intelligent agent is defined as a human and a robot with human-like intelligence; The human interaction detection module is used to track the actions and motion trajectories of the agent based on the types, positions, and states of objects in the scene at the start of human interaction detection, encapsulate the agent when the interaction conditions between the agent and the object, or between the objects, are met, and decapsulate the agent when the agent and the object, or between the objects, separate or the interaction disappears, to obtain action primitives between the start time and the moment of encapsulation or decapsulation of the agent; the action primitives are specifically the basic semantic units of the agent's movement; The semantic extraction module is used to extract the semantics of the detected action primitives by matching them with the action primitive library, analyze the semantics of all action primitives and the character interaction relationships during the character interaction process, and match the semantics of the action primitives and the character interaction relationships with the behavior knowledge base to output the semantic description results of the character interaction behavior.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the semantic detection method for human interaction behavior as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the semantic detection method of human interaction behavior described in any one of claims 1 to 7 are implemented.

Images