Interactive behavior analysis method, system, electronic device and storage medium

Abstract

The embodiments of the present application provide an interactive behavior analysis method, system, electronic device, and storage medium, which belong to the field of artificial intelligence technology. The method includes: obtaining the positioning data of the target object in the target space to obtain object positioning data; obtaining spatial audio data; obtaining spatial video data corresponding to the target space; performing shallow behavior analysis based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavior feature data set; constructing a knowledge graph based on each shallow behavior feature data to obtain interactive behavior graph data; performing correlation analysis on the interactive behavior graph data through a pre-trained behavior analysis model to obtain interactive behavior analysis information. The embodiments of the present application can identify and analyze complex interactive behaviors in fixed scenes.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interactive behavior analysis method, system, electronic device, and storage medium. Background Art

[0002] Behavior recognition involves identifying and understanding human behavior patterns by analyzing data such as images or sensors. Multimodal analysis is particularly important in behavior recognition because it involves using multiple types of data and information sources to improve the accuracy and robustness of behavior recognition.

[0003] While multimodal data analysis can easily capture simple behavioral information, it is challenging for complex social scenarios. For example, in teaching scenarios, where complex interactions occur between students and teachers, simple behavioral analysis results are not practical or meaningful. Therefore, identifying and analyzing complex interactions within fixed scenarios has become a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose an interactive behavior analysis method, system, electronic device and storage medium, aiming to identify and analyze complex interactive behaviors in fixed scenes.

[0005] To achieve the above-mentioned objectives, a first aspect of an embodiment of the present application provides an interactive behavior analysis method, the method comprising: acquiring positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0006] Acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object;

[0007] Acquiring spatial video data corresponding to the target space;

[0008] Performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset includes at least two different shallow behavior feature data, and the shallow behavior feature data represents interaction behavior information between the target object and at least one of the other objects;

[0009] Constructing a knowledge graph based on each of the shallow behavioral feature data to obtain interactive behavior graph data;

[0010] The interactive behavior graph data is subjected to association analysis by a pre-trained behavioral analysis model to obtain interactive behavior analysis information.

[0011] In some embodiments, performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset specifically includes:

[0012] Performing scene modeling on the target space according to the spatial video data to obtain spatial map data;

[0013] extracting the spatial video data related to the target object based on the object positioning data to obtain object video data;

[0014] Extract facial features based on the object video data to obtain facial feature data;

[0015] Extracting spectrum features from the spatial audio data to obtain spectrum feature data;

[0016] Respective shallow-layer behavior feature data are generated based on the space map data, the object video data, the facial feature data, and the spectrum feature data.

[0017] In some embodiments, the shallow behavioral feature data includes any one of the following: voice interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data. Generating each shallow behavioral feature data based on the spatial map data, the object video data, the facial feature data, and the spectral feature data includes:

[0018] Performing speech recognition based on the spectral feature data to obtain speech interaction recognition data;

[0019] obtaining the spatial movement data about the target object based on the spatial map data and the object positioning data;

[0020] Performing an emotion feature analysis based on the spectral feature data and the facial feature data to obtain the emotion interaction feature data;

[0021] The target object is subjected to character motion analysis based on the object video data to obtain the object interaction motion data.

[0022] In some embodiments, the voice interaction recognition data, the emotional interaction feature data, the spatial movement data, and the object interaction action data are all related to time series, and the knowledge graph is constructed based on each of the shallow behavior feature data to obtain the interaction behavior graph data, including:

[0023] Performing graph data integration on the voice interaction recognition data, the emotion interaction feature data, the spatial movement data, and the object interaction action data of the same time series to generate graph node data corresponding to the time series;

[0024] Performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining the other objects interacting with the target object, and obtaining interaction pointing information;

[0025] Generate behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information;

[0026] A spatiotemporal graph is generated based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

[0027] In some embodiments, performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining the other objects interacting with the target object, and obtaining interaction pointing information includes:

[0028] Acquire identity information of all objects in the target space, and obtain identity information of the interaction-directed object based on the identity information and the voice interaction recognition data;

[0029] Perform visual direction recognition based on the object interaction action data to obtain interactive visual direction information;

[0030] Performing object trajectory recognition based on the spatial movement data to obtain a target object movement trajectory;

[0031] Based on the target object's movement trajectory, the interactive visual direction information, and the interactive pointing object's identity information, the other objects interacting with the target object are determined to obtain the interactive pointing information.

[0032] In some embodiments, before performing association analysis on the interactive behavior graph data using a pre-trained behavior analysis model to obtain interactive behavior analysis information, the method further includes pre-training the behavior analysis model, specifically including:

[0033] Acquiring behavioral knowledge data; wherein the behavioral knowledge data represents knowledge theory data related to interactive behavior between at least two objects;

[0034] Acquire a training data set; wherein the training data set includes behavioral training data and behavioral indicator information corresponding to the behavioral training data;

[0035] Inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis model to perform behavior analysis to obtain training recognition information;

[0036] Perform error calculation based on the training identification information and the behavior indicator information to obtain comparison deviation data;

[0037] In some embodiments, after performing association analysis on the interactive behavior graph data using the pre-trained behavior analysis model to obtain interactive behavior analysis information, the method includes:

[0038] Based on the interaction behavior analysis information, the interaction behavior of the target object is evaluated to obtain interaction evaluation information; wherein the interaction evaluation information indicates the area in the target space where the target object has interacted;

[0039] generating interaction suggestion information based on the interaction evaluation information;

[0040] Push the interaction suggestion information to the target object.

[0041] To achieve the above objectives, a second aspect of an embodiment of the present application provides an interactive behavior analysis system, the system comprising:

[0042] a positioning data acquisition module, configured to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is configured to interact with at least one other object in the target space;

[0043] An audio data acquisition module, configured to acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object;

[0044] A video data acquisition module, configured to acquire spatial video data corresponding to the target space;

[0045] a shallow behavior analysis module, configured to perform shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset comprising at least two different shallow behavior feature data, each of which represents interactive behavior information between the target object and at least one of the other objects;

[0046] A knowledge graph construction module is used to construct a knowledge graph based on each of the shallow behavioral feature data to obtain interactive behavior graph data;

[0047] The large model association analysis module is used to perform association analysis on the interactive behavior graph data through a pre-trained behavior analysis large model to obtain interactive behavior analysis information.

[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the interactive behavior analysis method described in the first aspect when executing the computer program.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the interactive behavior analysis method described in the first aspect above.

[0050] The interactive behavior analysis method, system, electronic device and storage medium proposed in this application obtain the positioning data of the target object in the target space, obtain the object positioning data, obtain the spatial audio data, and obtain the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data and spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis by a pre-trained large behavior analysis model to obtain interactive behavior analysis information. Therefore, this application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data based on the shallow behavior feature data, associates the shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a large behavior analysis model with behavioral knowledge data, thereby being able to identify and analyze complex interactive behaviors in fixed scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the interactive behavior analysis method provided in an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Flowchart of step S104 in FIG.

[0053] Figure 3 yes Figure 2 Flowchart of step S205 in FIG.

[0054] Figure 4 yes Figure 1 Flowchart of step S105 in FIG.

[0055] Figure 5 yes Figure 4 Flowchart of step S402 in FIG.

[0056] Figure 6 yes Figure 1 Flowchart before step S106 in FIG.

[0057] Figure 7 yes Figure 1 The flowchart after step S106 in FIG.

[0058] Figure 8is an example schematic diagram of the interactive behavior analysis method provided in an embodiment of the present application;

[0059] Figure 9 is a structural diagram of the interactive behavior analysis system provided in an embodiment of the present application;

[0060] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] First, let’s analyze some of the terms used in this application:

[0065] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0066] Large AI models are machine learning models with extremely large parameters (usually over a billion) and complex computational structures. They are typically capable of processing massive amounts of data and completing complex tasks such as natural language processing and image recognition.

[0067] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP stands at the intersection of computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. It is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent identification, information extraction and filtering, text classification and clustering, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0068] In the field of multimodal behavioral analysis, individual behaviors are typically captured for analysis of emotion, semantics, or health status, but the complexity and subtlety of interpersonal interactions, particularly when precise spatial and social dynamics analysis is required, are often overlooked. For example, a single person's speech can be analyzed for emotion and semantics, but a conversation between two people requires consideration of the potential for communication. Similarly, a person's positional changes in space can be detected, but when two people are involved, changes in distance between them and the nature of their interaction need to be considered.

[0069] Based on this, the embodiments of the present application provide an interactive behavior analysis method, system, electronic device and storage medium, which aim to identify and analyze complex interactive behaviors in fixed scenarios.

[0070] The interactive behavior analysis method, system, electronic device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the interactive behavior analysis method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0072] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0073] The interactive behavior analysis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The interactive behavior analysis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the interactive behavior analysis method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0075] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Figure 1 This is an optional flowchart of the interactive behavior analysis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0077] Step S101, acquiring positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0078] Step S102: Acquire spatial audio data; wherein the spatial audio data at least includes audio data of a target object;

[0079] Step S103, obtaining spatial video data corresponding to the target space;

[0080] Step S104: performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset includes at least two different shallow behavior feature data, and the shallow behavior feature data represents interactive behavior information between the target object and at least one other object;

[0081] Step S105: constructing a knowledge graph based on each shallow behavior feature data to obtain interactive behavior graph data;

[0082] Step S106: performing association analysis on the interactive behavior graph data using a pre-trained behavior analysis model to obtain interactive behavior analysis information.

[0083] In steps S101 to S106 shown in the embodiment of the present application, the positioning data of the target object in the target space is obtained to obtain the object positioning data, obtain the spatial audio data, and obtain the spatial video data corresponding to the target space. Then, a shallow behavior analysis is performed based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis by a pre-trained large behavior analysis model to obtain interactive behavior analysis information. Thus, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data by using shallow behavior feature data, associates the shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data by using a large behavior analysis model with behavioral knowledge data, thereby being able to identify and analyze complex interactive behaviors in fixed scenes.

[0084] In step S101 of some embodiments, the target space may be a teaching space, a medical space, a performance stage, etc., and the target object is the object of analysis in the target space by the present application method. The target object in the teaching space may be a teacher or a student, and the target object in the medical space may be a patient or a doctor. The teaching space will be used as the primary example of the target space for subsequent explanations. The operations in other target spaces such as the medical space and the performance stage are similar to those in the teaching space.

[0085] The present application can obtain the positioning data of the target object in the target space through wireless communication technology. For example, by deploying an ultra-wideband (UWB) positioning system in the classroom, equipping each teacher and student with a positioning tag, and obtaining the position of the teacher and students in the classroom in real time to obtain the object positioning data.

[0086] For behavioral analysis in this multi-person social scenario, it is necessary to focus on the interaction between people, especially the changes in the relative positions between people. Therefore, compared with other multimodal behavior analysis technologies, this application also introduces the position analysis modality. Through the method of high-precision wireless positioning technology, the position of teachers and students in the target space and the changes in their relative positions can be accurately obtained, thereby enriching the details of the interaction between people and improving the quality of identifying and analyzing complex interactive behaviors in fixed scenes.

[0087] In step S102 of some embodiments, a high-quality omnidirectional microphone array can be installed to capture audio signals in the target space. For example, spatial audio data can be obtained by capturing audio signals in a classroom, recording the teacher's lectures, questions, and instructions, as well as students' responses and discussions. The spatial audio data includes at least audio data of the target object.

[0088] In step S103 of some embodiments, multiple high-definition cameras can be installed in the target space, covering every corner of the target space, to obtain spatial video data. This facilitates capturing information about the target object's interactions with other objects, such as their actions, facial expressions, and other interactions. For example, in a teaching scenario, this facilitates capturing the actions, facial expressions, and gestures of teachers and students, obtaining rich interactive information.

[0089] In step S104 of some embodiments, shallow behavioral analysis is performed based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavioral feature dataset. The shallow behavioral feature dataset includes at least two different shallow behavioral feature data, each representing information about the interaction between the target object and at least one other object. For example, features such as the target object's movement pattern and distance change are extracted from the object positioning data; features such as voice activity, voice inflection, and speech frequency are extracted from the spatial audio data; and features such as posture estimation, facial expression, and gestures are extracted from the spatial video data.

[0090] See also Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S205:

[0091] Step S201, performing scene modeling on the target space according to the spatial video data to obtain spatial map data;

[0092] Step S202: extracting spatial video data related to the target object based on the object positioning data to obtain object video data;

[0093] Step S203, extracting facial features based on the object video data to obtain facial feature data;

[0094] Step S204, extracting spectrum features based on the spatial audio data to obtain spectrum feature data;

[0095] Step S205 , generating various shallow behavioral feature data based on the spatial map data, the object video data, the facial feature data, and the spectrum feature data.

[0096] In step S201 of some embodiments, scene modeling of the target space is performed by analyzing spatial video data, constructing a two-dimensional or three-dimensional map of the target space, and identifying key locations in the teaching space, such as the podium, student seats, and blackboard. This data helps understand the layout of the target space and the relative positions of target objects in the target space.

[0097] In step S202 of some embodiments, the object positioning data is used to extract spatial video data related to a target object (e.g., a teacher or a specific student). In this way, object video data of the target object can be obtained, which will be used for subsequent analysis to identify and understand the target object's behavior pattern in the target space.

[0098] In step S203 of some embodiments, facial feature extraction is performed on the target subject's video data to obtain facial feature data. This typically involves using computer vision techniques to identify and track key facial features, such as the locations of the eyes, nose, and mouth. By subsequently analyzing this facial feature data, it is possible to further understand the target subject's emotional state and possible interactive behaviors.

[0099] In step S204 of some embodiments, spectral features are extracted from the spatial audio data. This includes analyzing the frequency composition of the audio signal and its changes over time. Spectral feature data can help identify and analyze the target subject's speech content, voice inflection, and the reactions of other subjects interacting with the target subject, all of which are important clues to understanding the target subject's interactive behavior. Spectral features, such as Mel-Frequency Cepstral Coefficients (MFCCs), can be extracted using methods such as Fourier transforms.

[0100] In step S205 of some embodiments, the spatial map data, object video data, facial feature data, and spectral feature data are integrated to generate shallow behavioral feature data. This step is to fuse the various data extracted above to form a comprehensive behavioral feature data set.

[0101] See also Figure 3 In some embodiments, the shallow behavioral feature data includes any of the following: voice interaction recognition data, emotional interaction feature data, spatial movement data, and object interaction action data. Step S205 may include, but is not limited to, steps S301 to S304:

[0102] Step S301, performing speech recognition based on the spectrum feature data to obtain speech interaction recognition data;

[0103] Step S302, obtaining spatial movement data of the target object based on the spatial map data and the object positioning data;

[0104] Step S303, performing emotional feature analysis based on the spectral feature data and the facial feature data to obtain emotional interaction feature data;

[0105] Step S304, performing character motion analysis on the target object based on the object video data to obtain object interaction motion data;

[0106] In step S301 of some embodiments, the spectral feature data is converted into text information to identify and understand the speech content of the conversation between the target object and other objects, that is, speech interaction recognition data, and an acoustic neural network model or a natural language processing neural network model can be used for speech recognition.

[0107] In step S302 of some embodiments, the object positioning data is used to determine the coordinate position of the target object in the target space from the spatial map data, and spatial movement data of the target object in the target space is obtained. The position coordinates of the target object can be obtained using a positioning tag, and then analyzed in conjunction with the spatial map data of the target object to obtain the movement path of the target object in the target space.

[0108] In step S303 of some embodiments, an emotion recognition algorithm is used to analyze the prosodic features (such as pitch and intensity) in the spectral feature data and the facial expression features in the facial feature data to obtain the emotional interaction feature data of the target object and other objects during the interaction process. Machine learning or deep learning models, such as support vector machines (SVMs), random forests, or neural networks, are used to train models to recognize different emotional states (such as happiness, sadness, anger, etc.). These models can be trained based on audio data or facial video data to recognize specific emotional expressions.

[0109] In step S304 of some embodiments, the object interaction action data includes head orientation information, body posture information, gesture information, etc. Computer vision technologies, such as skeleton tracking and deep learning models, are used to extract action features from the object video data. For example, capturing a student's hand-raising, nodding, or shaking head movements can facilitate subsequent analysis of student engagement and response to course content. A dynamic skeleton model can be constructed using a spatiotemporal graph convolutional network (ST-GCN) to obtain posture and body movement information of the target object when interacting with other objects.

[0110] Through steps S301 to S304, the interactive behavior of the target object in the target space can be analyzed from multiple dimensions, providing a richer and more detailed behavioral feature data set to support subsequent in-depth behavioral analysis and interactive decision-making.

[0111] In step S105 of some embodiments, a knowledge graph is constructed based on each shallow behavioral feature data to obtain interactive behavior graph data. By integrating each shallow behavioral feature data, a comprehensive behavior graph is constructed, which can represent and analyze in detail the interactive behavior of the target object with other objects in the target space.

[0112] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S404:

[0113] Step S401: integrating the speech interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data of the same time series into graph data to generate graph node data of the corresponding time series;

[0114] Step S402: performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining other objects that interact with the target object, and obtaining interaction direction information;

[0115] Step S403: generating behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information;

[0116] Step S404: Generate a spatiotemporal graph based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

[0117] In step S401 of some embodiments, voice interaction recognition data, emotional interaction feature data, spatial movement data and object interaction action data are all related to time series. Time series can be regarded as unit time in seconds, and data of different modalities may be inconsistent in time, such as synchronization problems between video frames and audio signals. Solving these inconsistencies is crucial for accurate behavior recognition, so it is necessary to process the various shallow behavior feature data collected within the same time series. Based on the time series, each shallow behavior feature data is integrated into a graph data structure, and related graph node data is created based on the target object and other objects. These graph node data contain all relevant behavior features of the target object in a time series.

[0118] In step S402 of some embodiments, interactive object detection is performed based on the voice interaction recognition data, spatial movement data, and object interaction action data. This process involves analyzing the behavioral relationships between entities, such as conversations between teachers and students, or collaborations between students, to determine which other objects the target object is interacting with. In this way, directional information about interactions between entities can be obtained, which helps understand the social dynamics of the target object within the target space.

[0119] See also Figure 5 In some embodiments, step S402 may include but is not limited to steps S501 to S504:

[0120] Step S501: Acquire identity information of all objects in the target space, and obtain identity information of the interaction-directed object based on the identity information and voice interaction recognition data;

[0121] Step S502: performing visual direction recognition based on the object interaction action data to obtain interactive visual direction information;

[0122] Step S503: performing object trajectory recognition based on the spatial movement data to obtain the target object movement trajectory;

[0123] Step S504 : Based on the target object's movement trajectory, the interactive visual direction information, and the interactive pointing object's identity information, other objects that interact with the target object are determined to obtain interactive pointing information.

[0124] In step S501 of some embodiments, the identity information of all objects in the target space is obtained, and the identity information of the interactive pointing object is obtained by combining the identity information and the voice interaction recognition data to determine the pointing object of the target object in the dialogue interaction. This can identify who is speaking and to which object their dialogue is directed. For example, by analyzing the teacher's voice and the students' responses, it can be determined whether the teacher's explanation or question is directed at a specific student or the entire class. In some embodiments, spatial movement data or object video data can also be combined to assist in identification.

[0125] In step S502 of some embodiments, the subject interaction action data is used to perform eye gaze direction recognition to obtain interactive eye gaze direction information. This involves analyzing the facial orientation and eye gaze direction of the target subject in the video to determine their focus. For example, if a teacher continuously gazes toward a student during a lecture, or a student gazes at a specific peer during a group discussion, this interactive eye gaze direction information can provide important clues about the subject they are interacting with.

[0126] In some embodiments, the head orientation information in the character motion data and the eye movement information in the facial feature data can also be combined to more accurately determine the interactive visual direction information of the target object.

[0127] In some embodiments, step S503 of the present invention analyzes the spatial motion data to obtain the target object's movement trajectory within the target space. The target object's movement trajectory not only includes the target object's starting and ending positions, but also reflects information such as the target object's movement path, speed, and dwell time within the target space. For example, a teacher might move from the podium to a student's seat or patrol the classroom during class. These behaviors can be captured and analyzed through trajectory recognition.

[0128] The target object's movement trajectory not only provides basic data for subsequent interaction analysis but also reveals the interaction dynamics within the target space, thereby determining which objects the target object may have interacted with. For example, by analyzing a teacher's movement trajectory, educators can identify which teaching strategies are most effective or changes in student engagement and attention during class. This in-depth trajectory analysis provides important insights for optimizing teaching methods and improving student learning outcomes.

[0129] In some embodiments, identity information can be configured in the positioning tag carried by the target object, so that the positional relationship between each object and the corresponding identity information can be obtained more accurately in the spatial map data, and the object information of the target object interacting can also be obtained based on the object positioning data.

[0130] In step S504 of some embodiments, the target object's movement trajectory, interactive visual direction information, and interactive pointing object identity information are combined to determine other objects interacting with the target object. This step involves a comprehensive analysis of the previously collected data to obtain interactive pointing information. For example, if a teacher moves from the podium to a student's seat, and their conversation pointing information and visual direction information indicate that the teacher is communicating with the student, then it can be determined that an interaction has occurred between the teacher and the student.

[0131] Through steps S501 to S504, not only can direct dialogues and exchanges within the target space be identified, but non-verbal communication and indirect interactions, such as intentions and focus expressed through sight and body language, can also be revealed, thereby obtaining the interaction information of the target object and determining the interaction direction information of the target object. For teaching scenarios, such analysis provides educational researchers and practitioners with valuable information for in-depth understanding of teacher-student interaction and student engagement, which helps to optimize teaching strategies and improve teaching quality. In addition, this method also provides data support for the development of intelligent education assistance systems, enabling the system to more accurately capture and respond to interactive dynamics within the classroom.

[0132] In step S403 of some embodiments, the generated graph node data and interaction direction information are used to create behavior graph sub-data corresponding to a specific time series. These behavior graph sub-data provide snapshot information, showing the behavior state of the target object in the target space at a specific point in time, as well as the interaction relationship between the target object and other objects.

[0133] In step S404 of some embodiments, the behavioral graph sub-data of different time series are integrated to generate a complete spatiotemporal graph, namely, interactive behavior graph data. This spatiotemporal graph not only contains the behavioral characteristics of the target object at different time points, but also shows the evolution of the target object's behavior over time and the dynamic changes in the interaction between the target object and other objects. The interactive behavior graph data provides rich information for in-depth analysis of the interactive behavior of the target object in the target scene, which can be used to study the appropriateness of the interactive strategy, evaluate the interactive participation and optimize the interactive behavior. For teaching scenarios, it can be used to study the effectiveness of teaching strategies, evaluate students' learning participation, and optimize classroom management.

[0134] Through steps S401 to S404, graph node data is generated from each shallow behavioral feature data of the same time series, and behavioral graph sub-data is generated based on the interaction pointing information and graph node data. The interactive behavior graph data is generated from the behavioral graph sub-data of different time series to show the evolution of the target object's behavior over time and the dynamic changes in the interaction between the target object and other objects. For teaching scenarios, it not only enhances the understanding of interactive behavior in the classroom, but also provides strong data support for the development of intelligent education systems and the improvement of teaching methods. This graph-based approach makes complex interaction data structured and analyzable, bringing new insights and application possibilities to the field of education.

[0135] In step S106 of some embodiments, a pre-trained behavior analysis model is used to perform in-depth correlation analysis on the constructed interaction behavior graph data to obtain interaction behavior analysis information. Interaction behavior analysis information refers to the analysis provided by the behavior analysis model regarding the target object's interaction behavior within the target space. The behavior analysis model is a specialized artificial intelligence model trained using a large amount of data and capable of understanding and recognizing complex human interaction behaviors.

[0136] Using large behavioral analytics models, we can capture the complexity of interactions and analyze patterns and trends in the target audience's interactions. For example, we can identify the relationship between a teacher's movement patterns in the classroom and student engagement, or discover student responses to specific teaching activities. Furthermore, the model can analyze how interactions change over time, revealing dynamic shifts in classroom atmosphere and student emotions.

[0137] During the correlation analysis process, the behavioral analysis model not only identifies overt interactions but also reveals underlying behavioral correlations and causal relationships. For example, it might discover a positive correlation between the frequency of teacher questions and the enthusiasm of students in answering them. Such analytical results are extremely valuable for educators, providing a basis for improving teaching methods and enhancing the quality of classroom interactions.

[0138] See also Figure 6 In some embodiments, before step S106, the process further includes pre-training the behavior analysis model, which may include but is not limited to steps S601 to S605:

[0139] Step S601: Acquire behavior knowledge data; wherein the behavior knowledge data represents knowledge theory data related to the interaction behavior between at least two objects;

[0140] Step S602: Acquire a training data set; wherein the training data set includes behavior training data and behavior indicator information corresponding to the behavior training data;

[0141] Step S603: inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis model to perform behavior analysis and obtain training recognition information;

[0142] Step S604: performing error calculation based on the training identification information and the behavior indicator information to obtain comparison deviation data;

[0143] Step S605: Update the model parameters of the original behavior analysis model based on the comparison deviation data to obtain a pre-trained behavior analysis model.

[0144] In some embodiments, step S601 collects behavioral knowledge data. This data includes psychological theories of interactive behaviors, theoretical knowledge about micro-movements and micro-expressions, and empirical data on interactive behaviors that match the target scenario. For example, in a teaching scenario, behavioral knowledge data is typically based on educational theory, psychological research, and prior teaching experience, providing basic knowledge and theoretical frameworks about teacher-student interactions. This knowledge and theoretical data provides the foundation for the model to understand and interpret behaviors.

[0145] In step S602 of some embodiments, a training dataset is obtained. The training dataset includes a large amount of behavioral training data and behavioral indicator information corresponding to the data. The behavioral training data may include video, audio, and sensor data, while the behavioral indicator information provides specific labels and explanations for the data, such as the teacher's questioning behavior and the student's response behavior.

[0146] In step S603 of some embodiments, the behavioral training data and behavioral knowledge data are input into a pre-set primary behavioral analysis model. The primary behavioral analysis model is an artificial intelligence model, which can be a large language model or another type of general artificial intelligence model. Through this step, the primary behavioral analysis model can learn how to identify and analyze the interactions between the target object and other objects from the behavioral training data, generating training recognition information.

[0147] In step S604 of some embodiments, an error calculation is performed based on the training identification information and the behavioral indicator information to obtain comparison deviation data. This step is critical in the model training process, allowing the model's performance to be evaluated and its deficiencies in behavior recognition to be identified. By comparing the model's predicted results with the actual behavioral indicators, the model's error can be calculated and adjusted accordingly.

[0148] In step S605 of some embodiments, the comparison deviation data is used to update the model parameters of the original behavior analysis model. This process may involve gradient descent or other optimization algorithms to minimize model errors and improve model accuracy. Through repeated training and parameter updates, a pre-trained behavior analysis model is ultimately obtained, capable of accurately analyzing and identifying interactions between the target object and other objects.

[0149] Through steps S601 to S605, the behavior analysis model is ensured to have high accuracy and reliability in practical applications. This pre-trained behavior analysis model provides a solid foundation for the association analysis in step S106, and can obtain in-depth and accurate analysis results from the interaction behavior graph data, thus providing strong support for interaction behavior decision-making and interaction behavior improvement.

[0150] See also Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S703:

[0151] Step S701: Based on the interaction behavior analysis information, the interaction behavior of the target object is evaluated to obtain interaction evaluation information; wherein the interaction evaluation information indicates the area in the target space where the target object has interacted;

[0152] Step S702: Generate interaction suggestion information based on the interaction evaluation information;

[0153] Step S703: Push interaction suggestion information to the target object.

[0154] In step S701 of some embodiments, the interactive behavior of the target object is evaluated based on the interactive behavior analysis information. In this process, the system will quantitatively evaluate the effectiveness and quality of the interactive behavior based on the analysis information extracted from the interactive behavior knowledge graph, such as the frequency of teacher-student interaction, student participation, teacher's teaching strategy, etc. This evaluation may include a multi-dimensional analysis of the teacher's teaching methods, students' participation behavior, and classroom atmosphere. Through this evaluation, interactive evaluation information can be obtained, providing detailed feedback on teaching interactions, including advantages and areas for improvement.

[0155] Based on the interaction evaluation information, the system compares the target object's interaction area in the target space with the entire target space, evaluates the target object's attention to different areas of the target space, and evaluates the target object's interaction range based on the attention, and provides suggestions and feedback information.

[0156] In step S702 of some embodiments, targeted interaction suggestion information is given based on the interaction evaluation information. The interaction suggestion information is intended to help the target object optimize the interaction behavior, such as helping teachers optimize their teaching methods and improve students' learning outcomes. For example, if the interaction evaluation information shows that students' participation is not high, teachers may be advised to adopt more interactive teaching techniques, such as group discussions or gamification learning, to increase students' participation and interest. If the interaction evaluation information shows that the teacher's movement pattern in the classroom is related to the students' learning outcomes, teachers may be advised to adjust their range and methods of activities in the classroom in order to better interact with students. If the interaction evaluation information shows that the classroom tends to ignore a certain area in class, the classroom may be advised to pay attention to the students in this area in a timely manner and increase interaction with the students in this area.

[0157] In step S703 of some embodiments, the system evaluates the target object's interactive behavior based on a predetermined time interval and pushes interaction suggestion information to the target object, which can be in the form of text messages or app messages, or can be pushed to the target object through an interaction summary report.

[0158] Through steps S701 to S703, not only can in-depth insights into interactive behaviors be provided, but also practical and feasible interaction improvement suggestions can be provided to the target audience. For example, in teaching scenarios, this method can help achieve personalized and precise teaching, thereby improving teaching quality and student learning outcomes. In addition, this method provides valuable data resources for educational researchers and policymakers, helping them better understand the various dynamics of the teaching process and formulate more effective education policies and measures accordingly.

[0159] See also Figure 8 In an exemplary embodiment, this application uses a classroom as the target scene, deploying high-definition cameras, omnidirectional microphone arrays, and ultra-wideband positioning base stations. Positioning tags are assigned to teachers and students. These tags can be placed on wristbands or ID cards, such as campus ID cards. With the teacher as the target object, object positioning data, spatial audio data, and spatial video data of the teacher within the classroom are collected.

[0160] It should be noted that, according to existing research, teachers' nonverbal behavior has a significant impact on students' learning outcomes and learning experience. Nonverbal communication (NVC) refers to all elements of communication other than language. Paralanguage such as tone and volume are important acoustic factors in nonverbal communication; gestures, facial expressions, etc. are non-vocal elements in nonverbal communication. As much as 65% of the meaning of human interaction comes from nonverbal communication. In the field of education, there is a large amount of research on teachers' nonverbal behavior. For example, teachers' gestures can help improve students' learning engagement and effectiveness; teachers' spatial distance and movement methods can affect students' learning engagement, learning motivation, and learning experience and outcomes. Therefore, it is very important to analyze interactive behaviors in teaching scenarios through facial feature data and character movement data.

[0161] Then, shallow behavioral analysis is performed based on the collected data: the classroom scene is modeled through spatial video data to obtain spatial map data, and the spatial movement data of the teacher in the classroom can be further obtained, so as to clarify the specific location information of students and teachers in the classroom and the relative location information between teachers and students. This can be achieved through a graph convolutional neural network model; and the audio data is analyzed using features such as Mel-frequency cepstral coefficients to extract information such as the teacher's speaking speed, tone, and sound intensity, and speech recognition is performed to extract the conversation information between the teacher and students; then, through character recognition technology, spatial video data related to the teacher is extracted as object video data based on object positioning data. The object video data may also include students who interact with the teacher; using graph-based dynamic skeleton modeling, character action analysis is performed based on the object video data to extract information about the teacher's or student's posture and body movements, and gesture information can also be extracted based on the neural network model; and through face recognition technology, facial features are extracted based on the object video data to obtain facial feature information of the teacher or student. Further combined with the spectral feature data, the emotional feature information when the teacher interacts with the student can be analyzed.

[0162] Based on time series, shallow behavioral feature data is integrated and stored in a knowledge graph database according to chronological order. This generates graph node data. The objects interacting within the classroom are identified based on gestures, voice, and location information, generating information about the interaction between these objects. This information and the graph node data are then combined to generate behavior graph sub-data. The interaction graph sub-data, which contains information about various aspects of teacher-student interaction within the classroom over a sequential period of time, is then used to generate interactive behavior graph data.

[0163] Finally, the interaction behavior graph data is fed into a pre-trained AI model for in-depth analysis. The AI ​​model is first fine-tuned and trained on behavioral analysis based on behavioral knowledge data. This behavioral knowledge data typically consists of educational theory, psychological research, and prior teaching experience, providing the AI ​​model with foundational knowledge about teacher-student interactions.

[0164] After obtaining the interactive behavior analysis information derived from the artificial intelligence large model, the interactive behavior of the target object can be evaluated based on the interactive behavior analysis information, including the teacher's teaching quality, the student's learning effect, etc., and teaching suggestions and interactive evaluation information can be pushed to the teacher at intervals of 15 minutes, thereby providing feedback based on the teacher's teaching activities, helping teachers discover deficiencies in their own teaching, and promoting teachers' professional development.

[0165] The embodiment of the present application obtains the positioning data of the target object in the target space, obtains the object positioning data, obtains the spatial audio data, and obtains the spatial video data corresponding to the target space. Then, a shallow behavior analysis is performed based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis by a pre-trained large behavior analysis model to obtain interactive behavior analysis information. Therefore, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data through shallow behavior feature data, associates the shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a large behavior analysis model with behavioral knowledge data, thereby being able to identify and analyze complex interactive behaviors in fixed scenes.

[0166] See also Figure 9 The present application also provides an interactive behavior analysis system that can implement the above-mentioned interactive behavior analysis method. The system includes:

[0167] a positioning data acquisition module, configured to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is configured to interact with at least one other object in the target space;

[0168] An audio data acquisition module, configured to acquire spatial audio data; wherein the spatial audio data at least includes audio data of a target object;

[0169] A video data acquisition module is used to acquire spatial video data corresponding to the target space;

[0170] A shallow behavior analysis module is configured to perform shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset includes at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one other object;

[0171] The knowledge graph construction module is used to construct the knowledge graph based on various shallow behavioral feature data to obtain interactive behavior graph data;

[0172] The large model association analysis module is used to perform association analysis on the interactive behavior graph data through the pre-trained behavior analysis large model to obtain interactive behavior analysis information.

[0173] The specific implementation of the interactive behavior analysis system is basically the same as the specific embodiment of the above-mentioned interactive behavior analysis method, and will not be repeated here.

[0174] In some embodiments, the interactive behavior graph data will be stored in a knowledge graph database so that the relevant interactive behavior graph data can be extracted at any time into the action analysis model. At the same time, a knowledge database is also set up to store theoretical knowledge related to interaction and some empirical cases of interaction. For example, for teaching scenarios, theoretical knowledge related to classroom teaching and sub-communication can be stored, as well as some practical experience and cases of teachers and education experts, and some text descriptions and analyses of classroom videos or clips. The content of this knowledge database can be used to train or fine-tune the action analysis model so that the action analysis model can better analyze the interactive behavior of the target object and other objects in depth based on this theoretical knowledge.

[0175] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned interactive behavior analysis method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0176] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0177] The processor 1001 may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0178] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the interactive behavior analysis method of the embodiments of this application.

[0179] Input / output interface 1003, used to implement information input and output;

[0180] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0181] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0182] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0183] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned interactive behavior analysis method is implemented.

[0184] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0185] The interactive behavior analysis method, system, electronic device and storage medium provided in the embodiments of the present application obtain the positioning data of the target object in the target space, obtain the object positioning data, obtain the spatial audio data, and obtain the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data and spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis by a pre-trained large behavior analysis model to obtain interactive behavior analysis information. Therefore, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data by using shallow behavior feature data, associates the shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data by using a large behavior analysis model with behavioral knowledge data, thereby being able to identify and analyze complex interactive behaviors in fixed scenes.

[0186] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0187] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0188] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0189] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0190] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0191] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, and can be electrical, mechanical or other forms.

[0193] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0196] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for analyzing interactive behavior, characterized in that: The method comprises: Acquiring positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space; Acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object; Acquiring spatial video data corresponding to the target space; Performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset includes at least two different shallow behavior feature data, and the shallow behavior feature data represents interaction behavior information between the target object and at least one of the other objects; A knowledge graph is constructed based on each of the shallow behavior feature data to obtain interactive behavior graph data; wherein, the knowledge graph is constructed based on each of the shallow behavior feature data to obtain interactive behavior graph data, including: integrating the shallow behavior feature data of the same time series into graph data to generate graph node data corresponding to the time series; performing interactive object detection based on the shallow behavior feature data to determine the other objects that interact with the target object to obtain interactive pointing information; generating behavioral graph sub-data corresponding to the time series based on the graph node data and the interactive pointing information; generating a spatiotemporal graph based on the behavioral graph sub-data of different time series to obtain the interactive behavior graph data; The interactive behavior graph data is subjected to association analysis by a pre-trained behavioral analysis model to obtain interactive behavior analysis information.

2. The method according to claim 1, characterized in that The shallow behavior analysis is performed based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset, specifically including: Performing scene modeling on the target space according to the spatial video data to obtain spatial map data; extracting the spatial video data related to the target object based on the object positioning data to obtain object video data; Extract facial features based on the object video data to obtain facial feature data; Extracting spectrum features from the spatial audio data to obtain spectrum feature data; Respective shallow-layer behavior feature data are generated based on the space map data, the object video data, the facial feature data, and the spectrum feature data.

3. The method according to claim 2, characterized in that The shallow behavior feature data includes any one of the following: voice interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data. The shallow behavior feature data generated based on the spatial map data, the object video data, the facial feature data, and the spectral feature data include: Performing speech recognition based on the spectral feature data to obtain speech interaction recognition data; obtaining the spatial movement data about the target object based on the spatial map data and the object positioning data; Performing an emotion feature analysis based on the spectral feature data and the facial feature data to obtain the emotion interaction feature data; The target object is subjected to character motion analysis based on the object video data to obtain the object interaction motion data.

4. The method according to claim 3, characterized in that The voice interaction recognition data, the emotional interaction feature data, the spatial movement data, and the object interaction action data are all related to time series. The knowledge graph is constructed based on each of the shallow behavior feature data to obtain the interaction behavior graph data, including: Performing graph data integration on the voice interaction recognition data, the emotion interaction feature data, the spatial movement data, and the object interaction action data of the same time series to generate graph node data corresponding to the time series; Performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining the other objects interacting with the target object, and obtaining interaction pointing information; Generate behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information; A spatiotemporal graph is generated based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

5. The method according to claim 4, characterized in that The performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining the other objects interacting with the target object, and obtaining interaction pointing information includes: Acquire identity information of all objects in the target space, and obtain identity information of the interaction-directed object based on the identity information and the voice interaction recognition data; Perform visual direction recognition based on the object interaction action data to obtain interactive visual direction information; Performing object trajectory recognition based on the spatial movement data to obtain a target object movement trajectory; Based on the target object's movement trajectory, the interactive visual direction information, and the interactive pointing object's identity information, the other objects interacting with the target object are determined to obtain the interactive pointing information.

6. The method according to any one of claims 1 to 5, characterized in that Before performing association analysis on the interactive behavior graph data using the pre-trained behavior analysis model to obtain interactive behavior analysis information, the method further includes pre-training the behavior analysis model, specifically including: Acquiring behavioral knowledge data; wherein the behavioral knowledge data represents knowledge theory data related to interactive behavior between at least two objects; Acquire a training data set; wherein the training data set includes behavioral training data and behavioral indicator information corresponding to the behavioral training data; Inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis model to perform behavior analysis to obtain training recognition information; Perform error calculation based on the training identification information and the behavior indicator information to obtain comparison deviation data; The model parameters of the original behavior analysis large model are updated based on the comparison deviation data to obtain the pre-trained behavior analysis large model.

7. The method according to any one of claims 1 to 5, characterized in that After performing correlation analysis on the interactive behavior graph data using the pre-trained behavior analysis model to obtain interactive behavior analysis information, the method includes: Based on the interaction behavior analysis information, the interaction behavior of the target object is evaluated to obtain interaction evaluation information; wherein the interaction evaluation information indicates the area in the target space where the target object has interacted; generating interaction suggestion information based on the interaction evaluation information; Push the interaction suggestion information to the target object.

8. An interactive behavior analysis system, characterized in that: The system comprises: a positioning data acquisition module, configured to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is configured to interact with at least one other object in the target space; An audio data acquisition module, configured to acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object; A video data acquisition module, configured to acquire spatial video data corresponding to the target space; a shallow behavior analysis module, configured to perform shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature dataset; the shallow behavior feature dataset comprising at least two different shallow behavior feature data, each of which represents interactive behavior information between the target object and at least one of the other objects; A knowledge graph construction module is used to construct a knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data; wherein, the construction of the knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data includes: integrating the shallow behavior feature data of the same time series into graph data to generate graph node data corresponding to the time series; performing interactive object detection based on the shallow behavior feature data to determine the other objects that interact with the target object to obtain interactive pointing information; generating behavioral graph sub-data corresponding to the time series based on the graph node data and the interactive pointing information; generating a spatiotemporal graph based on the behavioral graph sub-data of different time series to obtain the interactive behavior graph data; The large model association analysis module is used to perform association analysis on the interactive behavior graph data through a pre-trained behavior analysis large model to obtain interactive behavior analysis information.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the interactive behavior analysis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the interactive behavior analysis method according to any one of claims 1 to 7 is implemented.

Images