Interaction behavior analysis method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an interaction behavior analysis method and system, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring positioning data of a target object in a target space to obtain object positioning data; acquiring spatial audio data; acquiring space video data corresponding to the target space; performing shallow behavior analysis according to the object positioning data, the spatial audio data and the spatial video data to obtain a shallow behavior feature data set; performing knowledge graph construction based on the shallow behavior feature data to obtain interactive behavior graph data; and performing association analysis on the interaction behavior map data through a pre-trained behavior analysis large model to obtain interaction behavior analysis information. According to the embodiment of the invention, complex interaction behaviors in a fixed scene can be identified and analyzed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interactive behavior analysis method, system, electronic device and storage medium. Background Art

[0002] Behavior recognition refers to identifying and understanding human behavior patterns by analyzing data such as images or sensors. Multimodal analysis is particularly important in behavior recognition because it involves using multiple types of data and information sources to improve the accuracy and robustness of behavior recognition.

[0003] In related technologies, multimodal data analysis can easily obtain simple behavioral information, but it is challenging for complex social scenarios. For example, in teaching scenarios, there are complex interactions between students and teachers, and simple behavioral analysis results do not have practical value and significance. Therefore, how to identify and analyze complex interactive behaviors in fixed scenarios has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an interactive behavior analysis method, system, electronic device and storage medium, aiming to identify and analyze complex interactive behaviors in fixed scenes.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides an interactive behavior analysis method, the method comprising: acquiring positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0006] Acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object;

[0007] Acquire spatial video data corresponding to the target space;

[0008] Performing shallow behavior analysis according to the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one of the other objects;

[0009] Constructing a knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data;

[0010] The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis model to obtain interactive behavior analysis information.

[0011] In some embodiments, performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set specifically includes:

[0012] Performing scene modeling on the target space according to the spatial video data to obtain spatial map data;

[0013] extracting the spatial video data related to the target object based on the object positioning data to obtain object video data;

[0014] Extract facial features based on the object video data to obtain facial feature data;

[0015] Extracting frequency spectrum features according to the spatial audio data to obtain frequency spectrum feature data;

[0016] Respective shallow behavior feature data are generated based on the space map data, the object video data, the facial feature data and the spectrum feature data.

[0017] In some embodiments, the shallow behavior feature data includes any one of the following: voice interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data, and the generating of each shallow behavior feature data based on the spatial map data, the object video data, the facial feature data, and the spectrum feature data includes:

[0018] Perform speech recognition based on the frequency spectrum feature data to obtain speech interaction recognition data;

[0019] Based on the spatial map data and the object positioning data, obtaining the spatial movement data about the target object;

[0020] Performing an emotion feature analysis based on the frequency spectrum feature data and the facial feature data to obtain the emotion interaction feature data;

[0021] The target object is subjected to a character motion analysis based on the object video data to obtain the object interaction motion data.

[0022] In some embodiments, the voice interaction recognition data, the emotion interaction feature data, the spatial movement data, and the object interaction action data are all related to a time series, and the knowledge graph is constructed based on each of the shallow behavior feature data to obtain the interaction behavior graph data, including:

[0023] Performing graph data integration on the voice interaction recognition data, the emotion interaction feature data, the spatial movement data, and the object interaction action data of the same time series to generate graph node data corresponding to the time series;

[0024] Performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interactive action data, determining the other objects that interact with the target object, and obtaining interactive pointing information;

[0025] Generate behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information;

[0026] A spatiotemporal graph is generated based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

[0027] In some embodiments, the performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interaction action data, determining the other objects interacting with the target object, and obtaining the interaction pointing information includes:

[0028] Acquire identity information of all objects in the target space, and obtain identity information of interaction-directed objects based on the identity information and the voice interaction recognition data;

[0029] Perform visual direction recognition based on the object interaction action data to obtain interactive visual direction information;

[0030] Performing object trajectory recognition based on the spatial movement data to obtain a target object movement trajectory;

[0031] Based on the movement trajectory of the target object, the interactive visual direction information and the interactive pointing object identity information, the other objects that interact with the target object are determined to obtain the interactive pointing information.

[0032] In some embodiments, before performing association analysis on the interactive behavior graph data using a pre-trained behavior analysis model to obtain interactive behavior analysis information, the method further includes pre-training the behavior analysis model, specifically including:

[0033] Acquire behavior knowledge data; wherein the behavior knowledge data represents knowledge theory data related to the interaction behavior between at least two objects;

[0034] Acquire a training data set; wherein the training data set includes behavior training data and behavior indicator information corresponding to the behavior training data;

[0035] Inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis large model to perform behavior analysis to obtain training recognition information;

[0036] Perform error calculation based on the training identification information and the behavior indicator information to obtain comparison deviation data;

[0037] In some embodiments, after performing association analysis on the interactive behavior graph data using the pre-trained behavior analysis model to obtain interactive behavior analysis information, the method includes:

[0038] Based on the interaction behavior analysis information, the interaction behavior of the target object is evaluated to obtain interaction evaluation information; wherein the interaction evaluation information indicates the area in the target space where the target object has interacted;

[0039] generating interaction suggestion information based on the interaction evaluation information;

[0040] The interaction suggestion information is pushed to the target object.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an interactive behavior analysis system, the system comprising:

[0042] A positioning data acquisition module, used to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0043] An audio data acquisition module, used to acquire spatial audio data; wherein the spatial audio data at least includes the audio data of the target object;

[0044] A video data acquisition module, used to acquire spatial video data corresponding to the target space;

[0045] A shallow behavior analysis module, configured to perform shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents interaction behavior information between the target object and at least one of the other objects;

[0046] A knowledge graph construction module, used to construct a knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data;

[0047] The large model association analysis module is used to perform association analysis on the interactive behavior map data through a pre-trained behavior analysis large model to obtain interactive behavior analysis information.

[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the interactive behavior analysis method described in the first aspect when executing the computer program.

[0049] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the interactive behavior analysis method described in the first aspect.

[0050] The interactive behavior analysis method, system, electronic device and storage medium proposed in the present application obtain the positioning data of the target object in the target space, obtain the object positioning data, obtain the spatial audio data, and obtain the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data and spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis large model to obtain interactive behavior analysis information. Therefore, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data through shallow behavior feature data, associates shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a behavior analysis large model with behavior knowledge data, so as to identify and analyze complex interactive behaviors in fixed scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the interactive behavior analysis method provided in an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Flow chart of step S104 in FIG.

[0053] Figure 3 yes Figure 2 Flow chart of step S205 in FIG.

[0054] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0055] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0056] Figure 6 yes Figure 1 The flowchart before step S106 in FIG.

[0057] Figure 7 yes Figure 1 The flowchart after step S106 in FIG.

[0058] Figure 8is an example schematic diagram of the interactive behavior analysis method provided in an embodiment of the present application;

[0059] Fig. 9 is a structural diagram of an interactive behavior analysis system provided in an embodiment of the present application;

[0060] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] A large AI model refers to a machine learning model with ultra-large-scale parameters (usually more than one billion) and complex computing structures. It is usually able to process massive amounts of data and complete various complex tasks, such as natural language processing and image recognition.

[0067] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0068] In the field of multimodal behavior analysis, individual behaviors are usually captured to analyze emotions, semantics, or health status, but the complexity and subtlety of interactions between people are often not paid attention to, especially when precise spatial and social dynamic analysis is required. For example, a person's speech can be analyzed for emotions and semantics, but when it comes to a conversation between two people, the possibility of communication needs to be considered. Similarly, the change of a person's position in space can be detected, but when it comes to two people, the change of distance between them and the nature of the interaction needs to be considered.

[0069] Based on this, the embodiments of the present application provide an interactive behavior analysis method, system, electronic device and storage medium, which aim to identify and analyze complex interactive behaviors in fixed scenes.

[0070] The interactive behavior analysis method, system, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the interactive behavior analysis method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] The interactive behavior analysis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The interactive behavior analysis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the interactive behavior analysis method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Figure 1 is an optional flowchart of the interactive behavior analysis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0077] Step S101, acquiring positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0078] Step S102, acquiring spatial audio data; wherein the spatial audio data at least includes audio data of a target object;

[0079] Step S103, obtaining spatial video data corresponding to the target space;

[0080] Step S104, performing shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one other object;

[0081] Step S105, constructing a knowledge graph based on each shallow behavior feature data to obtain interactive behavior graph data;

[0082] Step S106, performing association analysis on the interactive behavior graph data using a pre-trained behavior analysis model to obtain interactive behavior analysis information.

[0083] Steps S101 to S106 shown in the embodiment of the present application are obtained by obtaining the positioning data of the target object in the target space, obtaining the object positioning data, obtaining the spatial audio data, and obtaining the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavior feature data set. A knowledge graph is then constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis large model to obtain interactive behavior analysis information. Thus, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data through shallow behavior feature data, associates shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a behavior analysis large model with behavioral knowledge data, so as to identify and analyze complex interactive behaviors in fixed scenes.

[0084] In step S101 of some embodiments, the target space may be a teaching space, a medical care space, a performance stage, etc., and the target object is the object that the method of the present application focuses on analyzing in the target space. The target object in the teaching space may be a teacher or a student, and the target object in the medical care space may be a patient or a doctor. The teaching space will be used as the main example of the target space for explanation later, and the operations in other target spaces such as medical care spaces and performance stages are similar to those in the teaching space.

[0085] The present application can obtain the positioning data of the target object in the target space through wireless communication technology, for example, by deploying an ultra-wideband (Ultra Wi de Band, UWB) positioning system in the classroom, equipping each teacher and student with a positioning tag, and obtaining the position of the teacher and students in the classroom in real time to obtain the object positioning data.

[0086] For the behavior analysis in this multi-person social scene, it is necessary to focus on the interaction between people, especially the changes in the relative positions between people. Therefore, compared with other multimodal behavior analysis technologies, this application also introduces the position analysis mode. Through the method of high-precision wireless positioning technology, the positions of teachers and students in the target space and the changes in their relative positions can be accurately obtained, thereby enriching the interaction details between people and improving the quality of identifying and analyzing complex interactive behaviors in fixed scenes.

[0087] In step S102 of some embodiments, a high-quality omnidirectional microphone array may be installed to capture audio signals in the target space, for example, by capturing audio signals in a classroom, recording the teacher's lectures, questions, instructions, and students' responses and discussions, and obtaining spatial audio data. The spatial audio data includes at least audio data of the target object.

[0088] In step S103 of some embodiments, multiple high-definition cameras may be installed in the target space to cover every corner of the target space to obtain spatial video data, which is conducive to capturing the target object and other objects' interactive information such as behavior, facial expressions, etc. Taking the teaching scene as an example, it is conducive to capturing the behavior, facial expressions and postures of teachers and students and obtaining rich interactive information.

[0089] In step S104 of some embodiments, shallow behavior analysis is performed based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavior feature data set. The shallow behavior feature data set contains at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one other object. For example, the target object's movement mode, distance change and other features are extracted from the object positioning data; the voice activity, tone change, speech frequency and other features are extracted from the spatial audio data; the posture estimation, facial expression, gesture and other features are extracted from the spatial video data.

[0090] See also Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S205:

[0091] Step S201, performing scene modeling on the target space according to the spatial video data to obtain spatial map data;

[0092] Step S202, extracting spatial video data related to the target object based on the object positioning data to obtain object video data;

[0093] Step S203, extracting facial features based on the object video data to obtain facial feature data;

[0094] Step S204, extracting spectrum features according to the spatial audio data to obtain spectrum feature data;

[0095] Step S205, generating various shallow behavior feature data based on the space map data, the object video data, the facial feature data and the spectrum feature data.

[0096] In step S201 of some embodiments, scene modeling of the target space is performed by analyzing the spatial video data, a two-dimensional or three-dimensional map of the target space is constructed, and key positions are marked to obtain spatial map data. Such as the podium, student seats, blackboard, etc. in the teaching space. These data are helpful to understand the layout of the target space and the relative position of the target object in the target space.

[0097] In step S202 of some embodiments, the object positioning data is used to extract spatial video data related to the target object (such as a teacher or a specific student). In this way, the object video data of the target object can be obtained, which will be used for subsequent analysis to identify and understand the behavior pattern of the target object in the target space.

[0098] In step S203 of some embodiments, facial features are extracted from the video data of the target object to obtain facial feature data. This usually involves using computer vision technology to identify and track key facial feature points, such as the positions of the eyes, nose and mouth. By analyzing these facial feature data, the emotional state and possible interactive behaviors of the target object can be further understood.

[0099] In step S204 of some embodiments, spectral features are extracted from the spatial audio data. This includes analyzing the frequency composition of the audio signal and its changes over time. Spectral feature data can help identify and analyze the target object's speech content, intonation changes, and the reactions of other objects interacting with the target object, which are important clues for understanding the target object's interactive behavior. Spectral features can be extracted by methods such as Fourier transform, such as Mel-frequency cepstral coefficients (MFCCs).

[0100] In step S205 of some embodiments, the spatial map data, object video data, facial feature data and spectrum feature data are integrated to generate shallow behavior feature data. This step is to fuse the various data extracted above to form a comprehensive behavior feature data set.

[0101] See also Figure 3 In some embodiments, the shallow behavior feature data includes any of the following: voice interaction recognition data, emotion interaction feature data, space movement data, and object interaction action data. Step S205 may include, but is not limited to, steps S301 to S304:

[0102] Step S301, performing speech recognition based on the frequency spectrum feature data to obtain speech interaction recognition data;

[0103] Step S302, obtaining spatial movement data about the target object based on the spatial map data and the object positioning data;

[0104] Step S303, performing emotional feature analysis based on the spectrum feature data and the facial feature data to obtain emotional interaction feature data;

[0105] Step S304, performing character motion analysis on the target object based on the object video data to obtain object interaction motion data;

[0106] In step S301 of some embodiments, the spectral feature data is converted into text information to recognize and understand the conversational speech content between the target object and other objects, i.e., speech interaction recognition data. An acoustic neural network model or a natural language processing neural network model may be used for speech recognition.

[0107] In step S302 of some embodiments, the coordinate position of the target object in the target space is determined in the spatial map data using the object positioning data, and the spatial movement data of the target object in the target space is obtained. The position coordinates of the target object can be obtained using the positioning tag, and then analyzed in combination with the spatial map data of the target object to obtain the movement path of the target object in the target space.

[0108] In step S303 of some embodiments, the emotion recognition algorithm is used to analyze the rhythmic features (such as pitch, intensity) in the spectral feature data and the facial expression features in the facial feature data to obtain the emotional interaction feature data of the target object and other objects during the interaction process. Using machine learning or deep learning models, such as support vector machines (SVM), random forests or neural networks, the model is trained to recognize different emotional states (such as happiness, sadness, anger, etc.). These models can be trained based on audio data or facial video data to recognize specific emotional expressions.

[0109] In step S304 of some embodiments, the object interaction action data includes head orientation information, body posture information, gesture information, etc. Computer vision technology, such as skeleton tracking and deep learning models, is used to extract action features from object video data. For example, capturing the action of students raising their hands, nodding or shaking their heads, which helps to analyze the students' participation and response to the course content. A dynamic skeleton model can be constructed through a spatiotemporal graph convolutional network (ST-GCN) to obtain the posture and body movement information of the target object when it interacts with other objects.

[0110] Through steps S301 to S304, the interactive behavior of the target object in the target space can be analyzed from multiple dimensions, providing a richer and more detailed behavioral feature data set to provide support for subsequent in-depth behavioral analysis and interactive decision-making.

[0111] In step S105 of some embodiments, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. By integrating each shallow behavior feature data, a comprehensive behavior graph is constructed, which can represent and analyze the interactive behavior of the target object with other objects in the target space in detail.

[0112] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S404:

[0113] Step S401, integrating the speech interaction recognition data, emotion interaction feature data, spatial movement data and object interaction action data of the same time series into graph data to generate graph node data of the corresponding time series;

[0114] Step S402, performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interactive action data, determining other objects that interact with the target object, and obtaining interactive pointing information;

[0115] Step S403, generating behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information;

[0116] Step S404, generating a spatiotemporal graph based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

[0117] In step S401 of some embodiments, voice interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data are all related to time series. The time series can be regarded as a unit time in seconds, and data of different modalities may be inconsistent in time, such as synchronization problems between video frames and audio signals. Solving these inconsistencies is crucial for accurate behavior recognition, so it is necessary to process the shallow behavior feature data collected in the same time series. Based on the time series, each shallow behavior feature data is integrated into a graph data structure, and related graph node data is created based on the target object and other objects. These graph node data contain all relevant behavior features of the target object in a time series.

[0118] In step S402 of some embodiments, interactive object detection is performed based on voice interaction recognition data, spatial movement data, and object interaction action data. This process involves analyzing behavioral relationships between entities, such as conversations between teachers and students, cooperation between students, etc., to determine which other objects the target object is interacting with. In this way, directional information about the interaction between entities can be obtained, which helps to understand the social dynamics of the target object in the target space.

[0119] See also Figure 5 In some embodiments, step S402 may include but is not limited to steps S501 to S504:

[0120] Step S501, obtaining identity information of all objects in the target space, and obtaining identity information of the interaction-pointing object based on the identity information and voice interaction recognition data;

[0121] Step S502, performing visual direction recognition based on the object interaction action data to obtain interactive visual direction information;

[0122] Step S503, performing object trajectory recognition based on the spatial movement data to obtain the movement trajectory of the target object;

[0123] Step S504, based on the target object movement trajectory, the interactive visual direction information and the interactive pointing object identity information, other objects that interact with the target object are determined to obtain interactive pointing information.

[0124] In step S501 of some embodiments, the identity information of all objects in the target space is obtained, and the identity information and the voice interaction recognition data are combined to obtain the interactive pointing object identity information to determine the pointing object for the target object to interact with. In this way, it is possible to identify who is speaking and which object their conversation is directed to. For example, by analyzing the teacher's voice and the student's response, it can be determined whether the teacher's explanation or question is directed to a specific student or the entire class. In some embodiments, spatial movement data or object video data can also be combined to assist in identification.

[0125] In step S502 of some embodiments, the object interaction action data is used to perform visual direction recognition to obtain interactive visual direction information. This involves analyzing the facial orientation and eye direction of the target object in the video to determine the focus of their attention. For example, if a teacher keeps looking at a certain student during a lecture, or a student looks at a specific peer during a group discussion, these interactive visual direction information are important clues about their interactive objects.

[0126] In some embodiments, the head orientation information in the character motion data and the eye movement information in the facial feature data may also be combined to more accurately determine the interactive visual direction information of the target object.

[0127] In step S503 of some embodiments, by analyzing the spatial movement data, the movement trajectory of the target object in the target space can be obtained. The movement trajectory of the target object not only includes the starting position and the ending position of the target object, but also reflects the movement path, speed, and residence time of the target object in the target space. For example, a teacher may move from the podium to the student's seat or patrol the classroom in class. These behaviors can be captured and analyzed through trajectory recognition.

[0128] The target object's movement trajectory not only provides basic data for subsequent interactive behavior analysis, but also reveals the interactive dynamics within the target space, thereby determining which objects the target object may have interacted with. For example, by analyzing the teacher's movement trajectory, educators can identify which teaching strategies are most effective, or the changes in students' participation and attention in class. This in-depth trajectory analysis provides an important reference for optimizing teaching methods and improving student learning outcomes.

[0129] In some embodiments, identity information can be configured in the positioning tag carried by the target object, so that the positional relationship between objects and the corresponding identity information can be obtained more accurately in the spatial map data, and the object information of the target object interacting can also be obtained based on the object positioning data.

[0130] In step S504 of some embodiments, the target object movement trajectory, the interactive visual direction information, and the interactive pointing object identity information are combined to determine other objects that interact with the target object. This step involves a comprehensive analysis of the previously collected data to obtain interactive pointing information. For example, if a teacher moves from a podium to a student's seat, and their conversation pointing information and visual direction information indicate that the teacher is communicating with the student, then it can be determined that an interaction has occurred between the teacher and the student.

[0131] Through steps S501 to S504, not only direct dialogue and communication in the target space can be identified, but also non-verbal communication and indirect interaction, such as intentions and focus expressed through sight and body language, can be revealed, thereby obtaining the interaction information of the target object and determining the interaction direction information of the target object. For teaching scenarios, such analysis provides educational researchers and practitioners with valuable information for in-depth understanding of teacher-student interaction and student participation, which helps to optimize teaching strategies and improve teaching quality. In addition, this method also provides data support for the development of intelligent education assistance systems, enabling the system to more accurately capture and respond to interactive dynamics in the classroom.

[0132] In step S403 of some embodiments, the generated graph node data and interaction pointing information are used to create behavior graph sub-data corresponding to a specific time series. These behavior graph sub-data provide a snapshot information, showing the behavior state of the target object in the target space at a specific time point, and the interaction relationship between the target object and other objects.

[0133] In step S404 of some embodiments, the behavior map sub-data of different time series are integrated to generate a complete spatiotemporal graph, i.e., interactive behavior map data. This spatiotemporal graph not only contains the behavioral characteristics of the target object at different time points, but also shows the evolution of the behavior of the target object over time and the dynamic changes of the interaction between the target object and other objects. The interactive behavior map data provides rich information for in-depth analysis of the interactive behavior of the target object in the target scene, which can be used to study the appropriateness of the interactive strategy, evaluate the interactive participation and optimize the interactive behavior. For teaching scenarios, it can be used to study the effectiveness of teaching strategies, evaluate students' learning participation, and optimize classroom management.

[0134] Through steps S401 to S404, graph node data is generated through each shallow behavior feature data of the same time series, and behavior graph sub-data is generated based on the interaction pointing information and graph node data. The interactive behavior graph data is generated through the behavior graph sub-data of different time series to show the evolution of the behavior of the target object over time and the dynamic changes of the interaction between the target object and other objects. For teaching scenarios, it not only enhances the understanding of interactive behaviors in the classroom, but also provides strong data support for the development of intelligent education systems and the improvement of teaching methods. This graph-based approach makes complex interaction data structured and analyzable, bringing new insights and application possibilities to the field of education.

[0135] In step S106 of some embodiments, the pre-trained behavior analysis big model is used to perform in-depth correlation analysis on the constructed interactive behavior map data to obtain interactive behavior analysis information. Interactive behavior analysis information refers to the analysis content given by the behavior analysis big model for the interactive behavior of the target object in the target space. The behavior analysis big model is a specialized trained artificial intelligence big model, which is trained with a large amount of data and can understand and recognize complex human interactive behaviors.

[0136] The use of behavioral analysis models can capture the complexity of interactive behaviors and analyze the patterns and trends of the target object's interactive behaviors. For example, it can identify the relationship between the teacher's movement pattern in the classroom and the student's participation, or discover the student response triggered by a specific teaching activity. In addition, the model can also analyze the changes in interactive behaviors over time, revealing the dynamic changes in classroom atmosphere and student emotions.

[0137] In the process of correlation analysis, the behavior analysis model can not only identify obvious interactive behaviors, but also reveal potential behavioral associations and causal relationships. For example, it may find that there is a positive correlation between the frequency of teachers asking questions and the enthusiasm of students to answer questions. Such analysis results are very valuable for educators and can provide a basis for improving teaching methods and improving the quality of classroom interaction.

[0138] See also Figure 6 In some embodiments, before step S106, the behavior analysis model is pre-trained, which may include but is not limited to steps S601 to S605:

[0139] Step S601, acquiring behavior knowledge data; wherein the behavior knowledge data represents knowledge theory data related to the interaction behavior between at least two objects;

[0140] Step S602, obtaining a training data set; wherein the training data set includes behavior training data and behavior indicator information corresponding to the behavior training data;

[0141] Step S603, inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis large model to perform behavior analysis and obtain training recognition information;

[0142] Step S604, performing error calculation based on the training identification information and the behavior index information to obtain comparison deviation data;

[0143] Step S605, updating the model parameters of the original behavior analysis large model based on the comparison deviation data to obtain a pre-trained behavior analysis large model.

[0144] In step S601 of some embodiments, behavioral knowledge data is collected, and the behavioral knowledge data includes theoretical knowledge such as psychological theories of interactive behaviors, micro-movements, micro-expressions, and experience data of interactive behaviors matching the target scene. For example, for teaching scenes, behavioral knowledge data is usually based on educational theories, psychological research, and previous teaching experience, providing basic knowledge and theoretical frameworks about teacher-student interactive behaviors. These knowledge and theoretical data provide the basis for the model to understand and explain behaviors.

[0145] In step S602 of some embodiments, a training data set is obtained, which includes a large amount of behavioral training data and behavioral indicator information corresponding to the data. The behavioral training data may include video, audio and sensor data, and the behavioral indicator information provides specific labels and explanations for the data, such as the teacher's questioning behavior, the student's response behavior, etc.

[0146] In step S603 of some embodiments, these behavior training data and behavior knowledge data are input into a preset original behavior analysis big model. The original behavior analysis big model is an artificial intelligence big model, which can be a big language model or other types of general artificial intelligence big models. Through this step, the original behavior analysis big model can learn how to identify and analyze the interaction between the target object and other objects from the behavior training data, and generate training recognition information.

[0147] In step S604 of some embodiments, error calculation is performed based on the training identification information and the behavior indicator information to obtain comparison deviation data. This step is key in the model training process, thereby evaluating the performance of the model and identifying the deficiencies of the model in behavior recognition. By comparing the prediction results of the model with the actual behavior indicators, the error of the model can be calculated and the model can be adjusted accordingly.

[0148] In step S605 of some embodiments, the model parameters of the original behavior analysis model are updated using the comparison deviation data. This process may involve gradient descent or other optimization algorithms, the purpose of which is to minimize the model error and improve the accuracy of the model. Through repeated training and parameter updates, a pre-trained behavior analysis model is eventually obtained, which can accurately analyze and identify the interaction between the target object and other objects.

[0149] Through steps S601 to S605, it is ensured that the behavior analysis model has high accuracy and reliability in practical applications. This pre-trained behavior analysis model provides a solid foundation for the association analysis in step S106, and can obtain in-depth and accurate analysis results from the interactive behavior map data, thereby providing strong support for interactive behavior decision-making and interactive behavior improvement.

[0150] See also Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S703:

[0151] Step S701, based on the interactive behavior analysis information, evaluating the interactive behavior of the target object to obtain interactive evaluation information; wherein the interactive evaluation information indicates the area where the target object has interacted in the target space;

[0152] Step S702, generating interaction suggestion information based on the interaction evaluation information;

[0153] Step S703: Push interaction suggestion information to the target object.

[0154] In step S701 of some embodiments, the interactive behavior of the target object is evaluated based on the interactive behavior analysis information. In this process, the system will quantitatively evaluate the effectiveness and quality of the interactive behavior based on the analysis information extracted from the interactive behavior knowledge graph, such as the frequency of teacher-student interaction, student participation, teacher's teaching strategy, etc. This evaluation may include a multi-dimensional analysis of the teacher's teaching methods, students' participation behavior, and classroom atmosphere. Through this evaluation, interactive evaluation information can be obtained, providing detailed feedback on teaching interactions, including advantages and areas for improvement.

[0155] Based on the comparison between the target object's interacted area in the target space and the entire area of ​​the target space in the interaction evaluation information, the system evaluates the target object's attention to different areas of the target space, and evaluates the target object's interaction scope based on the attention, and gives suggestions and feedback information.

[0156] In step S702 of some embodiments, targeted interaction suggestion information is given based on the interaction evaluation information. The interaction suggestion information is intended to help the target object optimize the interaction behavior, such as helping teachers optimize their teaching methods and improve students' learning outcomes. For example, if the interaction evaluation information shows that students' participation is not high, teachers may be advised to adopt more interactive teaching techniques, such as group discussions or gamification learning, to increase students' participation and interest. If the interaction evaluation information shows that the teacher's movement pattern in the classroom is related to the students' learning outcomes, teachers may be advised to adjust their range and manner of activities in the classroom in order to better interact with students. If the interaction evaluation information shows that the classroom tends to ignore a certain area in class, the classroom may be advised to pay attention to the students in this area in a timely manner and increase interaction with the students in this area.

[0157] In step S703 of some embodiments, the system evaluates the target object's interactive behavior based on a predetermined time interval and pushes interactive suggestion information to the target object, which may be in the form of text messages or app messages, or in the form of interactive summary reports.

[0158] Through steps S701 to S703, not only can in-depth insights into interactive behaviors be provided, but also practical and feasible interaction improvement suggestions can be provided to the target object. For example, for teaching scenarios, this method helps to personalize and accurately teach, thereby improving teaching quality and student learning outcomes. In addition, this method also provides valuable data resources for educational researchers and decision makers, helping them to better understand the various dynamics in the teaching process and formulate more effective educational policies and measures accordingly.

[0159] See also Figure 8 In an exemplary embodiment, the application uses the classroom as the target scene, deploys high-definition cameras, omnidirectional microphone arrays and ultra-wideband positioning base stations in the classroom, and assigns positioning tags to teachers and students. The positioning tags can be set on wristbands or identity cards, such as campus cards. The teacher is the target object, thereby collecting the object positioning data, spatial audio data and spatial video data of the teacher in the classroom.

[0160] It should be noted that according to existing research, teachers' nonverbal behavior has a significant impact on students' learning effects and learning experience. Nonverbal communication (NVC) refers to all communication elements other than language. Paralanguage such as tone and volume are important sound factors in nonverbal communication; gestures, facial expressions, etc. are non-vocal elements in nonverbal communication. Up to 65% of the meaning of human interaction comes from nonverbal communication. In the field of education, there are a lot of studies on teachers' nonverbal behavior. For example, teachers' gestures help improve students' learning participation and effectiveness; teachers' spatial distance and movement methods affect students' learning engagement, learning motivation, and learning experience and effects. Therefore, it is very important to analyze interactive behaviors in teaching scenarios through facial feature data and character action data.

[0161] Then, shallow behavior analysis is performed based on the collected data: the classroom scene is modeled through spatial video data to obtain spatial map data, and the spatial movement data of the teacher in the classroom can be further obtained, so as to clarify the specific location information of students and teachers in the classroom and the relative location information between teachers and students, which can be achieved through a graph convolutional neural network model; and the audio data is analyzed using features such as Mel-frequency cepstral coefficients to extract information such as the teacher's speaking speed, intonation, and sound intensity, and speech recognition is performed to extract the conversation information between the teacher and the students; then, through character recognition technology, spatial video data related to the teacher is extracted as object video data based on object positioning data, and the object video data may also include students who interact with the teacher; using graph-based dynamic skeleton modeling, character action analysis is performed based on object video data to extract information on the posture and body movements of the teacher or student, and gesture information can also be extracted based on the neural network model; and through face recognition technology, facial feature extraction is performed based on object video data to obtain facial feature information of the teacher or student, and further combined with spectrum feature data, emotional feature information when the teacher interacts with the student can be analyzed.

[0162] Based on time series, that is, by integrating shallow behavior feature data according to time sequence and storing it in the knowledge graph database, we obtain graph node data, and determine the objects of interaction in the classroom based on gestures, voice, positioning and other information, obtain the interaction direction information between objects, and generate behavior graph sub-data based on the interaction direction information and graph node data. Then, based on the behavior graph sub-data of different time series, we generate interactive behavior graph data, which contains all aspects of feature information of the interaction between teachers and students in the classroom in a sequential period of time.

[0163] Finally, the interactive behavior graph data is input into a pre-trained AI model for in-depth analysis. The AI ​​model will first fine-tune the model based on the behavioral knowledge data and train in behavioral analysis. The behavioral knowledge data is usually educational theory, psychological research, and previous teaching experience, which provides the AI ​​model with basic knowledge about the interactive behavior between teachers and students.

[0164] After obtaining the interactive behavior analysis information derived from the artificial intelligence large model, the interactive behavior of the target object can be evaluated based on the interactive behavior analysis information, including the teacher's teaching quality, the student's learning effect, etc., and teaching suggestions and interactive evaluation information can be pushed to the teacher at intervals of 15 minutes, so as to provide feedback based on the teacher's teaching activities, help teachers discover deficiencies in their own teaching, and promote teachers' professional development.

[0165] The embodiment of the present application obtains the positioning data of the target object in the target space, obtains the object positioning data, obtains the spatial audio data, and obtains the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data, and spatial video data to obtain a shallow behavior feature data set. A knowledge graph is then constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis large model to obtain interactive behavior analysis information. Therefore, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data through shallow behavior feature data, associates shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a behavior analysis large model with behavior knowledge data, so as to identify and analyze complex interactive behaviors in fixed scenes.

[0166] See also Fig. 9 The embodiment of the present application also provides an interactive behavior analysis system, which can implement the above-mentioned interactive behavior analysis method, and the system includes:

[0167] A positioning data acquisition module, used to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space;

[0168] An audio data acquisition module, used to acquire spatial audio data; wherein the spatial audio data at least includes audio data of a target object;

[0169] A video data acquisition module, used to acquire spatial video data corresponding to the target space;

[0170] A shallow behavior analysis module is used to perform shallow behavior analysis based on the object positioning data, the spatial audio data and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one other object;

[0171] The knowledge graph construction module is used to construct the knowledge graph based on each shallow behavior feature data to obtain the interactive behavior graph data;

[0172] The large model association analysis module is used to perform association analysis on the interactive behavior graph data through a pre-trained behavior analysis large model to obtain interactive behavior analysis information.

[0173] The specific implementation of the interactive behavior analysis system is basically the same as the specific implementation of the above-mentioned interactive behavior analysis method, and will not be repeated here.

[0174] In some embodiments, the interactive behavior graph data will be stored in a knowledge graph database so that the associated interactive behavior graph data can be extracted at any time into the action analysis model. At the same time, a knowledge database is also provided to store theoretical knowledge related to interaction and some experience cases of interaction. For example, for teaching scenarios, theoretical knowledge related to classroom teaching and sub-communication, some practical experience and cases of teachers and education experts, and text descriptions and analyses of some classroom videos or clips can be stored. The content of the knowledge database can be used to train or fine-tune the action analysis model so that the action analysis model can better analyze the interactive behavior of the target object and other objects based on these theoretical knowledge.

[0175] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned interactive behavior analysis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0176] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0177] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0178] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the interactive behavior analysis method of the embodiment of this application;

[0179] Input / output interface 1003, used to implement information input and output;

[0180] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0181] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0182] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0183] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned interactive behavior analysis method is implemented.

[0184] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0185] The interactive behavior analysis method, system, electronic device and storage medium provided in the embodiments of the present application obtain the positioning data of the target object in the target space, obtain the object positioning data, obtain the spatial audio data, and obtain the spatial video data corresponding to the target space. Then, shallow behavior analysis is performed based on the object positioning data, spatial audio data and spatial video data to obtain a shallow behavior feature data set. Then, a knowledge graph is constructed based on each shallow behavior feature data to obtain interactive behavior graph data. The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis large model to obtain interactive behavior analysis information. Therefore, the present application performs shallow behavior analysis on multimodal data, then constructs interactive behavior graph data through shallow behavior feature data, associates shallow behavior analysis information, obtains the interaction process and interactive behavior characteristics between different objects in a fixed scene, and then performs association analysis on the interactive behavior graph data through a behavior analysis large model with behavior knowledge data, so as to identify and analyze complex interactive behaviors in fixed scenes.

[0186] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0187] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0188] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0189] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0190] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0191] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0193] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0196] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An interactive behavior analysis method, characterized in that: The method comprises: Acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space; Acquire spatial audio data; wherein the spatial audio data at least includes audio data of the target object; Acquire spatial video data corresponding to the target space; Performing shallow behavior analysis according to the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents the interactive behavior information between the target object and at least one of the other objects; Constructing a knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data; The interactive behavior graph data is subjected to association analysis through a pre-trained behavior analysis model to obtain interactive behavior analysis information.

2. The method according to claim 1, characterized in that The shallow behavior analysis is performed according to the object positioning data, the spatial audio data and the spatial video data to obtain a shallow behavior feature data set, specifically including: Performing scene modeling on the target space according to the spatial video data to obtain spatial map data; extracting the spatial video data related to the target object based on the object positioning data to obtain object video data; Extract facial features based on the object video data to obtain facial feature data; Extracting frequency spectrum features according to the spatial audio data to obtain frequency spectrum feature data; Respective shallow behavior feature data are generated based on the space map data, the object video data, the facial feature data and the spectrum feature data.

3. The method according to claim 2, characterized in that The shallow behavior feature data includes any one of the following: voice interaction recognition data, emotion interaction feature data, spatial movement data, and object interaction action data. The shallow behavior feature data generated based on the spatial map data, the object video data, the facial feature data, and the spectrum feature data include: Perform speech recognition based on the frequency spectrum feature data to obtain speech interaction recognition data; Based on the spatial map data and the object positioning data, obtaining the spatial movement data about the target object; Performing an emotion feature analysis based on the frequency spectrum feature data and the facial feature data to obtain the emotion interaction feature data; The target object is subjected to a character motion analysis based on the object video data to obtain the object interaction motion data.

4. The method according to claim 3, characterized in that The voice interaction recognition data, the emotion interaction feature data, the spatial movement data and the object interaction action data are all related to the time series, and the knowledge graph is constructed based on each of the shallow behavior feature data to obtain the interaction behavior graph data, including: Performing graph data integration on the voice interaction recognition data, the emotion interaction feature data, the spatial movement data, and the object interaction action data of the same time series to generate graph node data corresponding to the time series; Performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interactive action data, determining the other objects that interact with the target object, and obtaining interactive pointing information; Generate behavior graph sub-data corresponding to the time series based on the graph node data and the interaction direction information; A spatiotemporal graph is generated based on the behavior graph sub-data of different time series to obtain interactive behavior graph data.

5. The method according to claim 4, characterized in that The performing interactive object detection based on the voice interaction recognition data, the spatial movement data, and the object interactive action data, determining the other objects interacting with the target object, and obtaining the interactive pointing information includes: Acquire identity information of all objects in the target space, and obtain identity information of interaction-directed objects based on the identity information and the voice interaction recognition data; Perform visual direction recognition based on the object interaction action data to obtain interactive visual direction information; Performing object trajectory recognition based on the spatial movement data to obtain a target object movement trajectory; Based on the movement trajectory of the target object, the interactive visual direction information and the interactive pointing object identity information, the other objects that interact with the target object are determined to obtain the interactive pointing information.

6. The method according to any one of claims 1 to 5, characterized in that: Before performing association analysis on the interactive behavior graph data by using the pre-trained behavior analysis big model to obtain the interactive behavior analysis information, it also includes pre-training the behavior analysis big model, specifically including: Acquire behavior knowledge data; wherein the behavior knowledge data represents knowledge theory data related to the interaction behavior between at least two objects; Acquire a training data set; wherein the training data set includes behavior training data and behavior indicator information corresponding to the behavior training data; Inputting the behavior training data and the behavior knowledge data into a preset original behavior analysis large model to perform behavior analysis to obtain training recognition information; Perform error calculation based on the training identification information and the behavior indicator information to obtain comparison deviation data; The model parameters of the original behavior analysis large model are updated based on the comparison deviation data to obtain the pre-trained behavior analysis large model.

7. The method according to any one of claims 1 to 5, characterized in that: After the interactive behavior graph data is subjected to association analysis by the pre-trained behavior analysis model to obtain interactive behavior analysis information, the method includes: Based on the interaction behavior analysis information, the interaction behavior of the target object is evaluated to obtain interaction evaluation information; wherein the interaction evaluation information indicates the area in the target space where the target object has interacted; generating interaction suggestion information based on the interaction evaluation information; The interaction suggestion information is pushed to the target object.

8. An interactive behavior analysis system, characterized in that: The system comprises: A positioning data acquisition module, used to acquire positioning data of a target object in a target space to obtain object positioning data; wherein the target object is used to interact with at least one other object in the target space; An audio data acquisition module, used to acquire spatial audio data; wherein the spatial audio data at least includes the audio data of the target object; A video data acquisition module, used to acquire spatial video data corresponding to the target space; A shallow behavior analysis module, configured to perform shallow behavior analysis based on the object positioning data, the spatial audio data, and the spatial video data to obtain a shallow behavior feature data set; the shallow behavior feature data set includes at least two different shallow behavior feature data, and the shallow behavior feature data represents interaction behavior information between the target object and at least one of the other objects; A knowledge graph construction module, used to construct a knowledge graph based on each of the shallow behavior feature data to obtain interactive behavior graph data; The large model association analysis module is used to perform association analysis on the interactive behavior map data through a pre-trained behavior analysis large model to obtain interactive behavior analysis information.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the interactive behavior analysis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the interactive behavior analysis method according to any one of claims 1 to 7 is implemented.

Images