A real-time conversation analysis method and system based on 2D digital human

By generating a dynamic interaction mapping model and identifying local operation models, combined with the strength of active interaction intentions, the problem of difficulty in judging the active manipulation of sudden dynamic interactive objects in the existing technology is solved, and the judgment accuracy and user interaction experience of the 2D digital human system are improved. .

CN118799464BActive Publication Date: 2024-11-29杨风军
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411260933.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-29
Estimated Expiration
2044-09-10

AI Technical Summary

Technical Problem

The existing 2D digital human system has limitations in judging whether the sudden dynamic interactive object is actively controlled by the user, making it difficult to distinguish the user's true intentions and unintentional behavior, resulting in misjudgment and affecting the consistency and accuracy of dialogue and interaction.

Method used

By obtaining the motion trajectory of the burst dynamic interactive object and the user's three-dimensional model data, a dynamic interaction mapping model is generated, a local operation model of the target user's moving part is identified, matching degree and active interaction intention intensity are calculated, and preliminary correlation scores are adjusted to determine whether the object is Actively interact for users.

Benefits of technology

It improves the accuracy and reliability of judging whether the sudden dynamic interactive object is actively controlled by the user, and enhances the interactive experience between 2D digital people and users and the intelligence of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799464B_ABST
    Figure CN118799464B_ABST
Patent Text Reader

Abstract

The present invention is applicable to the technical field of 2D digital human real-time dialogue analysis, and provides a real-time dialogue analysis method and system based on 2D digital human. When a sudden dynamic interactive object is detected in the user scene video, the preliminary association score of the sudden dynamic interactive object and the reference value for judging whether the sudden dynamic interactive object belongs to the user's active interaction are obtained. After the 2D digital human system recognizes the sudden dynamic interactive object, it can optimize the judgment of whether the object belongs to the user's active interaction through two key dimensions. First, based on the user scene video, the system identifies the target user's moving parts (such as hands, feet, etc.) that may actively manipulate the sudden object, and finds the local operation models corresponding to these parts in the dynamic interaction mapping model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of 2D digital human real-time dialogue analysis, and in particular relates to a real-time dialogue analysis method and system based on 2D digital human. Background Art

[0002] With the development of artificial intelligence technology, the application of 2D digital humans in real-time dialogue analysis is becoming more and more widespread, especially in the fields of virtual assistants, education and entertainment. 2D digital humans provide users with personalized interactive experiences through real-time analysis of user behavior. In these application scenarios, the system usually relies on user scene videos to capture the user's dynamic behavior, and analyzes and responds based on the user's interaction with objects in the environment. Existing 2D digital human systems often determine whether an object is an active interactive object of the user based on monitoring the spatial position relationship and motion trajectory between the user and the object. When a sudden dynamic interactive object appears in the scene, the system will evaluate whether the object is the result of active manipulation by the user, so as to decide whether it needs to be included in the interactive content with the user. Usually, this evaluation is mainly based on the physical properties of the object entering the user's field of view, such as the path, speed, and distance.

[0003] However, existing technologies have great limitations in judging whether sudden dynamic interactive objects are actively manipulated by users. Current systems usually rely only on simple spatial position relationships between objects and users (such as the distance of objects from users, the angle and path of objects entering the field of view) or basic movement patterns (such as the speed and direction of objects) to make preliminary judgments. This position and trajectory-based analysis method lacks detailed consideration of users' specific behaviors and intentions, and is therefore easily interfered by a variety of external factors. For example, when an object is blown into the user scene by the wind or accidentally rolls into the user scene due to a collision, the existing system may misjudge it as the result of active manipulation by the user based solely on the path of the object approaching the user. For the complex multiple actions of users (such as natural hand swings and unintentional touches) and the influence of other dynamic factors in the environment, existing methods find it difficult to effectively distinguish between users' true intentions (such as intention to grab or wave to refuse) and unintentional behaviors, which leads to system misjudgment and affects the coherence and accuracy of the dialogue and interaction between 2D digital humans and users. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time conversation analysis method and system based on 2D digital human, aiming to solve the problems raised in the background technology.

[0005] The present invention is implemented as follows: a real-time conversation analysis method based on 2D digital human, the method comprising:

[0006] When a sudden dynamic interactive object is detected in the user scene video, a preliminary correlation score of the sudden dynamic interactive object and a reference value for determining whether the sudden dynamic interactive object belongs to active user interaction are obtained;

[0007] Obtain user scene video, extract the motion trajectory of sudden dynamic interactive objects and the user's three-dimensional model data from it, and render the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model;

[0008] Based on the user scene video, determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to achieve the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model;

[0009] Obtaining a user operation video of the target user's motion part corresponding to the target local operation model within the retrospective time period, and analyzing the intensity of the active interaction intention of the target local operation model to the motion trajectory according to the user operation video;

[0010] The correction ratio is determined according to the matching degree of the target local operation model and the strength of the active interaction intention, and the preliminary association score is adjusted according to the correction ratio.

[0011] As a further limitation of the technical solution of the embodiment of the present invention, based on the user scene video, determining the target user movement parts that can actively manipulate the sudden dynamic interactive object, and finding the local operation models corresponding to these target user movement parts in the dynamic interaction mapping model, calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model. The steps include:

[0012] Based on the user scene video, identify and determine the target user's motion parts that can actively manipulate the sudden dynamic interactive objects, and extract their coordinate information in the 3D model space;

[0013] According to the coordinate information of the moving parts of each target user in the three-dimensional model space, the local operation model corresponding to the moving parts of each target user in the dynamic interactive mapping model is determined;

[0014] The matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object is calculated, and the local operation model with the highest matching degree is selected and set as the target local operation model.

[0015] As a further limitation of the technical solution of the embodiment of the present invention, the steps of calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model include:

[0016] Predict the executable trajectory of each local operation model and identify the similarity between the executable trajectory and the motion trajectory of the sudden dynamic interaction object;

[0017] The similarity between the executable motion trajectory of the local operation model and the motion trajectory of the sudden dynamic interaction object is defined as the matching degree of the motion trajectory of the sudden dynamic interaction object realized by the local operation model;

[0018] The matching degree of all local operation models to realize the motion trajectory of the sudden dynamic interactive object is screened, and the local operation model with the highest matching degree is set as the target local operation model.

[0019] As a further limitation of the technical solution of the embodiment of the present invention, the target user's movement part that can actively control the sudden dynamic interactive object specifically refers to the part of the user's body that can directly control or influence the sudden dynamic interactive object, including hands, mouth, arms, legs or feet.

[0020] As a further limitation of the technical solution of the embodiment of the present invention, the steps of obtaining a user operation video of a target user's motion part corresponding to the target local operation model within a retrospective time period, and analyzing the intensity of the active interaction intention of the target local operation model to the motion trajectory according to the user operation video include:

[0021] Obtain user scenario videos and capture user operation videos within the retrospective time period;

[0022] Analyze the user operation video and identify the key action features of the target user's moving parts corresponding to the target local operation model;

[0023] The trend correlation value between the key action features and the motion trajectory of the target user's moving parts is evaluated, and the trend correlation value is defined as the active interaction intention intensity of the target local operation model for the motion trajectory.

[0024] As a further limitation of the technical solution of the embodiment of the present invention, the step of determining the correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjusting the preliminary association score by the correction ratio includes:

[0025] The matching degree of the target local operation model and the strength of the active interaction intention are weighted, and the correction ratio is calculated through linear combination;

[0026] The optimized correlation score is calculated by multiplying the preliminary correlation score by the correction ratio;

[0027] The optimized association score is compared with the benchmark value to determine whether the sudden dynamic interactive object belongs to the user's active interaction.

[0028] A real-time dialogue analysis system based on 2D digital human, the system comprising: a sudden dynamic interaction object recognition module, a dynamic interaction mapping model generation module, a target local operation model determination module, an active interaction intention intensity generation module and a preliminary association score adjustment module, wherein:

[0029] A sudden dynamic interaction object recognition module is used to obtain a preliminary relevance score of the sudden dynamic interaction object and a reference value for determining whether the sudden dynamic interaction object belongs to the user's active interaction after detecting that the sudden dynamic interaction object appears in the user scene video;

[0030] A dynamic interaction mapping model generation module is used to obtain user scene videos, extract the motion trajectory of sudden dynamic interaction objects and the three-dimensional model data of the user, and render the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model;

[0031] The target local operation model determination module is used to determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object based on the user scene video, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model;

[0032] An active interaction intention strength generation module is used to obtain a user operation video of a target user's motion part corresponding to a target local operation model within a retrospective time period, and analyze the active interaction intention strength of the target local operation model for the motion trajectory based on the user operation video;

[0033] The preliminary association score adjustment module is used to determine the correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjust the preliminary association score according to the correction ratio.

[0034] As a further limitation of the technical solution of the embodiment of the present invention, the target local operation model determination module specifically includes:

[0035] A target user motion part recognition unit is used to recognize and determine the target user motion part that can actively manipulate the sudden dynamic interactive object based on the user scene video, and extract its coordinate information in the three-dimensional model space;

[0036] A local operation model determination unit, used to determine the local operation model corresponding to each target user's motion part in the dynamic interaction mapping model according to the coordinate information of each target user's motion part in the three-dimensional model space;

[0037] The target local operation model determination unit is used to calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model.

[0038] As a further limitation of the technical solution of the embodiment of the present invention, the active interaction intention intensity generation module specifically includes:

[0039] A user operation video capture unit is used to obtain user scene videos and capture user operation videos within a retrospective time period;

[0040] A key action feature recognition unit is used to analyze the user operation video and recognize the key action features of the target user's motion part corresponding to the target local operation model;

[0041] The active interaction intention strength generating unit is used to evaluate the trend correlation value between the key action features and the motion trajectory of the target user's motion part, and define the trend correlation value as the active interaction intention strength of the target local operation model for the motion trajectory.

[0042] As a further limitation of the technical solution of the embodiment of the present invention, the preliminary correlation score adjustment module specifically includes:

[0043] A correction ratio calculation unit is used to perform weighted processing on the matching degree of the target local operation model and the strength of the active interaction intention, and obtain the correction ratio through linear combination calculation;

[0044] A preliminary correlation score adjustment unit, used to multiply the preliminary correlation score by the correction ratio to calculate an optimized correlation score;

[0045] The user active interaction judgment unit is used to compare the optimized association score with the reference value to judge whether the sudden dynamic interaction object belongs to the user active interaction.

[0046] Compared with the prior art, the present invention obtains a preliminary association score of the sudden dynamic interaction object and a reference value for judging whether the sudden dynamic interaction object belongs to active user interaction after detecting that a sudden dynamic interaction object appears in a user scene video; obtains a user scene video, extracts a motion trajectory of the sudden dynamic interaction object and three-dimensional model data of the user therefrom, and renders the extracted motion trajectory and user model into a three-dimensional model space to generate a dynamic interaction mapping model; determines the target user motion parts that can actively manipulate the sudden dynamic interaction object based on the user scene video, finds local operation models corresponding to these target user motion parts in the dynamic interaction mapping model, calculates the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interaction object, selects the local operation model with the highest matching degree, and sets it as the target local operation model; obtains a user operation video of the target user motion part corresponding to the target local operation model within a retrospective time period, and analyzes the active interaction intention strength of the target local operation model to the motion trajectory according to the user operation video; determines a correction ratio according to the matching degree and active interaction intention strength of the target local operation model, and adjusts the preliminary association score according to the correction ratio.

[0047] After the 2D digital human system recognizes a sudden dynamic interactive object, it can optimize the judgment of whether the object belongs to the user's active interaction through two key dimensions. First, based on the user scene video, the system identifies the target user's moving parts (such as hands, feet, etc.) that may actively manipulate the sudden object, and finds the local operation models corresponding to these parts in the dynamic interaction mapping model. Then, the system calculates the matching degree of each local operation model to achieve the object's motion trajectory, and selects the model with the highest matching degree as the target operation model to evaluate the correlation between the moving part and the object. Secondly, the system obtains the user operation video within the retrospective time period, analyzes the behavioral characteristics of the target local operation model and the user's interaction trajectory with the object, and quantifies the intensity of the active interaction intention. By combining the two dimensions of matching degree and active interaction intention intensity, the system can dynamically correct the initial benchmark value used to determine whether the object belongs to the user's active interaction.

[0048] The reason for choosing these two dimensions is that they can fully reflect the interaction between users and sudden objects. The matching degree evaluates the physical position and movement relationship between the user's body parts and the object, ensuring the relevance of the identified target operation model and the object trajectory. The intensity of active interaction intention captures the user's behavioral motivation and intention, and reveals the user's attention or operation intention to the object by analyzing the trend of the user's action and the object's interaction. This method avoids the limitation of relying on a single factor for judgment. By integrating spatial position, action characteristics and behavioral intention, it enhances the accuracy and reliability of the system's judgment, thereby establishing a smarter and more natural interaction foundation for 2D digital human dialogue. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method provided by an embodiment of the present invention;

[0050] Figure 2 A flow chart of determining a local operation model corresponding to a target user's motion part in a dynamic interaction mapping model in a method provided by an embodiment of the present invention;

[0051] Figure 3 A flow chart of determining a target local operation model in the method provided in an embodiment of the present invention;

[0052] Figure 4 A flow chart of determining the intensity of active interaction intention of a target local operation model to a motion trajectory in the method provided by an embodiment of the present invention;

[0053] Figure 5 A flow chart of adjusting a preliminary correlation score by correcting a ratio in a method provided by an embodiment of the present invention;

[0054] Figure 6 An application architecture diagram of a system provided by an embodiment of the present invention;

[0055] Figure 7 A structural block diagram of a target local operation model determination module in a system provided by an embodiment of the present invention;

[0056] Figure 8 A structural block diagram of an active interaction intention intensity generation module in a system provided by an embodiment of the present invention;

[0057] Fig. 9 This is a structural block diagram of a preliminary correlation score adjustment module in a system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Figure 1 A flow chart of a method provided by an embodiment of the present invention is shown.

[0060] Specifically, a real-time conversation analysis method based on 2D digital human comprises the following steps:

[0061] Step S100, when a sudden dynamic interactive object is detected in the user scene video, a preliminary association score of the sudden dynamic interactive object and a reference value for determining whether the sudden dynamic interactive object belongs to active user interaction are obtained.

[0062] In the embodiment of the present invention, in the 2D digital human application, the user scene video is captured in real time by a camera, which can be a fixed RGB camera or a depth camera, for monitoring the dynamic changes of the user and his surrounding environment. The video content includes information such as the user's facial expressions, gestures, body movements, and objects and background dynamics in the environment. The video data is used to analyze user behavior, environmental changes, and object dynamics in real time, providing basic data support for the interactive response of the 2D digital human.

[0063] Sudden dynamic interactive objects refer to objects that suddenly appear in the user scene video. These objects may be actively introduced by the user, or passively enter the user scene due to external factors. For example, when a user brings a ball into the camera range with his hand, the ball is a sudden dynamic interactive object actively introduced by the user; if the ball flies in from outside the scene, it is a sudden dynamic interactive object that enters passively. Determining whether the object is actively introduced by the user or passively enters can help the system better understand the user's intentions and interaction needs.

[0064] The preliminary association score is a value used to evaluate whether a sudden dynamic interactive object may be actively introduced by the user. It is calculated by analyzing data such as user behavior (such as hand movements, body postures) and object motion trajectories. The baseline value is a threshold preset by the system to determine whether the object is an active interactive object of the user. When the preliminary association score is higher than the baseline value, the system determines that the object is actively introduced by the user; if it is lower than the baseline value, it is considered that the object passively enters the scene.

[0065] The purpose of this judgment is to distinguish between objects actively introduced by the user and objects that enter accidentally or passively, helping the 2D digital human to understand the user's intention more accurately and adjust the dialogue and interaction methods. By judging the source of the object, the 2D digital human can choose a response strategy more intelligently. For example, if the object is actively introduced by the user, the digital human can provide relevant interaction prompts or suggestions; if it is passively introduced, it can choose to ignore or ask the user's intention, thereby improving the user experience and interaction accuracy.

[0066] Existing technologies calculate preliminary association scores through computer vision and artificial intelligence algorithms (such as convolutional neural networks (CNNs), motion trajectory analysis, and posture estimation networks). The system can analyze the user's action patterns, the object's motion trajectory, and environmental changes to infer whether the sudden object enters actively or passively. The benchmark value is adjusted through analysis and machine learning of a large amount of data on user behavior patterns. These technologies have matured and are widely used in the fields of object recognition and interaction judgment in intelligent video surveillance, augmented reality (AR), and virtual reality (VR), providing support for scene understanding and real-time dialogue of 2D digital humans.

[0067] Furthermore, the real-time conversation analysis method based on 2D digital human also includes the following steps:

[0068] Step S200, obtaining a user scene video, extracting the motion trajectory of the sudden dynamic interactive object and the three-dimensional model data of the user, and rendering the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model.

[0069] In an embodiment of the present invention, the motion trajectory of a sudden dynamic interactive object is extracted based on the user scene video, and the object in the video frame is first detected using computer vision technology (such as target detection and tracking algorithms). Common target detection algorithms include YOLO (You Only Look Once) and Faster R-CNN, which can identify objects in the scene in real time. After the sudden dynamic interactive object is detected, a target tracking algorithm (such as Kalman filter, optical flow method or DeepSORT) is used to track the position changes of the object in each frame and extract its motion trajectory. By tracking the continuous motion path of the object, its motion trajectory in the scene can be constructed, which is usually expressed as a sequence of positions in a series of two-dimensional images.

[0070] The user's 3D model data extraction is achieved through deep learning algorithms (such as posture estimation networks or depth estimation networks). Technologies such as OpenPose or MediaPipe can be used to identify user key points (such as joints, hands, facial features, etc.) in video frames and use these key points to infer the user's 3D posture and shape. If an RGB camera is used, the depth information can be calculated in combination with methods such as binocular vision or structured light technology to generate the user's 3D point cloud data and further build the user's 3D model.

[0071] Once the motion trajectory of the sudden dynamic interactive object and the user's 3D model data are obtained, these data will be imported into the 3D coordinate system to establish a unified 3D model space. The rendering engine (such as Unity or Unreal Engine) is used to visualize the user's 3D model and motion trajectory. The motion trajectory is represented by the connecting line or motion vector of the trajectory points, and the user model is rendered using a 3D mesh or skeleton model inferred from the 2D image. During the rendering process, the pixel coordinates in the 2D image are converted to world coordinates in the 3D space to ensure that the user's posture and the object's motion trajectory are accurately superimposed in the 3D model space, thereby generating a dynamic interaction mapping model that truly reflects the interaction status between the user and the sudden object.

[0072] For example, in a 2D digital human dialogue scene, when a user brings a ball into view with his hand, the ball in the video can be detected by the YOLO algorithm and tracked using a Kalman filter to extract the ball's trajectory. At the same time, the OpenPose algorithm identifies the positions of key points such as the user's hands, shoulders, and elbows, builds a 3D skeleton model of the user, and infers the user's hand movements in 3D space. Next, the ball's trajectory and the user's 3D skeleton model are mapped to the 3D model space in the rendering engine to generate a dynamic interactive mapping model. This model is used to determine whether the user actively brings the ball into the scene or passively receives objects in the scene, thereby helping 2D digital humans understand user behavior and intentions more accurately.

[0073] Furthermore, the real-time conversation analysis method based on 2D digital human also includes the following steps:

[0074] Step S300, based on the user scene video, determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model.

[0075] Specifically, Figure 2 A flow chart of determining the local operation model corresponding to the target user's motion part in the dynamic interaction mapping model is shown.

[0076] Among them, based on the user scene video, the target user's motion parts that can actively manipulate the sudden dynamic interactive object are determined, and the local operation models corresponding to these target user's motion parts are found in the dynamic interaction mapping model, and the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object is calculated, and the local operation model with the highest matching degree is selected and set as the target local operation model. Specifically, the following steps are included:

[0077] Step S301, based on the user scene video, identifying and determining the target user's moving parts that can actively manipulate the sudden dynamic interactive object, and extracting its coordinate information in the three-dimensional model space;

[0078] Step S302, determining the local operation model corresponding to each target user's moving part in the dynamic interactive mapping model according to the coordinate information of each target user's moving part in the three-dimensional model space;

[0079] Step S303, calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model.

[0080] The target user movement part that can actively manipulate the sudden dynamic interactive object specifically refers to the part of the user's body that can directly manipulate or influence the sudden dynamic interactive object, including hands, mouth, arms, legs or feet.

[0081] In an embodiment of the present invention, the identification and determination of the target user's moving parts that can actively manipulate the sudden dynamic interactive objects needs to be achieved through image processing and computer vision technology in the user scene video. First, a deep learning algorithm (such as a posture estimation network, a key point detection algorithm) is used to identify the user's body parts, which include hands, mouth, arms, legs or feet, etc., which can directly manipulate or affect the sudden dynamic interactive objects. The algorithm obtains the 2D position data of each moving part of the user by detecting the user's skeletal key points (such as the OpenPose or MediaPipe algorithm), and generates the user's three-dimensional model data in combination with binocular vision, depth cameras or other technologies. These data are then converted into coordinate information in the three-dimensional model space to accurately describe the spatial position of the user's moving parts.

[0082] According to the coordinate information of each target user's moving parts in the three-dimensional model space, the system needs to further determine the local operation models corresponding to these moving parts in the dynamic interaction mapping model. Through coordinate matching and spatial mapping technology, the system compares the coordinate information of the user's moving parts in the three-dimensional model space with the various local operation models in the dynamic interaction mapping model. For example, if the coordinates of the user's hand in the three-dimensional space are shown to be close to an object, the system will identify the local operation model corresponding to the hand coordinates through mapping technology, and determine that the operation model can realize the manipulation of the object. In this way, the system can accurately identify the user's action intentions and interactive behaviors.

[0083] In the three-dimensional model space, the system matches all target user moving parts (for example, the user's hands, legs, and head) one by one based on these coordinate information. Through the coordinate matching algorithm, the system compares the three-dimensional coordinates of each moving part of the user with the local operation model in the dynamic interaction mapping model to find the local operation model that best matches each moving part. For example, the system may match the hand coordinates to the "hand grasping model", the leg coordinates to the "leg movement model", and the head coordinates to the "head tracking model".

[0084] This one-to-one matching process ensures that all target user motion parts in the user scene video can correspond to the appropriate local operation model. In this way, the system can fully capture and understand all the user's interaction intentions and behaviors in the scene, and effectively optimize the response and dialogue content of the 2D digital human.

[0085] Specifically, Figure 3A flow chart for determining a target local operating model is shown.

[0086] The steps of calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model specifically include the following steps:

[0087] Step S3031, predicting the executable running trajectory of each local operation model, and identifying the similarity between the executable running trajectory and the motion trajectory of the sudden dynamic interaction object;

[0088] Step S3032, defining the similarity between the executable motion trajectory of the local operation model and the motion trajectory of the sudden dynamic interaction object as the matching degree of the local operation model to realize the motion trajectory of the sudden dynamic interaction object;

[0089] Step S3033, screening the matching degree of all local operation models to realize the motion trajectory of the sudden dynamic interactive object, and setting the local operation model with the highest matching degree as the target local operation model.

[0090] In the embodiments of the present invention, an executable running trajectory refers to a reasonable motion path that a user's moving part (such as a hand, leg, etc.) can execute under normal circumstances. It describes the natural movements that a specific body part may perform based on human kinematics and physiological limitations. For example, the executable running trajectory of the hand may include movements such as grabbing, waving, pushing and pulling, while the executable running trajectory of the leg may include walking, kicking, etc. These trajectories start from a certain initial position and reach a possible terminal position, and take into account factors such as the speed, acceleration, and direction of the movement to reflect the natural range of motion and movement ability of the part.

[0091] First, the system extracts the motion trajectory of the sudden dynamic interactive object from the user scene video and determines the end position of the trajectory (such as the final position of a ball that suddenly flies into the scene). Next, predict the possible executable running trajectory for each user's moving part. For example, the executable trajectory of the hand starts from the initial position until it reaches the end point of the sudden object. When calculating these executable running trajectories, the system considers factors such as the starting position, speed, acceleration, and direction of the moving part. Then, the executable running trajectory of each local operation model is compared with the actual motion trajectory of the sudden dynamic interactive object through a trajectory similarity algorithm (such as the dynamic time warping algorithm DTW, Euclidean distance, or cosine similarity) to determine the degree of similarity between the two in path shape, length, and time characteristics. Finally, these similarities are defined as the matching degree of the local operation model to achieve the motion trajectory of the sudden dynamic interactive object.

[0092] By calculating and selecting the local operation model with the highest degree of match, the system can more accurately identify the user's actual operation intention and distinguish between the user's active operation and unintentional actions. This approach enables the system to dynamically adapt to various emergencies, whether the object suddenly enters the scene or is actively introduced by the user, thereby enhancing the flexibility of the system. In addition, after selecting the local operation model with the highest degree of match, the system can provide more accurate feedback and response, optimizing the user experience. By avoiding blind processing of all possible interaction situations, the system also makes more efficient use of computing resources and improves overall performance.

[0093] Furthermore, the real-time conversation analysis method based on 2D digital human also includes the following steps:

[0094] Step S400, obtaining a user operation video of a target user's motion part corresponding to a target local operation model within a retrospective time period, and analyzing the intensity of the active interaction intention of the target local operation model to the motion trajectory according to the user operation video.

[0095] Specifically, Figure 4 A flow chart for determining the intensity of active interaction intention of a target local operation model for a motion trajectory is shown.

[0096] The process of obtaining a user operation video of a target user's motion part corresponding to a target local operation model within a retrospective time period and analyzing the intensity of the active interaction intention of the target local operation model to the motion trajectory according to the user operation video specifically includes the following steps:

[0097] Step S401, obtaining a user scene video, and intercepting a user operation video within a retrospective time period;

[0098] Step S402, analyzing the user operation video to identify key action features of the target user's moving parts corresponding to the target local operation model;

[0099] Step S403 , evaluating the trend correlation value between the key action feature and the motion trajectory of the target user's motion part, and defining the trend correlation value as the intensity of the active interaction intention of the target local operation model to the motion trajectory.

[0100] In an embodiment of the present invention, the backtracking time period refers to a specific time range for backtracking analysis in the user scene video, which is used to capture the user's action characteristics on the sudden dynamic interactive object. The basis for setting the backtracking time period is mainly based on the user's behavioral reaction time and the characteristics of the interactive event. In general, the backtracking time period should cover the entire time window from the appearance of the sudden object to the user's possible interaction, to ensure that all potential actions of the user on the object can be captured. For example, the time period can be determined based on experimental data analysis, and is usually set to range from a few seconds to tens of seconds, depending on the complexity of the scene and the user's expected reaction speed. The reason for this setting is to accurately identify the user's actual reaction process to the sudden object and ensure that the system can judge the user's interaction intention.

[0101] Key action features refer to the important behavior patterns or motion features exhibited by specific parts of the user's body (such as hands and feet) during the interaction process. These features usually include parameters such as the direction, amplitude, speed, acceleration, and specific action types (such as grabbing, pushing, pulling, waving, etc.). The method of identifying key action features relies on computer vision technology, such as convolutional neural networks (CNN) and posture estimation algorithms (such as OpenPose and MediaPipe) in deep learning. These algorithms analyze the continuous frame data in the user's operation video, identify and track the movement trajectory of various parts of the user's body, and extract key points and feature parameters that can represent the user's action intention. For example, when the user's hand reaches forward quickly, the system can identify it as a "grabbing" action feature.

[0102] The trend association value is used to quantify the relationship between the key action features of the target user's moving part and its motion trajectory. When calculating the trend association value, it is first necessary to obtain the user's key action features (such as the grasping action of the hand) and the actual motion trajectory of the part. Then, the system uses a similarity measurement method (such as cosine similarity or Pearson correlation coefficient) to compare the degree of match between the direction, speed, amplitude, etc. of the key action features and the motion trajectory. Specifically, by comparing the angle or similarity distribution between the user's action feature vector and the trajectory vector, a numerical value (usually between 0 and 1) is obtained as the trend association value. The higher this value is, the higher the degree of fit between the user's key action features and the actual motion trajectory, indicating that the target local operation model has a greater active interaction intention intensity for the motion trajectory.

[0103] Furthermore, the real-time conversation analysis method based on 2D digital human also includes the following steps:

[0104] Step S500, determining a correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjusting the preliminary association score according to the correction ratio.

[0105] Specifically, Figure 5 A flow chart showing the adjustment of preliminary association scores by correction ratio.

[0106] The correction ratio is determined according to the matching degree of the target local operation model and the strength of the active interaction intention, and the preliminary association score is adjusted by the correction ratio, which specifically includes the following steps:

[0107] Step S501, weighting the matching degree and active interaction intention strength of the target local operation model, and obtaining a correction ratio through linear combination calculation;

[0108] Step S502, multiplying the preliminary correlation score by the correction ratio to calculate an optimized correlation score;

[0109] Step S503 : comparing the optimized association score with the reference value to determine whether the sudden dynamic interaction object belongs to the user's active interaction.

[0110] In an embodiment of the present invention, it is assumed that the system detects a sudden dynamic interactive object (such as a ball that suddenly enters the field of view) and identifies the user's hand as the target local operation model. The system first calculates the matching degree of the hand operation model (assuming it is 0.75), indicating that the hand motion trajectory is highly similar to the ball motion trajectory. At the same time, the system also calculates the active interaction intention strength of the hand (assuming it is 0.6), reflecting the possibility of the user's hand movement interacting with the object. To obtain the correction ratio, the linear combination formula can be used: Correction ratio = W1 × matching degree + W2 × active interaction intention strength. Assuming that the weight W1 is 0.6 and W2 is 0.4, the correction ratio calculation formula is as follows: Correction ratio = 0.6 × 0.75 + 0.4 × 0.6 = 0.69, and the correction ratio obtained is 0.69.

[0111] The optimized relevance score is calculated by multiplying the initial relevance score by the correction ratio. This process adjusts the original score, takes into account the matching degree of the local operation model and the strength of the user's active interaction intention, and improves the accuracy of the judgment. For example, if the initial relevance score is 0.7, the optimized relevance score is the product of 0.69 (correction ratio) and 0.7 (initial relevance score), which is 0.483.

[0112] The optimized association score (0.483) is compared with the preset benchmark value (assuming it is 0.5). If the optimized score is lower than the benchmark value, the system determines that the sudden object is not an active interaction object of the user; if it is higher than the benchmark value, the object is considered to be an object actively introduced or interacted with by the user. The benefit of this process is that by comprehensively considering the user's action intention and the matching of the operation model, the system can more accurately understand the user's true intention, reduce misjudgment, and improve the quality of human-computer interaction and user experience.

[0113] In summary, by introducing the matching degree of the local operation model and the strength of the user's active interaction intention, accurate optimization of the judgment value is achieved. In the prior art, it is usually only based on the spatial position relationship between the object and the user or a simple action pattern matching to determine whether the object has a subjective correlation with the user. This method is easily affected by environmental noise and diverse user behaviors, resulting in low accuracy of the judgment result. The present invention starts from two key aspects: first, based on the matching degree of the local operation model, the possibility of realizing the object's motion trajectory by calculating the user's body parts (such as hands, feet, etc.) is accurately evaluated to evaluate the user's operation intention; secondly, combined with the user's active interaction intention strength, the user's gaze, gestures and other behavioral characteristics of the object are analyzed to quantify the user's degree of initiative.

[0114] By weighting the matching degree and the strength of active interaction intention, the correction ratio is comprehensively calculated to effectively adjust the association score of the preliminary judgment. This method can accurately capture the actual interaction relationship between the user and the sudden object, significantly improving the reliability and effectiveness of the judgment. Compared with the existing technology, the present invention has the following advantages and scientificity: it can dynamically adapt to diverse user behavior patterns and complex interaction scenarios, reducing the misjudgment rate; at the same time, combined with multi-dimensional analysis, it ensures the system's deep understanding and intelligent response to the user's true intentions, and improves the interactive experience and satisfaction between the 2D digital human and the user.

[0115] Furthermore, Figure 6 The application architecture diagram of the system provided by the embodiment of the present invention is shown.

[0116] Among them, in another preferred embodiment provided by the present invention, a real-time conversation analysis system based on 2D digital human comprises:

[0117] The sudden dynamic interaction object recognition module 100 is used to obtain a preliminary relevance score of the sudden dynamic interaction object and a reference value for determining whether the sudden dynamic interaction object belongs to active user interaction after detecting that the sudden dynamic interaction object appears in the user scene video.

[0118] In the embodiment of the present invention, in the 2D digital human application, the user scene video is captured in real time by a camera, which can be a fixed RGB camera or a depth camera, for monitoring the dynamic changes of the user and his surrounding environment. The video content includes information such as the user's facial expressions, gestures, body movements, and objects and background dynamics in the environment. The video data is used to analyze user behavior, environmental changes, and object dynamics in real time, providing basic data support for the interactive response of the 2D digital human.

[0119] Sudden dynamic interactive objects refer to objects that suddenly appear in the user scene video. These objects may be actively introduced by the user, or passively enter the user scene due to external factors. For example, when a user brings a ball into the camera range with his hand, the ball is a sudden dynamic interactive object actively introduced by the user; if the ball flies in from outside the scene, it is a sudden dynamic interactive object that enters passively. Determining whether the object is actively introduced by the user or passively enters can help the system better understand the user's intentions and interaction needs.

[0120] The preliminary association score is a value used to evaluate whether a sudden dynamic interactive object may be actively introduced by the user. It is calculated by analyzing data such as user behavior (such as hand movements, body postures) and object motion trajectories. The baseline value is a threshold preset by the system to determine whether the object is an active interactive object of the user. When the preliminary association score is higher than the baseline value, the system determines that the object is actively introduced by the user; if it is lower than the baseline value, it is considered that the object passively enters the scene.

[0121] The purpose of this judgment is to distinguish between objects actively introduced by the user and objects that enter accidentally or passively, helping the 2D digital human to understand the user's intention more accurately and adjust the dialogue and interaction methods. By judging the source of the object, the 2D digital human can choose a response strategy more intelligently. For example, if the object is actively introduced by the user, the digital human can provide relevant interaction prompts or suggestions; if it is passively introduced, it can choose to ignore or ask the user's intention, thereby improving the user experience and interaction accuracy.

[0122] Existing technologies calculate preliminary association scores through computer vision and artificial intelligence algorithms (such as convolutional neural networks (CNNs), motion trajectory analysis, and posture estimation networks). The system can analyze the user's action patterns, the object's motion trajectory, and environmental changes to infer whether the sudden object enters actively or passively. The benchmark value is adjusted through analysis and machine learning of a large amount of data on user behavior patterns. These technologies have matured and are widely used in the fields of object recognition and interaction judgment in intelligent video surveillance, augmented reality (AR), and virtual reality (VR), providing support for scene understanding and real-time dialogue of 2D digital humans.

[0123] Furthermore, the real-time conversation analysis system based on 2D digital human also includes:

[0124] The dynamic interaction mapping model generation module 200 is used to obtain the user scene video, extract the motion trajectory of the sudden dynamic interaction object and the three-dimensional model data of the user, and render the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model.

[0125] In an embodiment of the present invention, the motion trajectory of a sudden dynamic interactive object is extracted based on the user scene video, and the object in the video frame is first detected using computer vision technology (such as target detection and tracking algorithms). Common target detection algorithms include YOLO (You Only Look Once) and Faster R-CNN, which can identify objects in the scene in real time. After the sudden dynamic interactive object is detected, a target tracking algorithm (such as Kalman filter, optical flow method or DeepSORT) is used to track the position changes of the object in each frame and extract its motion trajectory. By tracking the continuous motion path of the object, its motion trajectory in the scene can be constructed, which is usually expressed as a sequence of positions in a series of two-dimensional images.

[0126] The user's 3D model data extraction is achieved through deep learning algorithms (such as posture estimation networks or depth estimation networks). Technologies such as OpenPose or MediaPipe can be used to identify user key points (such as joints, hands, facial features, etc.) in video frames and use these key points to infer the user's 3D posture and shape. If an RGB camera is used, the depth information can be calculated in combination with methods such as binocular vision or structured light technology to generate the user's 3D point cloud data and further build the user's 3D model.

[0127] Once the motion trajectory of the sudden dynamic interactive object and the user's 3D model data are obtained, these data will be imported into the 3D coordinate system to establish a unified 3D model space. The rendering engine (such as Unity or Unreal Engine) is used to visualize the user's 3D model and motion trajectory. The motion trajectory is represented by the connecting line or motion vector of the trajectory points, and the user model is rendered using a 3D mesh or skeleton model inferred from the 2D image. During the rendering process, the pixel coordinates in the 2D image are converted to world coordinates in the 3D space to ensure that the user's posture and the object's motion trajectory are accurately superimposed in the 3D model space, thereby generating a dynamic interaction mapping model that truly reflects the interaction status between the user and the sudden object.

[0128] For example, in a 2D digital human dialogue scene, when a user brings a ball into view with his hand, the ball in the video can be detected by the YOLO algorithm and tracked using a Kalman filter to extract the ball's trajectory. At the same time, the OpenPose algorithm identifies the positions of key points such as the user's hands, shoulders, and elbows, builds a 3D skeleton model of the user, and infers the user's hand movements in 3D space. Next, the ball's trajectory and the user's 3D skeleton model are mapped to the 3D model space in the rendering engine to generate a dynamic interactive mapping model. This model is used to determine whether the user actively brings the ball into the scene or passively receives objects in the scene, thereby helping 2D digital humans understand user behavior and intentions more accurately.

[0129] Furthermore, the real-time conversation analysis system based on 2D digital human also includes:

[0130] The target local operation model determination module 300 is used to determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object based on the user scene video, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model.

[0131] Specifically, Figure 7 It shows a structural block diagram of the target local operation model determination module 300 in the system provided by an embodiment of the present invention.

[0132] Among them, in the preferred embodiment provided by the present invention, the target local operation model determination module 300 specifically includes:

[0133] The target user motion part identification unit 301 is used to identify and determine the target user motion part that can actively manipulate the sudden dynamic interactive object based on the user scene video, and extract its coordinate information in the three-dimensional model space;

[0134] The local operation model determination unit 302 is used to determine the local operation model corresponding to each target user's motion part in the dynamic interaction mapping model according to the coordinate information of each target user's motion part in the three-dimensional model space;

[0135] The target local operation model determination unit 303 is used to calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model.

[0136] The target user movement part that can actively manipulate the sudden dynamic interactive object specifically refers to the part of the user's body that can directly manipulate or influence the sudden dynamic interactive object, including hands, mouth, arms, legs or feet.

[0137] In an embodiment of the present invention, the identification and determination of the target user's moving parts that can actively manipulate the sudden dynamic interactive objects needs to be achieved through image processing and computer vision technology in the user scene video. First, a deep learning algorithm (such as a posture estimation network, a key point detection algorithm) is used to identify the user's body parts, which include hands, mouth, arms, legs or feet, etc., which can directly manipulate or affect the sudden dynamic interactive objects. The algorithm obtains the 2D position data of each moving part of the user by detecting the user's skeletal key points (such as the OpenPose or MediaPipe algorithm), and generates the user's three-dimensional model data in combination with binocular vision, depth cameras or other technologies. These data are then converted into coordinate information in the three-dimensional model space to accurately describe the spatial position of the user's moving parts.

[0138] According to the coordinate information of each target user's moving parts in the three-dimensional model space, the system needs to further determine the local operation models corresponding to these moving parts in the dynamic interaction mapping model. Through coordinate matching and spatial mapping technology, the system compares the coordinate information of the user's moving parts in the three-dimensional model space with the various local operation models in the dynamic interaction mapping model. For example, if the coordinates of the user's hand in the three-dimensional space are shown to be close to an object, the system will identify the local operation model corresponding to the hand coordinates through mapping technology, and determine that the operation model can realize the manipulation of the object. In this way, the system can accurately identify the user's action intentions and interactive behaviors.

[0139] In the three-dimensional model space, the system matches all target user moving parts (for example, the user's hands, legs, and head) one by one based on these coordinate information. Through the coordinate matching algorithm, the system compares the three-dimensional coordinates of each moving part of the user with the local operation model in the dynamic interaction mapping model to find the local operation model that best matches each moving part. For example, the system may match the hand coordinates to the "hand grasping model", the leg coordinates to the "leg movement model", and the head coordinates to the "head tracking model".

[0140] This one-to-one matching process ensures that all target user motion parts in the user scene video can correspond to the appropriate local operation model. In this way, the system can fully capture and understand all the user's interaction intentions and behaviors in the scene, and effectively optimize the response and dialogue content of the 2D digital human.

[0141] An executable trajectory is a reasonable motion path that a user's moving part (such as a hand, leg, etc.) can perform under normal circumstances. It describes the natural motion that a specific body part can perform based on human kinematics and physiological limitations. For example, an executable trajectory for a hand may include grabbing, waving, pushing, and pulling, while an executable trajectory for a leg may include walking, kicking, etc. These trajectories start from a certain initial position and reach a possible end position, and take into account factors such as the speed, acceleration, and direction of the motion to reflect the natural range of motion and motion ability of the part.

[0142] First, the system extracts the motion trajectory of the sudden dynamic interactive object from the user scene video and determines the end position of the trajectory (such as the final position of a ball that suddenly flies into the scene). Next, predict the possible executable running trajectory for each user's moving part. For example, the executable trajectory of the hand starts from the initial position until it reaches the end point of the sudden object. When calculating these executable running trajectories, the system considers factors such as the starting position, speed, acceleration, and direction of the moving part. Then, the executable running trajectory of each local operation model is compared with the actual motion trajectory of the sudden dynamic interactive object through a trajectory similarity algorithm (such as the dynamic time warping algorithm DTW, Euclidean distance, or cosine similarity) to determine the degree of similarity between the two in path shape, length, and time characteristics. Finally, these similarities are defined as the matching degree of the local operation model to achieve the motion trajectory of the sudden dynamic interactive object.

[0143] By calculating and selecting the local operation model with the highest degree of match, the system can more accurately identify the user's actual operation intention and distinguish between the user's active operation and unintentional actions. This approach enables the system to dynamically adapt to various emergencies, whether the object suddenly enters the scene or is actively introduced by the user, thereby enhancing the flexibility of the system. In addition, after selecting the local operation model with the highest degree of match, the system can provide more accurate feedback and response, optimizing the user experience. By avoiding blind processing of all possible interaction situations, the system also makes more efficient use of computing resources and improves overall performance.

[0144] Furthermore, the real-time conversation analysis system based on 2D digital human also includes:

[0145] The active interaction intention strength generation module 400 is used to obtain the user operation video of the target user movement part corresponding to the target local operation model within the retrospective time period, and analyze the active interaction intention strength of the target local operation model for the movement trajectory based on the user operation video.

[0146] Specifically, Figure 8 It shows a structural block diagram of the active interaction intention intensity generating module 400 in the system provided by an embodiment of the present invention.

[0147] Among them, in the preferred implementation mode provided by the present invention, the active interaction intention intensity generation module 400 specifically includes:

[0148] The user operation video interception unit 401 is used to obtain the user scene video and intercept the user operation video within the retrospective time period;

[0149] A key action feature recognition unit 402 is used to analyze the user operation video and recognize key action features of the target user's motion part corresponding to the target local operation model;

[0150] The active interaction intention strength generating unit 403 is used to evaluate the trend correlation value between the key action feature and the motion trajectory of the target user's motion part, and define the trend correlation value as the active interaction intention strength of the target local operation model to the motion trajectory.

[0151] Furthermore, the real-time conversation analysis system based on 2D digital human also includes:

[0152] The preliminary association score adjustment module 500 is used to determine a correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjust the preliminary association score according to the correction ratio.

[0153] Specifically, Fig. 9 It shows a structural block diagram of the preliminary correlation score adjustment module 500 in the system provided by the embodiment of the present invention.

[0154] In the preferred embodiment of the present invention, the preliminary correlation score adjustment module 500 specifically includes:

[0155] A correction ratio calculation unit 501 is used to perform weighted processing on the matching degree of the target local operation model and the active interaction intention strength, and obtain the correction ratio through linear combination calculation;

[0156] A preliminary correlation score adjustment unit 502, configured to multiply the preliminary correlation score by the correction ratio to calculate an optimized correlation score;

[0157] The user active interaction judgment unit 503 is used to compare the optimized association score with the reference value to judge whether the sudden dynamic interaction object belongs to the user active interaction.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time conversation analysis method based on 2D digital human, characterized in that: The method comprises: When a sudden dynamic interactive object is detected in the user scene video, a preliminary correlation score of the sudden dynamic interactive object and a reference value for determining whether the sudden dynamic interactive object belongs to active user interaction are obtained; The preliminary relevance score is a value used to evaluate whether the sudden dynamic interactive object is actively introduced by the user, which is calculated by analyzing the user behavior and the object's motion trajectory; Obtain user scene video, extract the motion trajectory of sudden dynamic interactive objects and the user's three-dimensional model data from it, and render the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model; The generation process of the dynamic interaction mapping model is as follows: after obtaining the motion trajectory of the sudden dynamic interaction object and the user's three-dimensional model data, these data are imported into a three-dimensional coordinate system to establish a unified three-dimensional model space, and the user's three-dimensional model and motion trajectory are visualized through a rendering engine. The motion trajectory is represented by a connecting line or motion vector of the trajectory points, and the user model is rendered using a three-dimensional mesh or skeleton model inferred from a 2D image. During the rendering process, the pixel coordinates in the 2D image are converted into world coordinates in a three-dimensional space to ensure that the user's posture and the object's motion trajectory are accurately superimposed in the three-dimensional model space, thereby generating a dynamic interaction mapping model; Based on the user scene video, determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to achieve the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model; The steps of determining target user motion parts that can actively manipulate the sudden dynamic interactive object based on the user scene video, finding local operation models corresponding to these target user motion parts in the dynamic interaction mapping model, calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model include: Based on the user scene video, identify and determine the target user's moving parts that can actively manipulate the sudden dynamic interactive objects, and extract their coordinate information in the 3D model space; According to the coordinate information of the moving parts of each target user in the three-dimensional model space, the local operation model corresponding to the moving parts of each target user in the dynamic interactive mapping model is determined; Calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model; Obtaining a user operation video of the target user's motion part corresponding to the target local operation model within the retrospective time period, and analyzing the intensity of the active interaction intention of the target local operation model to the motion trajectory according to the user operation video; The correction ratio is determined according to the matching degree of the target local operation model and the strength of the active interaction intention, and the preliminary association score is adjusted according to the correction ratio.

2. The real-time conversation analysis method based on 2D digital human according to claim 1 is characterized in that: The steps of calculating the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, selecting the local operation model with the highest matching degree, and setting it as the target local operation model include: Predict the executable trajectory of each local operation model and identify the similarity between the executable trajectory and the motion trajectory of the sudden dynamic interaction object; The similarity between the executable motion trajectory of the local operation model and the motion trajectory of the sudden dynamic interaction object is defined as the matching degree of the motion trajectory of the sudden dynamic interaction object realized by the local operation model; The matching degree of all local operation models to realize the motion trajectory of the sudden dynamic interactive object is screened, and the local operation model with the highest matching degree is set as the target local operation model.

3. The real-time conversation analysis method based on 2D digital human according to claim 1 is characterized in that: The target user movement part that can actively manipulate the sudden dynamic interactive object specifically refers to the part of the user's body that can directly manipulate or influence the sudden dynamic interactive object, including hands, mouth, arms, legs or feet.

4. The real-time conversation analysis method based on 2D digital human according to claim 1 is characterized in that: The steps of obtaining a user operation video of a target user's motion part corresponding to the target local operation model within a retrospective time period, and analyzing the active interaction intention intensity of the target local operation model to the motion trajectory according to the user operation video include: Obtain user scenario videos and capture user operation videos within the retrospective time period; Analyze the user operation video and identify the key action features of the target user's moving parts corresponding to the target local operation model; The trend correlation value between the key action features and the motion trajectory of the target user's moving parts is evaluated, and the trend correlation value is defined as the active interaction intention intensity of the target local operation model for the motion trajectory.

5. The real-time conversation analysis method based on 2D digital human according to claim 4 is characterized in that: The steps of determining a correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjusting the preliminary association score by the correction ratio include: The matching degree of the target local operation model and the strength of the active interaction intention are weighted, and the correction ratio is calculated through linear combination; The optimized correlation score is calculated by multiplying the preliminary correlation score by the correction ratio; The optimized association score is compared with the benchmark value to determine whether the sudden dynamic interactive object belongs to the user's active interaction.

6. A real-time conversation analysis system based on 2D digital human, characterized in that: The system comprises: a sudden dynamic interaction object recognition module, a dynamic interaction mapping model generation module, a target local operation model determination module, an active interaction intention intensity generation module and a preliminary association score adjustment module, wherein: A sudden dynamic interaction object recognition module is used to obtain a preliminary relevance score of the sudden dynamic interaction object and a reference value for determining whether the sudden dynamic interaction object belongs to the user's active interaction after detecting that the sudden dynamic interaction object appears in the user scene video; The preliminary relevance score is a value used to evaluate whether the sudden dynamic interactive object is actively introduced by the user, which is calculated by analyzing the user behavior and the object's motion trajectory; A dynamic interaction mapping model generation module is used to obtain user scene videos, extract the motion trajectory of sudden dynamic interaction objects and the three-dimensional model data of the user, and render the extracted motion trajectory and user model into the three-dimensional model space to generate a dynamic interaction mapping model; The generation process of the dynamic interaction mapping model is as follows: after obtaining the motion trajectory of the sudden dynamic interaction object and the user's three-dimensional model data, these data are imported into a three-dimensional coordinate system to establish a unified three-dimensional model space, and the user's three-dimensional model and motion trajectory are visualized through a rendering engine. The motion trajectory is represented by a connecting line or motion vector of the trajectory points, and the user model is rendered using a three-dimensional mesh or skeleton model inferred from a 2D image. During the rendering process, the pixel coordinates in the 2D image are converted into world coordinates in a three-dimensional space to ensure that the user's posture and the object's motion trajectory are accurately superimposed in the three-dimensional model space, thereby generating a dynamic interaction mapping model; The target local operation model determination module is used to determine the target user's motion parts that can actively manipulate the sudden dynamic interactive object based on the user scene video, and find the local operation models corresponding to these target user's motion parts in the dynamic interaction mapping model, calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model; The target local operation model determination module specifically includes: A target user motion part recognition unit is used to recognize and determine the target user motion part that can actively manipulate the sudden dynamic interactive object based on the user scene video, and extract its coordinate information in the three-dimensional model space; A local operation model determination unit, used to determine the local operation model corresponding to each target user's motion part in the dynamic interaction mapping model according to the coordinate information of each target user's motion part in the three-dimensional model space; A target local operation model determination unit is used to calculate the matching degree of each local operation model to realize the motion trajectory of the sudden dynamic interactive object, select the local operation model with the highest matching degree, and set it as the target local operation model; An active interaction intention strength generation module is used to obtain a user operation video of a target user's motion part corresponding to a target local operation model within a retrospective time period, and analyze the active interaction intention strength of the target local operation model for the motion trajectory based on the user operation video; The preliminary association score adjustment module is used to determine the correction ratio according to the matching degree of the target local operation model and the strength of the active interaction intention, and adjust the preliminary association score according to the correction ratio.

7. The real-time conversation analysis system based on 2D digital human according to claim 6 is characterized in that: The active interaction intention intensity generation module specifically includes: A user operation video capture unit is used to obtain user scene videos and capture user operation videos within a retrospective time period; A key action feature recognition unit is used to analyze the user operation video and recognize the key action features of the target user's motion part corresponding to the target local operation model; The active interaction intention strength generating unit is used to evaluate the trend correlation value between the key action features and the motion trajectory of the target user's motion part, and define the trend correlation value as the active interaction intention strength of the target local operation model for the motion trajectory.

8. The real-time conversation analysis system based on 2D digital human according to claim 7 is characterized in that: The preliminary correlation score adjustment module specifically includes: A correction ratio calculation unit is used to perform weighted processing on the matching degree of the target local operation model and the strength of the active interaction intention, and obtain the correction ratio through linear combination calculation; A preliminary correlation score adjustment unit, used to multiply the preliminary correlation score by the correction ratio to calculate an optimized correlation score; The user active interaction judgment unit is used to compare the optimized association score with the reference value to judge whether the sudden dynamic interaction object belongs to the user active interaction.

Citation Information

Patent Citations

  • Pedestrian and personal sensitive article tracking method fusing visual relationship

    CN111325279A

  • 2D digital human interaction system and 2D digital human interaction method

    CN118298482A