Virtual human driving and action generating system applied to three-dimensional space interaction

Abstract

The invention relates to the technical field of virtual human driving and action generation, and discloses a virtual human driving and action generation system applied to three-dimensional space interaction, and the system comprises a rhythm heartbeat generation module which is used for generating an internal abstract rhythm signal of a virtual human; the multi-channel intention rhythm acquisition module is used for acquiring a plurality of parallel competitive intention rhythm signals; and the core driving module spontaneously selects a dominant signal from competitive intent rhythm signals to modulate abstract rhythm signals by using resonance characteristics of a nonlinear oscillator system, and deals with state self-sustaining when the signals are lost by an endogenous damping mechanism. And non-linear resonance is carried out on the multi-channel intention of the user, so that a continuous action which contains personalized details and adapts to signal uncertainty can be generated on universal hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to a virtual human driving and motion generation system applied to three-dimensional space interaction, belonging to the technical field of virtual human driving and motion generation. BACKGROUND

[0002] In the current three-dimensional space interaction application, in order to improve the immersion and interaction experience of users, the industry is generally committed to pursuing the high fidelity and natural feeling of virtual human motion. The mainstream technical path mainly relies on the fusion and reproduction of massive professional motion capture data or the high-precision simulation of virtual human and its interaction with the environment based on complex physical engines. These driving methods with the core of pursuing accurate reproduction of physical form can indeed generate highly realistic motions visually, and with the progress of graphic computing power, their application is becoming more and more widespread.

[0003] However, when we push virtual humans from specific performance environments to more extensive and low-cost mobile terminal or integrated XR device applications, the inherent limitations of the above mainstream technical path, which is designed to pursue high fidelity, large data, large models, and rigid driving, begin to emerge. This way transfers the high computing power overhead and ideal operating environment requirements to the terminal device and the user, forming a consensus dilemma that restricts the development of the industry, that is, there is a deep contradiction between the quality of virtual human interaction experience and the accessibility of its large-scale application. More deeply, this contradiction is not only reflected in the hardware cost, but also permeates the dimensions of real-time interaction, adaptability, and emotional expression. The stability and response ability of existing technologies in real and non-ideal interaction environments are facing continuous challenges, and the motion driven by fixed data sets or preset models also cannot reflect the user's constantly changing internal emotional state in real time and delicately.

[0004] Specifically, the prior art mainly has the following deficiencies: 1. The high computational load makes it difficult to achieve smooth and low-latency real-time interaction on a general-purpose device with limited computing power, limiting the popularization of applications; 2. The action generation logic is highly dependent on idealized sensor input and environmental data, and is vulnerable to signal noise, data loss or unexpected sudden conditions in the real world, which can easily lead to action distortion or system crash; 3. The action style is determined by the data set or model in advance, and lacks real-time deep coupling with the user's current internal emotional state, making it difficult to generate actions that reflect the user's personalized features. Therefore, how to build a new driving system that can work on general-purpose hardware with limited computing power without relying on large-scale data sets and precise physical simulation, which needs to be able to real-time nonlinearly couple the user's multi-channel high-order internal intention with the virtual person's underlying action rhythm, so as to generate continuous actions that can adapt to real-world signal uncertainty, rich in personalized emotional expression and have a sense of life, has become a technical problem to be solved by the present application. SUMMARY

[0005] The present application provides a virtual person driving and action generation system applied to three-dimensional space interaction, which mainly aims to solve the problem that the prior art relies on high computing power, large-scale data and precise physical simulation, making it difficult to realize real-time action generation that can adapt to real-world uncertainty and deeply couple with the user's internal emotion and has personalized life sense on general-purpose hardware.

[0006] To achieve the above purpose, the present application provides a virtual person driving and action generation system applied to three-dimensional space interaction, characterized in that the system comprises:

[0007] A rhythm heartbeat generation module, the rhythm heartbeat generation module is built-in nonlinear oscillator system, used for generating abstract rhythm signal as virtual person bottom layer action driving base frequency;

[0008] A multi-channel intention rhythm acquisition module, the multi-channel intention rhythm acquisition module acquires original input data from multiple independent sensors, and for each kind of original input data, generates an independent instantaneous intention rhythm signal;

[0009] a resonance coupling and state self-sustaining driving module, the resonance coupling and state self-sustaining driving module is configured to: when receiving a plurality of independent instantaneous intention rhythm signals, the plurality of independent instantaneous intention rhythm signals are taken as parallel competitive external driving forces, and are applied to a nonlinear oscillator system simultaneously, so as to utilize the resonance capture characteristics of the nonlinear oscillator system, to make the nonlinear oscillator system resonate with the instantaneous intention rhythm signal that is currently strongest in energy and most stable in rhythm spontaneously, and take the signal amplified after resonance as a dominant driving signal; and when detecting that the time gradient of any instantaneous intention rhythm signal is lower than a signal loss threshold, the application of all external driving forces is suspended, and an endogenous damping decay function is applied to the nonlinear oscillator system, so as to make the dynamic characteristics of the abstract rhythm signal return to a resting baseline state in an exponential law.

[0010] an animation mapping module, the animation mapping module is used for mapping the dominant driving signal or the abstract rhythm signal that returns under the action of the damping decay function to driving of overall skeletal animation parameters of the virtual human, to generate continuous actions.

[0011] Preferably, the multi-channel intention rhythm acquisition module extracts a volume envelope line of the audio data representing the voice of the user as a first instantaneous intention rhythm signal, and extracts a norm of angular velocity of the motion sensor data representing the limb movement of the user as a second instantaneous intention rhythm signal; the resonance coupling and state self-sustaining driving module realizes spontaneous selection of a dominant expression intention of the user by taking the first instantaneous intention rhythm signal and the second instantaneous intention rhythm signal as parallel competitive external driving forces.

[0012] Preferably, the application of the endogenous damping decay function makes the amplitude and frequency of the nonlinear oscillator system gradually attenuate in a manner conforming to biological habits in the working condition of signal transient loss, so that the action of the virtual human is smoothly transitioned to a resting state, to avoid mechanical behavior sudden stop.

[0013] Preferably, the system further comprises a state introspection module connected with the resonance coupling and state self-sustaining driving module; the state introspection module is configured to: while the resonance coupling and state self-sustaining driving module modulates and solves the nonlinear oscillator system, calculate a system chaos degree index R(t) representing the phase synchronization degree of each oscillation sub-phase in the system in real time, in the following manner: wherein R(t) is an instantaneous value of the system chaos degree index at time t, N is the total number of oscillation subs in the nonlinear oscillator system, e is the base number of a natural logarithm, i is an imaginary unit, θ j (t) is a phase of the jth oscillation sub as a state solution of the nonlinear oscillator system at time t; the system chaos degree index is used as an objective basis for quantifying the current running confidence of the virtual human system.

[0014] Preferably, the state introspection module is further configured to compare the real-time value of the system chaos degree indicator R(t) with at least one confidence threshold; and trigger a predefined meta-behavior when the system chaos degree indicator is lower than the confidence threshold, the meta-behavior being imposed on the virtual human through an independent animation overlay to externalize the level of internal running determinacy of the system as a body language that can be perceived by the user.

[0015] Preferably, the system further comprises an environmental rhythm extraction module for extracting an environmental pulse signal representing the environmental common rhythm from the sound information of the three-dimensional interactive environment through band-pass filtering and envelope detection; and the resonant coupling and state self-sustained driving module is further configured to apply the environmental pulse signal as a weakly coupled modulation signal to the nonlinear oscillator system, the coupling strength of the weakly coupled modulation signal being lower than that of any instantaneous intention rhythm signal, to utilize the injection locking effect to make the fundamental frequency of the abstract rhythm signal adaptively tend to synchronize with the environmental common rhythm while keeping the core dynamics dominated by the user intention.

[0016] Preferably, when the input of the multi-channel intention rhythm acquisition module is derived from a depth camera for acquiring three-dimensional spatial geometry data, the system further comprises a geometry constraint distortion suppression module between the depth camera and the multi-channel intention rhythm acquisition module for calculating the depth gradient variance of each local region in the three-dimensional spatial geometry data acquired by the depth camera in real time; and when the depth gradient variance of a certain region continuously exceeds an artifact threshold, interpolating and repairing the geometry data of the region using the reliable data adjacent to the region in the spatial and temporal dimensions to suppress the artifact distortion caused by physical reflection or projection.

[0017] Preferably, the nonlinear oscillator system is a group of Kuramoto oscillators with a number greater than or equal to two and mutually coupled; and the resonant coupling and state self-sustained driving module realizes the modulation of the nonlinear oscillator system by changing the inherent frequency of the Kuramoto oscillators or the coupling strength between the oscillators.

[0018] Preferably, each independent instantaneous intention rhythm signal generated by the multi-channel intention rhythm acquisition module has dynamic characteristics reflecting at least one dimension of the user's interactive intention, the dimension being selected from an energy level dimension determined by the voice volume and the amplitude of the body movement, an interactive inclination dimension determined by the emotional vocabulary and the tone rise and fall, and a certainty dimension determined by the continuity and clarity of the input signal.

[0019] Preferably, the animation mapping module is configured to map the amplitude of the abstract rhythm signal to an adjustment of the swing amplitude of the inverse dynamics target of the virtual human skeleton, and map the frequency of the abstract rhythm signal to an adjustment of the playback rate of the virtual human body posture animation clip, to achieve overall changes in the energy and rhythm of the virtual human motion.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] 1. By constructing the internal nonlinear dynamics system of the virtual human and making it nonlinearly resonant coupled with the multi-channel intention information of the external user, a new motion generation mechanism is provided. Since the motion of the virtual human is no longer a rigid reproduction of pre-stored data or a rigid calculation of physical rules, but a self-consistent dynamic state presentation of the internal rhythm heartbeat under the continuous disturbance of the user's intention field, the system can deduce motion sequences with personalized details and highly unified style on general-purpose hardware without relying on high computing power or large motion databases, avoiding the cost and computing power constraints faced by high-quality virtual human driving in popular applications.

[0022] 2. By setting up parallel inputs of multi-channel intention rhythm and using the internal resonant capture characteristics of the rhythm heartbeat oscillator system, a mechanism for handling multi-modal information conflicts is established for the virtual human. When facing complex interactive situations where the user's internal intentions are contradictory, the system does not simply weight and fuse or logically decide the conflicting input signals, but allows these signals to be competitive driving forces. According to its internal physical laws, the rhythm heartbeat, as the core dynamics system, spontaneously resonates with the current stable intention rhythm. This process converts a high-order attention focus selection cognitive problem into a low-cost and spontaneously emerging physical phenomenon, enabling the virtual human to exhibit a logically clear and focused human-like behavior when receiving ambiguous or contradictory signals, rather than falling into meaningless motion chaos, allowing the virtual human to express the user's dominant intention.

[0023] 3. By establishing an internal self-sustaining stability mechanism to cope with external signal transient loss in the resonant coupling and state self-sustaining driving module, the mechanism can smoothly switch the system driving mode from external modulation to internal damping attenuation by monitoring the time gradient of the intention rhythm signal in real time when judging signal transmission interruption. The motion rhythm of the virtual human will not be interrupted or frozen in a cliff-like manner, but will follow an exponential law consistent with physiological habits, gradually and smoothly returning to a natural resting state. This design ensures the continuity of the virtual human's life sense when encountering extreme working conditions such as network fluctuations or sensor signal loss in the real world, enhancing the continuity of the user experience of the entire system in actual deployment and application. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 System architecture diagram of the virtual human driving and motion generation system applied to three-dimensional space interaction of the present application;

[0025] Figure 2 Contrast diagram of behavior characteristics of the system of the present application under signal transient loss working condition;

[0026] Figure 3 System state transition logic diagram of the core driving module of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings, but it should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0028] The virtual human driving and motion generation system applied to three-dimensional space interaction disclosed in the present application mainly includes: a rhythm heartbeat generation module, a multi-channel intention rhythm acquisition module, a resonance coupling and state self-maintaining driving module, and an animation mapping module; in a typical application scenario, for example, a virtual human real-time interaction application running on a mobile terminal or an integrated XR device, there are generally technical challenges of limited computing resources and unstable sensor input signals in this scenario, and the present application is designed to address this challenge; in order to establish a dynamic basis that is consistent with biological intuition inside the virtual human, the system first constructs a kinetic core running at a lower computing cost through the rhythm heartbeat generation module, which internally has a nonlinear oscillator system, which in this embodiment is specifically implemented as a group of Kuramoto oscillators with a number greater than or equal to two and coupled to each other; without external input, the oscillator system can continuously generate an abstract rhythm signal with weak jitter and rhythm changes, which is used as the bottom driving fundamental frequency of all subsequent motions of the virtual human, so that the virtual human has natural micro-dynamics in the resting state, avoiding the rigidity problem caused by the static model in traditional technologies.

[0029] Further, to enable the virtual human's actions to reflect the user's internal interaction intention in real time, while avoiding the dependence on complex semantic recognition, the system is configured with a multi-channel intention rhythm acquisition module, which is used to extract multiple instantaneous intention rhythm signals from multiple independent universal sensor raw data in parallel; wherein, a preferred interaction path is to extract the volume envelope line of the audio data representing the user's voice through the envelope detection algorithm, and take the envelope line as the first instantaneous intention rhythm signal; at the same time, a parallel backup interaction path is to extract the norm of the angular velocity of the motion sensor data representing the user's body movement, such as the built-in gyroscope of the mobile device, and take the norm as the second instantaneous intention rhythm signal; the dynamic characteristics of each independent instantaneous intention rhythm signal reflect at least one dimension of the user's interaction intention, which is selected from the energy level dimension determined by the voice volume and the body movement amplitude; the interaction tendency dimension determined by the emotional vocabulary and the tone rise and fall; and the certainty dimension determined by the continuity and clarity of the input signal; such a multi-channel design ensures that the system can still obtain the user's interaction intention through other signal sources when a single signal source is missing or invalid.

[0030] In real interaction, the intention expressed by the user through different channels may be ambiguous or conflicting, and signal loss is also a common working condition. To cope with this core challenge, the core driving logic of the system is executed by a resonance coupling and state self-sustaining driving module; the module receives the first and second instantaneous intention rhythm signals and takes them as parallel and mutually competitive external driving forces, which are applied to the nonlinear oscillator system in the rhythm heartbeat generation module; according to the resonance capture physical characteristics of the nonlinear oscillator system, the system will spontaneously resonate with the instantaneous intention rhythm signal with the strongest energy and the most stable rhythm, and the resonance will sharply amplify the captured signal, so that it is selected as the only dominant driving signal; at the same time, the module also establishes a self-sustaining stabilization mechanism to cope with the transient loss of external signals in an endogenous way, which monitors the time gradient of any instantaneous intention rhythm signal in real time, and judges that the signal transmission is interrupted when the gradient is continuously lower than a preset signal loss threshold; the calibration procedure of the signal loss threshold includes: recording the maximum gradient value caused by background noise when the system receives stable continuous signals, and setting the threshold to a certain value slightly higher than the maximum gradient value; once the interruption is judged, the module will suspend the application of all external driving forces, and apply an endogenous damping decay function, such as exponential decay function e ―ktwherein the attenuation coefficient k is determined by an offline calibration procedure aiming to solve the corresponding coefficient value according to a preset target attenuation duration T, for example, 1.5 seconds; the application of damping causes the amplitude and frequency of the abstract rhythm signal to gradually attenuate in a manner consistent with biological inertia, thereby smoothing the transition of the virtual human's motion to a resting state, and this mechanism ensures the continuity of the virtual human's life sense.

[0031] To further enhance the realism of the interaction, so that the virtual human not only resonates with the user, but also coordinates with the atmosphere of the environment, the system also includes an environment rhythm extraction module for extracting an environment pulse signal representing the public rhythm of the environment from the sound information of the three-dimensional interactive environment through band-pass filtering and envelope detection; the resonant coupling and state self-maintaining driving module is also configured to apply the environment pulse signal as a weakly coupled modulation signal to the nonlinear oscillator system, and the coupling strength of the weakly coupled modulation signal is set to be lower than the coupling strength of any instantaneous intention rhythm signal, so as to utilize the injection locking effect to make the fundamental frequency of the abstract rhythm signal adaptively and non-invasively tend to synchronize with the public rhythm of the environment while maintaining the core dynamics dominated by the user's intention; when the input of the multi-channel intention rhythm acquisition module of the system is derived from a depth camera used to acquire three-dimensional spatial geometry data, to deal with the aliasing distortion problem caused by reflective surfaces such as glass and mirrors in the real physical environment, the system also includes a geometric constraint distortion suppression module; the module is located between the depth camera and the multi-channel intention rhythm acquisition module, and is used to calculate the depth gradient variance of each local region in the three-dimensional spatial geometry data acquired by the depth camera in real time; when the depth gradient variance of a certain region continuously exceeds the aliasing threshold, the system uses the spatially and temporally adjacent reliable data of the region to interpolate and repair the geometry data of the region; the determination procedure of the aliasing threshold is as follows: by statistically analyzing a large amount of sample data containing real object surfaces and typical reflection aliasing, the threshold is set to a segmentation value that can effectively distinguish the variance distribution of the two types of data.

[0032] To enable the system to perceive and express the certainty level of its internal operation, the system also includes a state introspection module that calculates the system chaos degree index N(t) representing the degree of synchronization of each oscillation phase in the system while the resonant coupling and state self-maintaining driving module modulates and solves the nonlinear oscillator system in parallel, and the calculation method is as follows: wherein N is the total number of oscillators, e is the base of natural logarithm, i is the imaginary unit, and θ jR(t) represents the phase of the j-th oscillator at time t; the range of R(t) is [0,1], where a value close to 1 indicates high internal synchronization and high operational confidence, while a value close to 0 indicates internal system chaos and low confidence; furthermore, the state introspection module compares the real-time value of the system chaos index R(t) with at least one confidence threshold, and when the index is below the threshold, for example, R(t) < 0.3, triggers a predefined meta-behavior, which is applied to the virtual human through an independent animation overlay, thereby externalizing the system's inherent operational determinism into body language that can be perceived by the user; finally, the rhythmic signal output by the resonant coupling and state self-sustaining drive module is transmitted to An animation mapping module is used to transform this abstract, low-dimensional rhythmic signal into a driver for the overall skeletal animation parameters of the virtual human. To this end, the animation mapping module is configured to: map the amplitude of the abstract rhythmic signal to adjust the swing amplitude of the inverse dynamic target of the virtual human skeleton, and simultaneously map the frequency of the abstract rhythmic signal to adjust the playback rate of the virtual human body posture animation clips. Through this decoupled mapping method, the system can achieve the overall and coordinated changes in the energy and rhythm of the virtual human's movements without the need for a large motion database or complex physics engine calculations. Ultimately, on hardware with limited computing power, it can generate continuous movements rich in personalized details and adaptable to the uncertainties of the real world.

[0033] Example 1: In a virtual human interaction application deployed on a mobile smart terminal for high-risk remote business negotiations, User A communicates with User B in real time through the virtual human he controls. At a critical moment in the negotiation, User A's voice tone remains stable, and it is parsed by the multi-channel intention rhythm acquisition module through the audio channel as a first instantaneous intention rhythm signal with low amplitude and stable frequency characteristics. However, at the same moment, due to inner tension, the user's limbs holding the mobile terminal involuntarily produce high-frequency micro-tremors. These micro-tremors are captured by the motion sensor built into the terminal and parsed by the module as a second instantaneous intention rhythm signal with high amplitude but irregular frequency characteristics.

[0034] Given the inherent conflict of such multimodal signals, systems relying on traditional signal weighting fusion algorithms will produce an averaged and chaotic action without clear intention due to input cancellation. However, the resonant coupling and state-sustaining driving module of this invention uses these two independent and competing instantaneous intention rhythm signals as parallel external driving forces, simultaneously applied to its internal nonlinear oscillator system. The discrete data provided by the multi-channel intention rhythm acquisition module creates the preconditions for the nonlinear oscillator system to exert its physical selection characteristics. According to the principle of resonance capture, the energy-dominant second instantaneous intention rhythm signal causes the oscillator system to resonate with it, and its amplitude is sharply amplified, while the first instantaneous intention rhythm signal representing stable speech is suppressed. The system thus transforms a high-order problem of adjudicating cognitive intention conflicts into a physical process in which the internal dynamic mechanism spontaneously completes the selection based on the physical properties of the signal.

[0035] Correspondingly, during this period, the state introspection module detects that the phase synchronization of the internal oscillators has decreased due to the system being driven by a strong but irregular signal. The calculated system chaos index R(R) drops from a high confidence range, such as 0.85, to a medium confidence range, such as 0.5, but this value does not fall below the preset confidence threshold used to trigger meta-behavior. Finally, the amplified dominant driving signal after resonance is transmitted to the animation mapping module. As a result, in the perspective of user B, user A's virtual human maintains cooperation in verbal language, but its body posture exhibits a non-relaxed defensive slight tilt. At the same time, the playback rate of its hand gesture animation also shows a slight and irregular acceleration corresponding to the limb tremor. These non-verbal behaviors, which emerge from the system and are highly consistent with the user's internal physiological rhythm, convey more fundamental interactive information to user B than verbal content. This process makes the final manifestation of the virtual human behavior output by the system determined by an internal physical competition mechanism based on signal energy, rather than by an external decision rule based on semantics or logic.

[0036] Example 2: To objectively verify the actual performance of the technical solution of the present invention on a hardware platform with limited computing resources in terms of core performance indicators, the following comparative experiment was conducted. The purpose of this experiment is to quantitatively evaluate the differences between the system of the present invention and a conventional system based on a lightweight neural network under the same hardware and input conditions in terms of computational overhead, response latency, and robustness to non-ideal signal input. The experimental platform was built on a mid-range mobile smart terminal with hardware configurations that are currently mainstream in the market, to simulate the operating environment of a widely used device. Two parallel virtual human driving systems were deployed on this platform: System A, which is the complete technical solution of the present invention; System B, which is... As a control group, the standard technical solution employed a pre-trained lightweight long short-term memory network model for fusing multi-sensor inputs and selecting corresponding animation clips. The experimental input source used a pre-recorded dataset containing raw audio and motion sensor data streams simulating different user interaction scenarios to ensure consistent input conditions between the two experiments. During the experiment, the sampling frequency of the sensor data was set to 30Hz. This value was set to balance the real-time performance of data acquisition with the system processing burden. The decision was based on the following principle: while meeting the Nyquist sampling requirements for general human speech envelopes and body movements, a certain degree of redundancy was retained to cope with sudden high-frequency movements.

[0037] After the experiment was started, System A and System B were loaded and run to drive the same virtual human model, and their key indicators were recorded using performance analysis tools. In the specific case of transient signal loss, i.e., when the motion sensor signal flow was interrupted for 500ms, the two systems were observed to exhibit significantly different behaviors. The virtual human in System A maintained dynamic inertia in the initial stage of signal loss, and then its movement amplitude and frequency gradually decayed, eventually transitioning to a natural resting posture. In contrast, the virtual human in System B froze its movements at the moment of signal loss, exhibiting a mechanical behavioral stop, and only resumed movement after the signal was restored. See Table 1, which records the performance data of the two systems under several typical conditions.

[0038] Table 1: Performance data comparison between System A and System B under different operating conditions.

[0039] Test conditions Test system Average CPU occupancy (%) Response latency (ms) Behavioral robustness score (1-10) Smooth input System A 1.8 12 10 Smooth input System B 6.5 35 10 High energy input System A 2.1 14 9 High energy input System B 7.2 41 8 Signal transient loss System A 1.5 N / A 9 Signal transient loss System B 3.1 N / A 2 Contradictory signal input System A 2.0 16 9 Contradictory signal input System B 6.8 45 4

[0040] The data differences shown in Table 1 stem from the different internal mechanisms of the two systems. The high behavioral robustness of System A under transient signal loss is a direct result of the triggering of the endogenous damping decay function in its resonant coupling and state-sustaining drive module. This mechanism enables the system's dynamic core to autonomously and smoothly return to the baseline state in a manner consistent with physiological inertia after losing external driving force. In contrast, the drive logic of System B relies on continuous external data input in its architecture. Once the input stream is interrupted, its neural network model cannot generate effective output, which inevitably leads to action freeze. In addition, under all operating conditions, the CPU utilization and response latency of System A are lower than those of System B. This is because the core operation of System A is based on solving a lightweight nonlinear oscillator system, and its computational complexity is lower than the neural network inference operation required by System B.

[0041] Example 3: This example combines Figures 1 to 3 This paper describes the implementation of a virtual human driving and motion generation system applied to three-dimensional spatial interaction, such as... Figure 1 As shown, the system acquires information from multiple external input sources, specifically including audio data of user speech and motion sensor data of user body movements. A multi-channel intention rhythm acquisition module generates parallel, competitive instantaneous intention rhythm signals. Simultaneously, the system can also acquire 3D spatial geometric data from a depth camera. This data is first processed by a geometric constraint distortion suppression module to suppress artifacts caused by physical reflections, and the repaired geometric data is then input to the multi-channel intention rhythm acquisition module. Furthermore, the system extracts environmental pulse signals from the sound information of the 3D interactive environment as weakly coupled modulation through an environmental rhythm extraction module. All these signals converge into the core resonant coupling and state self-sustaining drive module. The module spontaneously selects the dominant signal using the resonant characteristics of a nonlinear oscillator and employs an endogenous damping mechanism to handle signal loss. Its operating state is modulated and driven by the fundamental frequency of the virtual human's underlying motion generated by the rhythmic heartbeat generation module. Simultaneously, a state introspection module monitors the real-time state of the core driving module, quantifies the system's operational confidence by calculating the system's chaos index, and triggers predefined meta-behaviors when the confidence is too low. Finally, the dominant driving signal output by the core driving module is sent to the animation mapping module, which maps the abstract rhythmic signal characteristics to drive the virtual human's skeletal animation parameters, thereby generating continuous virtual human motions rich in personalized details and adaptable to signal uncertainties.

[0042] like Figure 2As shown in the figure, the horizontal axis represents time in seconds (s), and the vertical axis represents the normalized motion amplitude. The dashed line representing system B indicates that its motion amplitude drops instantaneously from 1.0 to 0 when the signal is interrupted at time t = 2.0s, exhibiting a kind of mechanical stop. In contrast, the solid line representing system A of the present invention shows that at the same time point, its motion amplitude begins to decay smoothly and gradually in an exponential manner, reflecting the smooth decay characteristics that conform to biological inertia, ultimately ensuring the continuity of user experience.

[0043] like Figure 3 As shown, after system initialization, the system enters a resting state. At this time, the rhythmic heartbeat module spontaneously generates abstract rhythmic signals, enabling the virtual human to exhibit natural, subtle dynamics. Upon receiving an instantaneous intentional rhythmic signal, the system transitions to an externally driven resonance state. In this state, it receives multi-channel intentional rhythmic signals, causing the nonlinear oscillator system to resonate with the dominant signal, and the animation mapping module generates the main actions. During this resonance state, if the system's chaos index falls below the confidence threshold, a meta-behavior superposition state is triggered. In this state, the state introspection module calculates the low confidence, triggers predefined meta-behaviors, and applies them through an independent animation superposition layer. Once the chaos recovers or the behavior ends, the system returns to the externally driven resonance state. If the signal time gradient is detected to be below the loss threshold in the resonance state, the system transitions to an endogenous damping decay state. In this state, all external driving forces are suspended, and an endogenous damping decay function is applied, allowing the action to smoothly transition to the resting state. Finally, after the system returns to the resting baseline, it returns to the initial resting state.

[0044] Example 4: In the final calibration and verification phase before the commercial deployment of a virtual human system, the core engineering challenge lies in how to transform the advanced behaviors exhibited by the system in response to uncertain inputs into a deterministic system rigorously defined by a series of objective and reproducible engineering parameters. To achieve this goal, the system chaos index R(t) and the meta-behavior triggering logic in the state introspection module are first calibrated. The initial state of this process is defined as: a state containing manually labeled, clearly defined intentions, vague intentions, and contradictory intentions. Figure Three The calibration process involves a library of input signal segments categorized by type. After the process starts, the signals in the library are sequentially input into the uncalibrated system, and the instantaneous R(t) values ​​generated by the nonlinear oscillator system at each time step are recorded synchronously. After data acquisition, the probability distribution statistics of the R(t) value set belonging to the clear intention category and the R(t) value set belonging to the fuzzy intention category are performed. The confidence threshold used to trigger the first-level hesitant element behavior is set at the fifth percentile of the clear intention distribution, which is the statistical boundary value. Similarly, the threshold for triggering the second-level confused element behavior is set at the fifth percentile of the fuzzy intention distribution. This calibration process quantifies a confidence concept into two statistically based deterministic floating-point thresholds.

[0045] Next, the endogenous damping attenuation function in the resonant coupling and state-sustaining drive module was calibrated. The aim was to ensure that the virtual human's motion attenuation process after experiencing signal transient loss visually conformed to the general human expectation of biological inertia. The quantification procedure for this process involved multiple testers performing double-blind scoring on the naturalness of the motion transition of the virtual human from a standard high-energy action state to a resting state under different attenuation coefficients k. Finally, the attenuation coefficient k with the highest arithmetic mean score was selected as the system's global default configuration. This process assigned specific parameters to the mathematical function that conformed to the principles of cognitive science. Finally, the meta-behavior was... The animation overlay mechanism is configured such that for each meta-behavior triggered by a confidence threshold, the blending mode applied to the virtual human skeleton in the animation system is set to overlay, and its blending weight is set to a fixed low value, namely 0.2. This configuration ensures that the action of the meta-behavior is a kind of micro-expression or body language, which is slightly and non-invasively overlaid on the basic action generated by the dominant driving signal, rather than interrupting or replacing the latter. Through this set of calibration and verification processes, all key non-linear behaviors in the system are transformed into engineered modules controlled by deterministic parameters and processes, and their performance in any deployment environment has traceability and consistency.

[0046] Example 5: When deploying this system in a commercial hall containing a large glass curtain wall and highly reflective metal decorations, to ensure that the geometric constraint distortion suppression module can accurately distinguish between real physical surfaces and artifacts generated by the depth camera due to physical reflection, a preliminary visual environment calibration procedure needs to be performed when the system is first started. The steps of this procedure are as follows: First, the depth camera is operated to continuously scan a known standard physical surface in the scene that has diffuse reflection characteristics, and the depth gradient variance of each local region in the acquired three-dimensional spatial geometric data is calculated in real time, and its maximum value V in the steady state is recorded. base Subsequently, the depth camera was aimed at a highly reflective surface in the scene, and the minimum value V of its depth gradient variance was recorded again. artifact Finally, the artifact threshold used to distinguish artifacts within the geometric constraint distortion suppression module is deterministically set to V. base +0.5×(V artifact ―V base This procedure enables the system to adaptively establish its artifact recognition benchmark based on the optical environment characteristics of a specific deployment scenario.

[0047] To enable the environmental rhythm extraction module to effectively extract the core pulse signal representing the common environmental rhythm from the unique complex acoustic environment of the commercial hall, an acoustic environment adaptive procedure needs to be executed simultaneously. This procedure involves acquiring a 60-second sample of the ambient sound using a microphone and performing a Fast Fourier Transform on the sample to obtain the sound spectrum distribution of the scene. The system then analyzes the spectrum to identify the dominant frequency bands with stable and continuous energy. Based on the upper and lower boundary frequencies of these dominant frequency bands, the passband range of the built-in bandpass filter in the environmental rhythm extraction module is automatically set. This procedure ensures that the system can separate the common environmental rhythm required for weak coupling modulation from the complex acoustic environment, thereby guaranteeing the effectiveness of the synchronization between the virtual human's movements and the environmental atmosphere.

[0048] Example 6: When instantiating and deploying a virtual human driving and motion generation system, to ensure that its core rhythmic heartbeat generation module can generate a stable and inherently complex resting fundamental frequency that conforms to biological characteristics without external intention input, an initial parameter matrix generation procedure for the module's built-in nonlinear oscillator system needs to be executed. This procedure first sets the natural frequencies of the N Kuramoto oscillators in the oscillator system. The setting method is to randomly sample and assign values ​​from a Lorentz distribution with a center frequency of 0.5 Hz and a full width at half maximum (FWHM) of 0.2 Hz. This aims to inject a certain initial state into the system. This process helps generate complex dynamic inherent heterogeneity. Subsequently, the global coupling strength K between oscillators is calibrated. This process starts with K=0 and gradually increases the value of K in a fixed step size. At each step size, the mean value of the system chaos index R(t) output after the system is running stably is monitored until the mean value of R(t) is observed to be in a non-synchronous state that never approaches zero. The system then transitions and stabilizes in an intermediate value range that represents partial synchronization, namely the interval between 0.4 and 0.6. The corresponding K value at this point is determined as the reference coupling strength of the system. This procedure initializes the system at a critical dynamic point to ensure its sensitivity to subsequent external intentions.

[0049] To enhance the system's tolerance to sensor signal anomalies in real-world physical environments, particularly to address input signal saturation or clipping distortion caused by sudden loud noises or violent movements, an input saturation suppression logic gate is integrated into the front end of the multi-channel intention rhythm acquisition module. This logic gate operates by real-time monitoring of the raw input data stream from the microphone or motion sensor and identifying clipping events by determining whether the signal amplitude has continuously exceeded three sampling points while remaining at its quantifiable maximum or minimum value. Once a clipping event is detected, the logic gate prevents the distorted signal from being directly used to generate the instantaneous intention rhythm signal. Instead, it clamps the amplitude of the instantaneous intention rhythm signal output from that channel to a preset upper limit threshold. This upper limit threshold is determined during offline calibration by analyzing the maximum amplitude distribution of all legal signal samples under the high-energy input category. This clamping state continues until the raw input signal de-saturates and remains saturated for an additional fixed, short period. This procedure protects the core resonant coupling drive module from non-physiological saturation signal impacts, thereby preventing the virtual human from generating meaningless violent movements.

[0050] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A virtual human driving and motion generation system for three-dimensional spatial interaction, characterized in that, The system includes: A rhythmic heartbeat generation module, which has a built-in nonlinear oscillator system, is used to generate an abstract rhythmic signal that serves as the fundamental frequency for driving the underlying movements of the virtual human. A multi-channel intention rhythm acquisition module acquires raw input data from multiple independent sensors and generates an independent instantaneous intention rhythm signal for each type of raw input data. A resonant coupling and state-sustaining driving module is configured to: when receiving multiple independent instantaneous intention rhythm signals, apply the multiple independent instantaneous intention rhythm signals as parallel competitive external driving forces to a nonlinear oscillator system, so as to utilize the resonant capture characteristics of the nonlinear oscillator system to spontaneously resonate with the current instantaneous intention rhythm signal with the strongest energy and the most stable rhythm, and use the amplified signal after resonance as the dominant driving signal; And when the time gradient of any instantaneous intentional rhythm signal is detected to be lower than the signal loss threshold, the application of all external driving forces is suspended, and an endogenous damping decay function is applied to the nonlinear oscillator system so that the dynamic characteristics of the abstract rhythm signal smoothly return to a resting reference state according to an exponential law. An animation mapping module is used to map the dominant driving signal or the abstract rhythmic signal regressing under the action of the damping decay function into the driving force of the overall skeletal animation parameters of the virtual human to generate continuous motion.

2. The virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The multi-channel intention rhythm acquisition module extracts the volume envelope from the audio data representing the user's speech as the first instantaneous intention rhythm signal, and extracts the norm of the angular velocity from the motion sensor data representing the user's limb movements as the second instantaneous intention rhythm signal; the resonant coupling and state self-sustaining drive module uses the first instantaneous intention rhythm signal and the second instantaneous intention rhythm signal as parallel competing external driving forces.

3. The virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The application of the endogenous damping decay function causes the amplitude and frequency of the nonlinear oscillator system to gradually decay in a manner consistent with biological inertia under the condition of signal transient loss, thereby enabling the virtual human's movements to smoothly transition to a resting state.

4. The virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The system also includes a state introspection module, which is connected to the resonant coupling and state self-sustaining drive module. The state introspection module is configured to: while the resonant coupling and state self-sustaining drive module modulates and solves the nonlinear oscillator system, simultaneously calculate the system chaos index R(t), which characterizes the phase synchronization degree of each oscillator within the system. The calculation method is as follows: Where R(t) is the instantaneous value of the system's chaos index at time t, N is the total number of oscillators in the nonlinear oscillator system, e is the base of the natural logarithm, i is the imaginary unit, and θ j (t) represents the phase of the j-th oscillator, which is the state solution of the nonlinear oscillator system, at time t; the system chaos index is used as an objective basis for quantifying the current operating confidence of the virtual human system.

5. A virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 4, characterized in that, The state introspection module is also configured to compare the real-time value of the system chaos index R(t) with at least one confidence threshold; and when the system chaos index is below the confidence threshold, to trigger a predefined meta-behavior, which is applied to the virtual human through a separate animation overlay to externalize the system’s inherent operational determinism into body language that can be perceived by the user.

6. The virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The system also includes an environmental rhythm extraction module, which extracts environmental pulse signals representing the common environmental rhythm from the sound information of the three-dimensional interactive environment through bandpass filtering and envelope detection. The resonant coupling and state self-sustaining drive module is further configured to apply the environmental pulse signal as a weakly coupled modulation signal to the nonlinear oscillator system. The coupling strength of this weakly coupled modulation signal is lower than that of any instantaneous intention rhythm signal, so as to utilize the injection lock-in effect to make the fundamental frequency of the abstract rhythm signal adaptively tend to synchronize with the common environmental rhythm while maintaining the core dynamics dominated by the user intention.

7. A virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, When the input to the multi-channel intention rhythm acquisition module comes from a depth camera used to acquire 3D spatial geometric data, the system also includes a geometric constraint distortion suppression module. This geometric constraint distortion suppression module is located between the depth camera and the multi-channel intention rhythm acquisition module. It is used to calculate the depth gradient variance of each local region in the 3D spatial geometric data acquired by the depth camera in real time. When the depth gradient variance of a certain region is continuously higher than the artifact threshold, the geometric data of that region is interpolated and repaired using reliable data that are adjacent to that region in the spatial and temporal dimensions.

8. A virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The nonlinear oscillator system consists of a set of two or more mutually coupled Kuramoto oscillators; the resonant coupling and state-sustaining drive module changes the natural frequency of the Kuramoto oscillators or the coupling strength between the oscillators.

9. A virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, Each independent instantaneous intention rhythm signal generated by the multi-channel intention rhythm acquisition module reflects at least one dimension of the user's interaction intention in terms of its dynamic characteristics. The dimensions are selected from the energy level dimension determined by the volume of free speech and the amplitude of body movements, the interaction tendency dimension determined by emotional vocabulary and tone rise and fall, and the determinism dimension determined by the continuity and clarity of the input signal.

10. A virtual human driving and motion generation system for three-dimensional spatial interaction according to claim 1, characterized in that, The animation mapping module is configured to map the amplitude of the abstract rhythm signal to an adjustment of the swing amplitude of the inverse dynamic target of the virtual human skeleton, and to map the frequency of the abstract rhythm signal to an adjustment of the playback rate of the virtual human body posture animation clips.

Images