Multi-modal digital human content full-link generation method based on human-agent

Abstract

The present application relates to the technical field of artificial intelligence and computer graphics, and discloses a multi-modal digital human content full-link generation method based on human-agent, which comprises the following steps: performing analysis on streaming text data, and outputting a text block sequence and a micro-intention instruction stream; performing acoustic waveform synthesis and macro motion parameter table lookup in parallel in a double-track thread; writing generated audio waveform block data, heterogeneous micro-movement and macro motion parameter vectors into a delay queue; calculating local time deformation rates of acoustic peaks and displacement key frames, triggering timing backpressure signaling or performing timing alignment interpolation based on a kinematics boundary threshold; calling a sparse mapping operator to map heterogeneous motion parameters to an isomorphic space and assign weights, and combining a collision truncation detection mechanism based on the shortest spatial Euclidean distance to generate a final fused motion parameter tensor. The present application solves the problems of audio-visual timing dislocation and model interference, and improves the physical rationality and stability of streaming rendering.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer graphics technology, specifically to a method for generating multimodal digital human content across the entire chain based on a persona agent. Background Technology

[0002] With the development of artificial intelligence technology, text-driven multimodal digital humans are widely used in virtual interaction scenarios. The existing digital human streaming generation process needs to generate speech and torso movements synchronously according to the input text instructions. When processing asynchronous input streaming instructions, acoustic feature synthesis and the acquisition of macroscopic body motion parameters are usually in independent data processing chains. Due to the natural differences between the speech production cycle and the action execution cycle, existing methods are difficult to achieve temporal alignment between the two. Conventional methods usually forcibly stretch or compress the action timeline, which causes the rendered output action to deviate from the natural human kinematics, producing local acceleration or lag of the action, and thus causing temporal misalignment between speech and action presentation.

[0003] Furthermore, the final rendering of multimodal digital humans requires the merging and driving of facial micro-movements and torso macro-movements. Existing technologies generally use data overlay to directly input data into the 3D graphics rendering pipeline when processing the fusion of heterogeneous motion parameters, without introducing interference detection and dynamic constraint mechanisms in the physical space dimension. When motion parameters without spatial topology verification are overlaid and applied to the 3D mesh model, conflicts in local joint rotation angles can easily lead to clipping interference phenomena, such as hands penetrating the torso or head. This rendering error caused by the lack of physical deformation constraints not only undermines the visual rationality of the digital human presentation but also reduces the stability of continuous audio and video stream output. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for the full-link generation of multimodal digital human content based on the Agent persona, which solves the problems of temporal misalignment between speech and action presentation and interference caused by multimodal action merging in existing digital human streaming generation.

[0005] To address the above problems, the present invention provides the following technical solution: This invention provides a method for generating multimodal digital human content across the entire chain based on a persona agent, the method comprising the following steps: The system acquires streaming text data output from an external large language model and the current digital human identity identifier. It performs discretization truncation and semantic feature extraction on the streaming text data, and outputs a sequence of text blocks with independent boundaries and a micro-intent command stream, respectively. The system receives the text block sequence, the micro-intent instruction stream, and the current digital human identity identifier. It performs acoustic waveform synthesis inference and macroscopic kinematic parameter lookup operations in parallel on independent data tracks to generate audio waveform block data, heterogeneous micro-motion parameter vectors, and heterogeneous macroscopic motion parameter vectors. The generated data sources are then centrally written into the delayed queue submodule for queue caching. Read the data source cached in the delay queue submodule, extract the acoustic peak time series and displacement key frame time series and calculate the optimal alignment path, calculate the local time deformation rate based on the optimal alignment path, and suspend the timing alignment operation and trigger timing backpressure signaling when it is determined that the local time deformation rate value deviates from the preset kinematic boundary threshold. The heterogeneous micro-motion parameter vector and heterogeneous macro-motion parameter vector, which have been aligned by the time axis, are subjected to isomorphic space remapping operation. Nonlinear interpolation and Hadamard product allocation are performed using the retrieved character joint priority weight tensor. Combined with the spatial topology collision truncation detection mechanism, the final fused motion parameter tensor in the isomorphic state space is generated. The final fused action parameter tensor and the processed audio waveform block data are subjected to a presentation timestamp synchronization encapsulation operation based on a common clock reference, which drives the rendering pipeline to dynamically update the mesh skeleton transformation matrix and encode and output multimedia audio and video streams in real time.

[0006] Preferably, the step of outputting the text block sequence with independent boundaries and the micro-intent command stream respectively includes: The streaming text data is discretized using a dynamic delimitation algorithm, which converts the streaming text data without physical boundaries into a sequence of text blocks containing explicit start and end bytes, and assigns a unique block identifier to each text block. The text block sequence is controlled to enter the sliding text window component in sequence, and a local context window for semantic feature extraction is established in the memory area; Scan the character entity features contained in the current independent text block residing in the local context window, input the character entity features into a pre-configured action mapping dictionary for query and comparison, extract the action classification label, emotion polarity scalar, action amplitude parameter and expected action cycle that have a mapping relationship with the character entity features, and use the extracted parameter set to encapsulate and generate the micro-intent command stream.

[0007] Furthermore, the system utilizes a dynamic delimiting component to establish a sliding reading interval, establishes physical boundaries for continuously input text streams, and extracts multi-dimensional semantic parameter combinations carried by the text through a rule engine, providing structured parameter guidance for downstream concurrent generation.

[0008] Preferably, the step of generating audio waveform block data and heterogeneous micro-motion parameter vectors includes: The text-to-speech engine is invoked to perform front-end text normalization and acoustic feature prediction on the text block sequence, and the audio waveform block data is synthesized and output; a two-dimensional feature matrix containing the Mel frequency band is generated by applying the short-time Fourier transform algorithm, and the two-dimensional feature matrix is ​​input into the acoustically driven inference network to perform forward propagation operation; Read the sentiment polarity scalar bound to the unique identifier of the current text block, and input the sentiment polarity scalar as an auxiliary control parameter into the parameter output layer of the acoustically driven inference network to adjust the facial blending deformation weights and output the heterogeneous micro-motion parameter vector with constant dimension.

[0009] Furthermore, by constructing an acoustically driven inference network based on a long short-term memory network, facial driving primitives are extracted from waveform features, and semantic-level emotional polarity scalars are introduced to adjust the weights of local features, thereby achieving cross-modal conversion of acoustic feature facial micro-motion parameters.

[0010] Preferably, the step of generating the heterogeneous macroscopic motion parameter vector includes: reading the motion classification labels contained in the micro-intent instruction stream; performing a structured concatenation operation with delimiters on the motion classification labels and the current digital human identity identifier to generate a hash key value; using the hash key value to perform address matching in the pre-loaded motion primitive hash table in memory to obtain the skeletal node rotation quaternion sequence bound to the hash key value; performing time scale pre-adjustment on the skeletal node rotation quaternion sequence according to the expected motion cycle; converting it into Euler angle matrix format and outputting the heterogeneous macroscopic motion parameter vector; when the hash key value fails to hit a valid storage address in the motion primitive hash table, calling the default standby skeletal pose constant matrix preset in the system configuration file to replace the missing spatial motion data and outputting the heterogeneous macroscopic motion parameter vector.

[0011] Furthermore, a locality-sensitive hashing (LSH) retrieval mechanism is adopted to directly read skeletal motion data from memory, and a constant parameter padding strategy is set to avoid matrix order reduction caused by retrieval failure, thereby improving the acquisition rate and computational continuity of macroscopic kinematic parameters.

[0012] Preferably, the step of centrally writing the generated data sources into the delay queue submodule for queue caching includes: constructing the delay queue submodule at the underlying level using a data structure combining a lock-free circular buffer and a timestamp indexed min-heap; generating content presentation timestamp tags based on the audio sampling point position, action primitive frame sequence number, and unified rendering time step; and performing ordered enqueue buffering operations on the parallel-imported audio waveform block data, the heterogeneous micro-motion parameter vector, and the heterogeneous macro-motion parameter vector, using the unique block identifier and the content presentation timestamp tags as enqueue sorting benchmark weights.

[0013] Furthermore, a lock-free ring structure and heap sort logic are used to process asynchronous data generated by dual-track parallel computation. Alignment caching of multimodal data is implemented based on identifiers and timestamps to avoid data overwrite race conditions caused by concurrent read and write operations.

[0014] Preferably, the step of calculating the local time deformation rate based on the optimal alignment path includes: performing frame-by-frame calculation on the audio waveform block data using the root mean square energy algorithm to extract the short-time energy envelope of the audio, and filtering and extracting the acoustic peak time series through a set energy envelope threshold; A first-order difference operation is performed on the spatial Euler angle matrix contained in the heterogeneous macroscopic motion parameter vector to filter out the timestamp nodes whose displacement change rate exceeds the preset kinematic extreme value point and aggregate them to generate the displacement key frame time series. The distance matrix is ​​constructed by calling the dynamic time warping algorithm and the path with the minimum cumulative distance is found to obtain the optimal alignment path. The local time deformation rate is calculated by using the ratio of the time difference between adjacent action time nodes matched on the optimal alignment path to the time difference between adjacent acoustic time nodes.

[0015] Furthermore, for heterogeneous multimodal data, a time series control group is constructed by extracting waveform energy peaks and spatial displacement extremes respectively. The nonlinear warping path is calculated by combining dynamic time warping algorithm, providing a quantitative basis for deformation judgment for audio-visual timing coordination.

[0016] Preferably, the step of triggering the timing backpressure signaling includes: calculating the difference between the local time deformation rate value and the upper and lower limits of the preset kinematic boundary threshold to obtain the deviance amount, and generating the timing backpressure signaling containing the current deviance amount; When the local time deformation rate is greater than the upper limit of the kinematic boundary threshold, the input clock reference of the upstream unsynthesized text block sequence is actively delayed according to the deviation amount carried in the timing back pressure signaling, and a silence compensation tag of corresponding duration is inserted at the beginning of the subsequent text block sequence entering the queue. When the local time deformation rate is less than the lower limit of the kinematic boundary threshold, the current macroscopic motion parameter vector is inserted into the motion end hold frame or transition posture frame according to the time-series backpressure signaling response mechanism to extend the actual execution cycle of the macroscopic motion.

[0017] Furthermore, a bottom-level reverse pressure communication loop is constructed. When the time distortion rate exceeds the physical reasonable limit, the interpolation calculation is suspended, and the front-end audio generation gap or extended motion state is controlled in reverse to compensate for the timing difference from the source and solve the defect of limb movement stretching and deformation.

[0018] Preferably, the step of performing isomorphic space remapping operation and merging to generate the final fused action parameter tensor in isomorphic state space includes: calling the sparse mapping operator, filling zeros at the tensor positions corresponding to the limb bone nodes for the heterogeneous micro-action parameter vector, filling zeros at the tensor positions corresponding to the facial deformation primitives for the heterogeneous macro-action parameter vector, and mapping and outputting isomorphic micro-action parameter vectors and isomorphic macro-action parameter vectors with the same dimension respectively. Retrieve the persona joint priority weight tensor corresponding to the current digital human identity identifier from the memory database, and use the persona joint priority weight tensor to perform nonlinear interpolation and Hadamard product allocation on the isomorphic micro-motion parameter vector and the isomorphic macro-motion parameter vector, and merge to generate a preliminary fused motion parameter tensor.

[0019] Furthermore, the problem of feature dimension mismatch in multi-source data is solved by zero-value filling expansion in high-dimensional space, and a static weight matrix reflecting personalized body habits is introduced to participate in the product operation, thus completing the dimensional integration of multimodal data in state space.

[0020] Preferably, the step of combining the spatial topology collision truncation detection mechanism includes: calculating the three-dimensional spatial coordinates of the digital human whole-body skeleton nodes in real time based on the preliminary fused motion parameter tensor, and constructing a bounding sphere with the hand end effector as the center and a bounding box with the head node as the center. Calculate the shortest spatial Euclidean distance between the surface of the bounding sphere and the surface of the bounding box, and compare the shortest spatial Euclidean distance with a preset safety interference physical threshold. When the shortest spatial Euclidean distance is less than the safe interference physical threshold, a collision truncation mechanism is triggered. The rotation angle increment of the hand bone that causes the distance to shorten in the preliminary fusion action parameter tensor is reduced in reverse by a set step size. If the shortest spatial Euclidean distance is still less than the safe interference physical threshold after a specified number of reductions, all rotation increments that cause interference generated in the current time step are forcibly discarded and the hand bone pose is rolled back to the pose of the previous safe rendering frame.

[0021] Furthermore, spatial geometric interference calculations are performed before merging data-driven rendering. Inverse angle reduction and forced attitude rollback are used to block the accumulation of rotational increments that pose a risk of clipping, ensuring the physical spatial rationality of the 3D mesh presentation.

[0022] Preferably, the step of dynamically updating the mesh skeleton transformation matrix in the driving rendering pipeline includes: using the facial blending deformation scalar weights and skeleton rotation Euler angles carried in the final fusion action parameter tensor, calling the linear blending skinning algorithm to generate a global transformation matrix to update the vertex coordinates of the 3D mesh model; Using the audio clock as a reference, compare the video frame timestamp generated by the presentation timestamp encapsulation with the current audio playback timestamp fed back by the underlying audio playback component; When the video frame timestamp lags behind the current audio playback timestamp and the difference exceeds the theoretical rendering time threshold of one frame, the current video frame is forcibly discarded and the parameters of the next frame are extracted and rendered faster; when the video frame timestamp is ahead of the current audio playback timestamp, the rendering engine enters a spin-wait state until the current audio playback timestamp catches up with the video frame timestamp or reaches the preset maximum spin timeout threshold.

[0023] Furthermore, during the independent output of audio and video, a low-level hardware synchronization control logic based on the audio clock is established. By calculating the absolute difference of timestamps, frame loss compensation or spin waiting is triggered to eliminate the audio and video stream misalignment defects caused by rendering time fluctuations.

[0024] The specific innovative principles and effects of this invention are as follows: A dual-track concurrent architecture is constructed to separate the data stream and the intent stream. After the text parsing module divides and delimits the continuous data stream, two independent asynchronous computing threads are opened in the underlying operating system. This decouples the computationally intensive acoustic feature inference network from the addressing-intensive macroscopic kinematic hash lookup table operation. This structure reduces the probability of mutual blocking between multimodal data generation nodes and improves the system throughput and the first frame output response rate.

[0025] To address the issue of misalignment between audio and video clocks, this invention proposes a timing control mechanism based on dynamic time warping and reverse pressure scheduling. The system establishes characteristic time sequences based on the short-time sound wave energy envelope and the spatial displacement gradient of the skeleton, respectively, and measures the local time deformation rate under the real physical dimension. When the deformation rate exceeds the set kinematic boundary, the system transmits correction signals in reverse along the constructed reverse pressure communication loop. The generation period is adjusted from the source of the time axis by delaying the text queue input or inserting transitional posture data. This mechanism avoids the pulling distortion caused by simply relying on time domain interpolation, which violates human inertia.

[0026] In the multimodal feature fusion stage, sparse remapping and topological interference blocking mechanisms are introduced. The system applies dimensional expansion and zero-value matrix filling to reconstruct the one-dimensional facial scalar weights and multi-dimensional skeleton rotation features that are in an independent and heterogeneous state into a unified mathematical isomorphic space. The corresponding digital human personality feature is configured with human joint priority weight tensors to perform feature superposition. Before driving rendering, the shortest distance detection is established based on the bounding box and bounding sphere of the spatial analytical domain. For outputs that exceed the interference safety limit, the step size reduction layer by layer and the state forced rollback algorithm are applied to directly correct the rotation increment that causes clipping, thereby improving the safety of spatial skeleton movement and the visual rendering quality.

[0027] This invention provides a method for the end-to-end generation of multimodal digital human content based on a persona agent. It has the following beneficial effects: 1. This invention calculates the local temporal deformation rate of the acoustic peak time series and the displacement keyframe time series using a dynamic time warping algorithm, and uses a preset kinematic boundary threshold for judgment. When the deformation rate deviates from the threshold range, the system triggers a timing backpressure signaling to insert a silence compensation tag into the audio stream or a hold frame into the action sequence. The mechanism limits the extent of human intervention in the action cycle through physical thresholds. While solving the timing misalignment problem caused by asynchronous processing of speech synthesis and macroscopic motion lookup tables, it avoids the sudden changes and distortions in rendering actions caused by directly stretching or compressing the time axis.

[0028] 2. This invention utilizes sparse mapping operators to process the facial micro-motion parameter vector and the body macro-motion parameter vector output by asynchronous computing threads. By expanding the dimension and performing zero-value matrix multiplication, these vectors are mapped to a homogeneous state space. Subsequently, combined with the extracted human-designed joint priority weight tensor, Hadamard product allocation is performed on the features of the homogeneous space. The method directly completes the merging of heterogeneous modal data and the allocation of personalized action force weights through low-level algebraic operations, avoiding the computational delay caused by building complex multimodal fusion neural network layers and reducing the computational resource overhead of the system.

[0029] 3. Before performing skeletal skinning calculation, this invention calculates the three-dimensional spatial coordinates based on motion parameters and constructs the hand bounding sphere and the head bounding box to calculate the shortest spatial Euclidean distance. When the distance is less than the safe interference physical threshold, the system triggers a collision truncation mechanism to reversely reduce the incremental bone rotation angle that causes interference. When continuous reduction is ineffective, it forces a rollback to the previous safe posture frame. This collision constraint mechanism based on spatial topology calculation prevents the limb clipping phenomenon caused by the superposition of micro-motions and macro-motions driving the three-dimensional mesh model, ensuring the continuity and physical rationality of streaming video rendering. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 This is a schematic diagram of the streaming text parsing and micro-intent command stream generation process of the present invention; Figure 4 This is a schematic diagram of the dual-track parallel generation and delay queue buffering process of the present invention; Figure 5 This is a schematic diagram of the local time deformation calculation and timing back pressure control process of the present invention; Figure 6 This is a schematic diagram of the heterogeneous motion parameter fusion and spatial collision truncation processing flow of the present invention; Figure 7 This is a schematic diagram of the audio and video timestamp synchronization comparison and rendering output process of the present invention; Figure 8 This is a schematic diagram comparing the end-to-end response delay and audio-visual alignment error of the present invention. Figure 9 This is a schematic diagram comparing the frequency of clipping through the 3D mesh with the frame drop rate in the system rendering test of this invention.

[0031] Among them, 110 is the text stream parsing module; 120 is the dual-track concurrent generation module; 121 is the acoustic feature submodule; 122 is the kinematic pre-reading submodule; 123 is the delay queue submodule; 130 is the temporal alignment control module; 140 is the spatial topology fusion module; and 150 is the synchronous rendering output module. Detailed Implementation

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] See attached document Figure 1 The present invention provides a multimodal digital human content full-link generation system based on human agent, the system including: a text stream parsing module 110, a dual-track concurrent generation module 120, a temporal alignment control module 130, a spatial topology fusion module 140, and a synchronous rendering output module 150.

[0034] It also includes a multimodal digital human concurrent generation subsystem, which is further configured with a persona Agent instance. The persona Agent instance is bound to the digital human identity identifier and is used to load the persona configuration data of the corresponding digital human. The persona configuration data includes at least voice timbre parameters, motion mapping dictionary, motion primitive hash table, persona joint priority weight tensor, default standby skeleton pose constant matrix, and rendering resource index information.

[0035] The text stream parsing module 110, the kinematics pre-reading sub-module 122, the spatial topology fusion module 140, and the synchronous rendering output module 150 call the persona configuration data loaded by the persona Agent instance to complete text intent parsing, action primitive retrieval, action fusion weight allocation, and digital human resource loading, respectively.

[0036] The text stream parsing module 110 connects to an external large language model, receives streaming text data and digital human identity identifiers, performs discretization segmentation, and outputs a text block sequence with independent boundaries; at the same time, it controls the text block sequence to enter the sliding text window component, calls the rule engine component to extract semantic keyframe information in the local context window, and outputs the micro-intent command stream and the transparent digital human identity identifier.

[0037] The dual-track concurrent generation module 120 includes an acoustic feature submodule 121, a kinematic pre-reading submodule 122, and a delay queue submodule 123. The acoustic feature submodule 121 acquires a text block sequence, calls a text-to-speech engine to convert it into audio waveform block data, and inputs it into an acoustically driven inference network to extract facial action features and generate a heterogeneous micro-action parameter vector. The kinematic pre-reading submodule 122 retrieves the action primitive hash table based on the micro-intention instruction stream and digital human identity identifier to generate a heterogeneous macro-action parameter vector. The delay queue submodule 123 receives and caches the above-mentioned audio waveform block data, heterogeneous micro-action parameter vector, and heterogeneous macro-action parameter vector.

[0038] The timing alignment control module 130 extracts the energy envelope peak time series and the displacement key frame time series of the heterogeneous macroscopic motion parameter vector from the audio waveform block data in the delay queue submodule 123, calculates the optimal alignment path and outputs the local time deformation rate value; when it is determined that the local time deformation rate value exceeds the preset threshold range, the timing alignment control module 130 sends a timing backpressure signal to the acoustic feature submodule 121 along the internally constructed backpressure communication loop, triggering it to insert a silence compensation tag at the beginning of the subsequent text block sequence.

[0039] The spatial topology fusion module 140 calls the sparse mapping operator to perform dimension completion calculations on the heterogeneous micro-motion parameter vectors and heterogeneous macro-motion parameter vectors, and outputs two sets of motion parameter vectors in the isomorphic state space. Then, it uses the retrieved corresponding character joint priority weight tensor to perform nonlinear interpolation and Hadamard product operations, and performs spatial topology collision truncation detection to output the corrected fused motion parameter tensor.

[0040] The synchronous rendering output module 150 performs clock reference rendering timestamp encapsulation on the audio waveform block data and fusion action parameter tensor, generates multi-channel composite data frames and pushes them into the annular memory area of ​​the headless graphics rendering pipeline. The headless graphics rendering pipeline calls this data source to update the 3D mesh skeleton transformation matrix and encodes and generates a continuous multimedia audio and video stream.

[0041] See attached document Figure 2 This invention provides a method for generating multimodal digital human content across the entire chain based on a persona agent. The method includes the following steps: S100, the text stream parsing module 110 obtains the streaming text data output by the external large language model and the current digital human identity identifier. It uses the internally configured sliding text window component and rule engine component to perform discretization truncation operation and semantic feature extraction on the streaming text data, and outputs a text block sequence with independent boundaries, as well as a micro-intention instruction stream containing action classification labels, sentiment polarity scalar, action amplitude parameters, priority parameters and expected action cycle, and the transparent digital human identity identifier. S200, the dual-track parallel generation module 120 receives text block sequences, micro-intent command streams and digital human identity identifiers, controls the acoustic feature submodule 121 and kinematic pre-reading submodule 122 to perform acoustic waveform synthesis inference and macroscopic kinematic parameter lookup operations in parallel on mutually independent data tracks, generates audio waveform block data, heterogeneous micro-motion parameter vectors and heterogeneous macroscopic motion parameter vectors, and writes the generated data sources into the delay queue submodule 123 for queue caching; S300, the timing alignment control module 130 reads the data source cached in the delay queue submodule 123, calculates the local time deformation rate of the acoustic peak time series and the displacement key frame time series, and suspends the timing alignment operation and triggers timing back pressure signaling when the local time deformation rate value deviates from the preset kinematic boundary threshold; when the local time deformation rate is greater than the upper limit of the kinematic boundary threshold, the timing alignment control module 130 controls the acoustic feature submodule 121 to actively delay the input clock reference of the upstream unsynthesized text sequence through timing back pressure signaling, and inserts a silence compensation label at the beginning of the subsequent text block sequence; When the local time deformation rate is less than the lower limit of the kinematic boundary threshold, the timing alignment control module 130 controls the kinematic pre-reading submodule 122 to extend the action end hold frame, insert the transition posture frame, or call the long-cycle action primitive to extend the actual execution cycle of the macroscopic action, thereby ensuring the rationality of the physical deformation generated by the action. S400, the spatial topology fusion module 140 calls the sparse mapping operator to perform isomorphic space remapping operation on the dual-track parameter vector that has been aligned by the time axis, and performs nonlinear interpolation and Hadamard product allocation using the retrieved character joint priority weight tensor. Combined with the spatial topology collision truncation detection mechanism, it merges to generate the final fusion action parameter tensor in the isomorphic state space. S500, the synchronous rendering output module 150 performs a presentation timestamp synchronization encapsulation operation based on a common clock reference on the final fused motion parameter tensor and the processed audio waveform block data. The encapsulated multi-channel composite data frame is continuously written into the ring memory buffer of the headless graphics rendering pipeline, driving the rendering pipeline to dynamically update the mesh skeleton transformation matrix and encode and output multimedia audio and video streams in real time.

[0042] See attached document Figure 3In specific implementations, step S100 provided by the present invention may include the following sub-steps: S110, the physical data interface of the text stream parsing module 110 connects to the external large language model through the network transmission protocol. The text stream parsing module 110 continuously receives the streaming text data sent by the external large language model inference, and loads and binds the identity identifier corresponding to the current digital human in the system's memory at the initial stage of the process. The text stream parsing module 110 uses a dynamic delimiting algorithm to perform discretization segmentation on the streaming text data. It uses the punctuation code set configured by the system and natural language sentence break markers to block and intercept the continuously written character stream, converting the streaming text data without physical boundaries into a sequence of text blocks containing clear start and end bytes. Each text block is assigned a globally incrementing text block number and a unique block identifier. The text block number is used to maintain the semantic generation order of the streaming text, and the unique block identifier is used to perform cross-module data pairing in subsequent acoustic waveform synthesis, micro-motion generation, macro-motion lookup, temporal alignment, spatial fusion, and synchronous rendering processes.

[0043] S120, the text stream parsing module 110 writes the text block sequence into the memory queue and controls the text block sequence to enter the sliding text window component in sequence. The sliding text window component allocates a sliding reading interval with a preset fixed character length in the memory space. The preset fixed character length is set to a value range of 50 to 200 characters. The sliding text window component controls the text block sequence to move into the sliding reading interval step by step according to the order of time generation. A local context window for semantic feature extraction is established in the memory area. The memory pointer step displacement and queue data expiration release rules of the sliding text window component can be based on the existing conventional cache queue scheduling algorithm.

[0044] S130, the text stream parsing module 110 predicts the needs of the digital human's physical performance in advance by identifying the sentiment and action-oriented words in the text block sequence, providing basic guidance for subsequent action generation. In specific implementation, the text stream parsing module 110 calls the rule engine component to perform semantic keyframe information extraction operation on the current independent text block residing in the local context window. The rule engine component scans the character entity features contained in the current independent text block and inputs the character entity features into the action mapping dictionary pre-configured by the rule engine component for query and comparison. Each dictionary record in the action mapping dictionary includes a keyword field, an action classification label field, a sentiment polarity field, an action amplitude field, an expected action cycle field, and a priority field. When the current independent text block matches multiple dictionary records, the rule engine component determines the target action category label according to the value of the priority field, and selects the dictionary record with the longest keyword matching length as the output result when the priorities are the same. Extract the action identifier and polarity parameter that have a mapping relationship with the character entity features, and simultaneously extract the corresponding action amplitude parameter and expected action cycle; When a character entity feature does not match any entry in the action mapping dictionary, the rule engine component outputs a pre-configured silent action identifier, neutral sentiment polarity parameter, default action amplitude parameter, and default expected action period as default values. The default action amplitude parameter is set to 1.0, and the default expected action period is set to 2.0s to ensure continuous execution of subsequent processes and maintain process coherence. The extracted parameter set is used to encapsulate and generate a micro-intent command stream. The generation mechanism of the micro-intent command stream is calculated based on the following extraction mapping function expression: ; In the formula, The first output of the feature extraction model The semantic encoding vectors corresponding to each text sub-block; Mapping functions for neural network models that perform semantic feature extraction; For the input of the first Each segment of text is a sub-block; This is the weight parameter matrix of the feature extraction model; This is the index number of the text block sequence.

[0045] S140, the micro-intent command stream collection data body contains action category labels, emotion polarity scalars, action amplitude parameters, priority parameters, unique identifiers for text blocks, and expected action cycles in the underlying data structure. The action category label is a category index key value that instructs the downstream device system to retrieve and load a specific skeletal animation resource file. The emotion polarity scalar is a floating-point numerical parameter that quantifies the strength of the facial emotion corresponding to the current independent text block. The action amplitude parameter is used to adjust the skeletal rotation amplitude and facial deformation amplitude of the target action primitive. The priority parameter is used to determine the action output order when there are multiple candidate actions in the same local context window. The unique identifier for the text block is used to bind the micro-intent command stream with the corresponding text block, audio waveform block data, and action parameter vector. The expected action cycle is a scalar of the physical time span occupied by a single action category from the start frame to the end frame. The text stream parsing module 110 drives the hardware interface to perform the underlying data splitting and output operation. The first output end of the text stream parsing module 110 sends a sequence of text blocks composed of independent text blocks to the downstream hardware link. The second output end of the text stream parsing module 110 simultaneously sends the micro-intent instruction stream set data body and the transparent digital human identity identifier to the downstream hardware link.

[0046] See attached document Figure 4 In specific implementations, step S200 provided by the present invention may include the following sub-steps: S210, the dual-track concurrent generation module 120 receives the text block sequence, micro-intention instruction stream, and transparent digital human identity identifier sent by the text stream parsing module 110 through the inter-process communication interface. The dual-track concurrent generation module 120 opens two independent and non-blocking asynchronous computing threads at the operating system level. The dual-track concurrent generation module 120 allocates the text block sequence to the acoustic feature submodule 121 in the first computing thread. The dual-track concurrent generation module 120 synchronously allocates the micro-intention instruction stream and the pre-loaded digital human identity identifier to the kinematic pre-reading submodule 122 in the second computing thread. The multi-threading mechanism is used to optimize the concurrent computing efficiency of data processing.

[0047] S220, in order to convert text into audible audio and extract feature parameters that control facial muscle changes from the audio signal, the acoustic feature submodule 121 calls the internally configured text-to-speech engine to perform front-end text normalization and acoustic feature prediction on the text block sequence, and the acoustic feature submodule 121 synthesizes and outputs audio waveform block data. For the vocoder waveform reconstruction stage of the text-to-speech engine, in this embodiment, based on the conventional speech synthesis algorithm, the acoustic feature submodule 121 further extracts the Mel spectrum feature matrix of the audio waveform block data. The extraction process resamples the audio waveform block data at a sampling rate of 16kHz and applies the short-time Fourier transform algorithm to generate a two-dimensional feature matrix containing 80-dimensional Mel frequency bands. The acoustic feature submodule 121 inputs the Mel spectrum feature matrix into the acoustically driven inference network to perform forward propagation operation. Meanwhile, the acoustic feature submodule 121 reads the emotional polarity scalar bound to the unique identifier of the current text block and inputs the emotional polarity scalar as an auxiliary control parameter into the parameter output layer of the acoustically driven inference network, or performs gain adjustment on the facial hybrid deformation weights corresponding to eyebrows, eyelids, corners of mouth and cheeks after the parameter output layer, so that the facial micro-motion parameters can reflect the emotional intensity of the current text block. In specific implementation, the acoustic-driven inference network adopts a one-dimensional convolutional hybrid architecture based on a long short-term memory network. The acoustic-driven inference network includes a data input layer, three one-dimensional convolutional feature extraction layers, two long short-term memory network temporal layers, and a parameter output layer composed of fully connected layers. The data input layer receives a two-dimensional feature matrix containing eighty-dimensional Mel frequency bands. The parameter output layer outputs a heterogeneous micro-motion parameter vector with constant dimension. In the underlying physical state, the heterogeneous micro-motion parameter vector represents fifty-one continuous scalar values ​​used to control the weights of the digital human face hybrid deformation. The 51 continuous scalar values ​​are forcibly normalized by the algorithm to a floating-point range of 0 to 1 to map the activation state of each facial muscle from relaxation to extreme contraction. The generation logic of the heterogeneous micro-motion parameter vector satisfies the following relational definition: ; In the formula, In order to time step The generated heterogeneous micro-action parameter vector; Mapping function for the neural network model of voice-driven facial generation; The input is a sequence of acoustic features; This represents the time step in the continuous streaming generation process.

[0048] To enable the acoustically driven inference network to perform forward inference mapping computation, the acoustically driven inference network needs to be trained before system deployment. The training steps include collecting a synchronized audio and video dataset of real people speaking, using a visual motion capture algorithm to extract the mixed deformation weight sequence of real faces as ground truth labels, extracting Mel spectrum from the corresponding audio waveform segments as training samples, inputting the training samples into the acoustically driven inference network in the parameter initialization state to obtain predicted action vectors, and calculating the mean squared error loss function between the predicted action vectors and the ground truth labels. The internal node connection weights of the acoustic-driven inference network are updated using the backpropagation algorithm combined with the adaptive moment estimation algorithm until the output value of the mean square error loss function converges to a pre-calibrated threshold range. The pre-calibrated threshold range is set to 0.001 to 0.01, thus completing the model construction of the acoustic-driven inference network.

[0049] S230, in the second computing thread, the kinematics pre-reading submodule 122 directly loads pre-made limb movement data from the system memory by reading pre-extracted semantic tags. The kinematics pre-reading submodule 122 reads the action classification tags contained in the micro-intention instruction stream. The kinematics pre-reading submodule 122 performs a structured concatenation operation with delimiters on the action classification tags and the digital human identity identifier, or combines the action classification tags, the digital human identity identifier, and the emotional polarity interval identifier into a key-value tuple to generate a globally unique hash key-value. The kinematics pre-reading submodule 122 uses the hash key value generated by the kinematics pre-reading submodule 122 to perform address matching in the preloaded motion primitive hash table in memory, and obtains the bone node rotation quaternion sequence bound to the hash key value generated by the kinematics pre-reading submodule 122. The data items bound to the hash key value in the motion primitive hash table include a list of bone node numbers, a sequence of bone node rotation quaternions, the original motion cycle, the motion amplitude coefficient, and the default transition frame sequence. The kinematics pre-reading submodule 122 performs time scale pre-adjustment on the skeletal node rotation quaternion sequence based on the expected motion cycle in the micro-intention instruction stream, and performs amplitude scaling on the skeletal node rotation amplitude based on the motion amplitude parameter. The kinematics pre-reading submodule 122 converts the skeletal node rotation quaternion sequence into Euler angle matrix format and outputs a heterogeneous macroscopic motion parameter vector. When the hash key value generated by the kinematics prefetching submodule 122 fails to find a valid storage address in the action primitive hash table, the kinematics prefetching submodule 122 intercepts the abnormal retrieval state of not finding a valid storage address. The kinematics prefetching submodule 122 calls the default standby skeleton pose constant matrix preset in the system configuration file to replace the missing spatial action data. It uses a constant parameter padding strategy to avoid algorithmic logic dead zones caused by matrix dimension reduction in downstream spatial transformation calculations. The retrieval and loading process of the kinematics prefetching submodule 122 satisfies the following relational expression: ; In the formula, In order to time step The retrieved heterogeneous macroscopic action parameter vector; This is a locality-sensitive hash retrieval function; The first output of the feature extraction model The semantic encoding vectors corresponding to each text sub-block; This is the currently defined digital human identity identifier; This refers to the time step in the continuous streaming generation process; This is the index number of the text block sequence.

[0050] S240, the dual-track concurrent generation module 120 faces the data convergence problem where the processing time of the output of the two asynchronous computing nodes differs. The delay queue submodule 123 adopts a data structure combining a lock-free circular buffer and a timestamp index min-heap at its bottom layer. The lock-free circular buffer is used to store audio waveform block data, heterogeneous micro-motion parameter vectors, and heterogeneous macro-motion parameter vectors. The timestamp index min-heap is used to maintain the reading order of each data block. After generating the data body corresponding to the output, the acoustic feature submodule 121 and the kinematic pre-reading submodule 122 respectively inherit the unique block identifier corresponding to the current text block, and generate content presentation timestamp labels according to the audio sampling point position, action primitive frame sequence number, and unified rendering time step. The delayed queue submodule 123 uses a unique block identifier and a content presentation timestamp label as the enqueue sorting basis weight. The delayed queue submodule 123 performs ordered enqueue buffering operations on the parallel-input audio waveform block data, heterogeneous micro-motion parameter vectors, and heterogeneous macro-motion parameter vectors to avoid data mismatch in different text blocks caused by the difference in generation time of different computing threads, and avoids the data overwrite race condition caused by the forced writing of parallel data streams to the same memory address.

[0051] See attached document Figure 5 In specific implementations, step S300 provided by the present invention may include the following sub-steps: S310, the timing alignment control module 130 reads the audio waveform block data and heterogeneous macroscopic motion parameter vector cached in the delay queue submodule 123, and reads the heterogeneous microscopic motion parameter vector with the same unique block identifier as the audio waveform block data. The heterogeneous microscopic motion parameter vector is generated by the audio waveform block data. The timestamp of the heterogeneous microscopic motion parameter vector inherits the content presentation timestamp of the audio waveform block data. Therefore, the heterogeneous microscopic motion parameter vector and the audio waveform block data are on the same time base. The timing alignment control module 130 mainly performs time axis resampling processing on the heterogeneous macroscopic motion parameter vector, and sends the resampled heterogeneous macroscopic motion parameter vector and the heterogeneous microscopic motion parameter vector with audio timestamp to the spatial topology fusion module 140.

[0052] The timing alignment control module 130 uses the root mean square energy algorithm to perform frame-by-frame calculation on the audio waveform block data to extract the short-time energy envelope of the audio. The calculation window length of the energy envelope is set to 20 to 30 ms to ensure short-time stability. The timing alignment control module 130 filters and extracts the acoustic peak time series of the audio waveform block data through the set energy envelope threshold. The energy envelope threshold is determined by multiplying the global average energy value of the audio waveform block data by an empirical coefficient of 0.75. Simultaneously, the timing alignment control module 130 performs a first-order difference operation on the spatial Euler angle matrix contained in the heterogeneous macroscopic motion parameter vector to calculate the displacement change rate of the spatial coordinates corresponding to the heterogeneous macroscopic motion parameter vector. The timing alignment control module 130 filters out the timestamp nodes whose displacement change rate exceeds the preset kinematic extreme point. The preset kinematic extreme point is calibrated based on the maximum reasonable displacement tolerance between frames of the standard digital human skeleton animation. The value range of the preset kinematic extreme point is set to 0.05 to 0.15 m / s. The timing alignment control module 130 aggregates the timestamp nodes whose displacement change rate exceeds the preset kinematic extreme point in ascending temporal order to generate a displacement key frame time series.

[0053] S320, the timing alignment control module 130 calls the internally configured dynamic time warping algorithm to calculate the optimal alignment path between the acoustic peak time series and the displacement keyframe time series. The dynamic time warping algorithm achieves nonlinear time warping by constructing a distance matrix between the acoustic peak time series and the displacement keyframe time series and finding the path with the minimum cumulative distance from the starting point to the ending point. The timing alignment control module 130 calculates the local time deformation rate based on the acoustic time nodes and action time nodes matched on the optimal alignment path. The local time deformation rate is used to quantify the degree of physical stretching or compression that must occur for the current limb action duration to adapt to the speech duration. The calculation logic of the local time deformation rate follows the following relationship setting: ; In the formula, The calculated local time deformation rate scalar value; For the displacement keyframe time series on the optimal alignment path, the first The timestamps corresponding to each matching node; For the displacement keyframe time series on the optimal alignment path, the first The timestamps corresponding to each matching node; For the acoustic peak time series on the optimal alignment path, the first... The timestamps corresponding to each matching node; For the acoustic peak time series on the optimal alignment path, the first... The timestamps corresponding to each matching node; This is the index number of the matching node on the optimal alignment path.

[0054] S330, the timing alignment control module 130 pre-calibrates the kinematic boundary threshold. The kinematic boundary threshold is set to a value range of 0.8 to 1.2. The kinematic boundary threshold is used to limit the maximum time scaling ratio of the digital human skeleton animation in real-time rendering mode without significantly violating the physical inertia of the human body. When the timing alignment control module 130 determines that the local time deformation rate value falls within the preset kinematic boundary threshold range, the timing alignment control module 130 performs a cubic spline interpolation resampling operation on the heterogeneous macroscopic motion parameter vector according to the local time deformation rate value, so that the start and end time axes of the heterogeneous macroscopic motion parameter vector are aligned with the time axis of the audio waveform block data. The basis function selection rule for the cubic spline interpolation resampling operation can be based on the existing conventional numerical interpolation algorithm. The timing alignment control module 130 sends the heterogeneous macroscopic motion parameter vector that has completed the timing alignment processing to the downstream spatial topology fusion module 140.

[0055] S340, in order to prevent the algorithm logic from getting stuck in the execution dead zone and to avoid the digital human from exhibiting severe pulling and distortion movements, when the timing alignment control module 130 determines that the local time deformation rate value deviates from the preset kinematic boundary threshold range, the timing alignment control module 130 suspends the timing alignment interpolation operation for the current heterogeneous macroscopic motion parameter vector, and writes the previous safe motion frame, the default standby motion frame, or the current motion end hold frame as a temporary output frame into the downstream buffer, so as to ensure that the synchronous rendering output module 150 still has a continuously available motion data source during the suspension period; The timing alignment control module 130 generates a timing backpressure signaling containing the current deviance amount. The deviance amount is calculated by the difference between the local time deformation rate value and the upper and lower limits of the kinematic boundary threshold. The timing alignment control module 130 sends the timing backpressure signaling in reverse to the acoustic feature submodule 121 or the kinematic pre-reading submodule 122 along the backpressure communication loop constructed at the underlying physical layer. When the local time deformation rate is greater than the upper limit of the kinematic boundary threshold, the acoustic feature submodule 121 responds to the timing back pressure signaling, the acoustic feature submodule 121 actively delays the input clock reference of the upstream unsynthesized text block sequence, and the acoustic feature submodule 121 inserts a silence compensation tag of corresponding duration at the beginning of the subsequent text block sequence entering the queue according to the deviation amount carried in the timing back pressure signaling. When the local time deformation rate is less than the lower limit of the kinematic boundary threshold, the kinematic pre-reading submodule 122 responds to the timing back pressure signaling and inserts the current macroscopic motion parameter vector into the motion end hold frame or transition posture frame. Once the compensation duration or number of compensation frames reaches the compensation requirement corresponding to the deviation amount, the timing alignment control module 130 resumes the timing alignment interpolation operation of the current heterogeneous macroscopic motion parameter vector. The corresponding duration is calculated based on the deviation amount multiplied by the original duration of the current motion. The physical silence gap is used to reserve sufficient execution time for macroscopic limb movements, thereby ensuring the rationality of the audio-visual physical deformation of the multimodal digital human concurrent generation subsystem in continuous streaming rendering mode.

[0056] See attached document Figure 6 In specific implementations, step S400 provided by the present invention may include the following sub-steps: S410, the spatial topology fusion module 140 configures the global state space memory and sparse mapping operators based on a high-performance key-value pair in-memory database such as Redis; the spatial topology fusion module 140 receives heterogeneous micro-motion parameter vectors and heterogeneous macro-motion parameter vectors that have completed time-series alignment processing. The heterogeneous micro-motion parameter vectors contain one-dimensional facial hybrid deformation scalar weights, and the heterogeneous macro-motion parameter vectors contain multi-dimensional skeleton rotation Euler angles. The heterogeneous micro-motion parameter vectors and heterogeneous macro-motion parameter vectors are in a completely incompatible heterogeneous state in terms of data structure and feature dimensions. To unify the features of different modalities into the same mathematical space for algebraic operations, the spatial topology fusion module 140 calls the sparse mapping operator to perform isomorphic space remapping operation on the heterogeneous micro-motion parameter vector and the heterogeneous macro-motion parameter vector. It uses dimension expansion filling and zero-value matrix multiplication to map the heterogeneous micro-motion parameter vector and the heterogeneous macro-motion parameter vector to an isomorphic state space with a unified feature dimension, and outputs isomorphic micro-motion parameter vector and isomorphic macro-motion parameter vector respectively. The filling dimension in the sparse mapping operator mapping process is determined based on the sum of the values ​​of the digital human's whole body driveable skeleton nodes and facial deformation primitives. During execution, the sparse mapping operator fills all the tensor positions corresponding to the limb bone nodes with 0 for the heterogeneous micro-motion parameter vector, and fills all the tensor positions corresponding to the facial deformation primitive for the heterogeneous macro-motion parameter vector, thereby ensuring that the dimensional length of the isomorphic micro-motion parameter vector and the isomorphic macro-motion parameter vector remains consistent.

[0057] S420, the spatial topology fusion module 140 retrieves the corresponding human character joint priority weight tensor in the memory database based on the current digital human identity identifier; the human character joint priority weight tensor is a static configuration matrix containing floating-point values ​​from 0 to 1, and the dimension length of the human character joint priority weight tensor is consistent with the dimension length of the isomorphic state space. Each dimension of the human character joint priority weight tensor corresponds to a facial deformation primitive or a bone rotation degree of freedom. When the digital human is an expressive persona, the weight corresponding to the facial deformation primitive is higher than the weight corresponding to the degrees of freedom of the torso and limbs. When the digital human is a limb-expressive persona, the weight corresponding to the degrees of freedom of the shoulder, arm, hand, and torso is higher than the weight corresponding to the facial deformation primitive. When the memory database does not retrieve the persona joint priority weight tensor corresponding to the current digital human identity identifier, the spatial topology fusion module 140 loads the default weight tensor. The weight of each channel in the default weight tensor takes a preset neutral value, which is set to 0.5. The persona joint priority weight tensor is used to represent the personalized limb exertion habits and facial expression activity ratio of a specific digital human identity when performing actions. The spatial topology fusion module 140 uses the character design joint priority weight tensor to perform nonlinear interpolation and Hadamard product distribution on the isomorphic micro-motion parameter vector and the isomorphic macro-motion parameter vector. The spatial topology fusion module 140 generates a preliminary fused motion parameter tensor through the merging calculation of the underlying mathematical operators. The merging calculation logic of the preliminary fused motion parameter tensor satisfies the following relationship: ; In the formula, In order to time step The calculated preliminary fusion action parameter tensor; This is a priority weight tensor for the user profile joints retrieved based on the current digital human identity identifier. This is the Hadamard product operator; The nonlinear interpolation smoothing coefficient is set; In order to time step The isomorphic micro-action parameter vector obtained by mapping; In order to time step The isomorphic macroscopic action parameter vector obtained by mapping; This represents the time step in the continuous streaming generation process.

[0058] S430, because the preliminary fusion motion parameter tensor directly drives the 3D mesh before physical space verification, causing clipping interference between the limbs and torso, the spatial topology fusion module 140 performs spatial topology collision truncation detection on the preliminary fusion motion parameter tensor; the spatial topology fusion module 140 calculates the 3D spatial coordinate position of the digital human's whole-body skeleton nodes in real time based on the preliminary fusion motion parameter tensor; the spatial topology fusion module 140 constructs a bounding sphere with the hand end effector as the center and a bounding box with the head node as the center. The geometric radius of the bounding sphere is obtained by dynamic envelope calculation based on the maximum spatial span of the digital human hand 3D model, and the length, width and height dimensions of the bounding box are extracted and generated based on the extreme boundary of the three orthogonal axes of the head 3D mesh in the world coordinate system. The spatial topology fusion module 140 calculates the shortest spatial Euclidean distance between the surface of the bounding sphere and the surface of the bounding box. The spatial topology fusion module 140 compares the shortest spatial Euclidean distance with a preset safety interference physical threshold. The safety interference physical threshold is set based on the normal of the digital human model mesh offset outward by 3 to 5 cm to reserve a reasonable safety buffer thickness.

[0059] S440, when the spatial topology fusion module 140 determines that the shortest spatial Euclidean distance is greater than or equal to the safe interference physical threshold, the spatial topology fusion module 140 determines that there is no risk of overlapping interference in the current action state, and the spatial topology fusion module 140 directly assigns the initial fusion action parameter tensor and outputs it as the final fusion action parameter tensor. When the spatial topology fusion module 140 determines that the shortest spatial Euclidean distance is less than the safe interference physical threshold, the spatial topology fusion module 140 triggers a collision truncation mechanism to prevent the algorithm from entering the dead zone of clipping rendering. The spatial topology fusion module 140 reversely reduces the incremental hand bone rotation angle that causes the distance to be shortened in the tensor of the initial fusion action parameters. The step size of each reverse reduction is set to 5% of the incremental hand bone rotation angle. After each reduction, the shortest spatial Euclidean distance is recalculated. If the shortest spatial Euclidean distance is still less than the safe interference physical threshold after 20 consecutive reductions. The spatial topology fusion module 140 forcibly discards all rotation increments that cause interference generated in the current time step and rolls the hand bone pose back to the pose of the previous safe rendering frame, thereby breaking the potential computational dead loop, until the shortest spatial Euclidean distance is restored and maintained above the safe interference physical threshold. The spatial topology fusion module 140 outputs the final fusion action parameter tensor with no overlapping interference after spatial topology collision truncation detection and repair. The spatial topology fusion module 140 writes the final fusion action parameter tensor into the memory bus and sends it to the downstream synchronous rendering output module 150 for processing.

[0060] See attached document Figure 7 In specific implementations, step S500 provided by the present invention may include the following sub-steps: S510, the synchronous rendering output module 150 receives the final fusion action parameter tensor output by the spatial topology fusion module 140 through the high-bandwidth memory bus. At the same time, the synchronous rendering output module 150 reads audio waveform block data with the same time step as the final fusion action parameter tensor from the audio shared memory configured by the delay queue submodule 123. The synchronous rendering output module 150 initializes the internal three-dimensional graphics rendering engine and the underlying audio playback component. The three-dimensional graphics rendering engine loads the three-dimensional mesh model and material map resources bound to the pre-set digital human identity identifier according to the pre-set digital human identity identifier. It uses the loaded three-dimensional mesh model and material map resources to establish a basic rendering pipeline for the visualization of multimodal data.

[0061] S520, the synchronous rendering output module 150 uses the final fused motion parameter tensor to drive the loaded 3D mesh model. The synchronous rendering output module 150 extracts the facial blending deformation scalar weights and skeleton rotation Euler angles carried in the final fused motion parameter tensor. For driving facial expressions, existing facial blending deformation algorithms can be used. For driving the body skeleton, the synchronous rendering output module 150 calls the linear blending skinning algorithm to recalculate the vertex spatial coordinates to generate dynamic body poses. The linear blending skinning algorithm achieves smooth joint bending deformation by weighted calculation of the influence of multiple bones on a single mesh vertex. The vertex coordinate recalculation logic of the linear blending skinning algorithm satisfies the following spatial transformation mapping relationship definition: ; In the formula, For the third 3D mesh model The updated position vector of each vertex in the world coordinate system; For the first The total number of skeletal joints that generate physical effects at each vertex; For the first The first skeletal joint point is the first The floating-point value of the skinning weight of each vertex; For the first Each skeletal joint is generated by a global transformation matrix from the skeleton rotation Euler angle transformation contained in the final fused motion parameter tensor; For the first The initial local coordinate vectors of each vertex in the bound pose; The index number of the skeletal joint; This refers to the vertex index number of the 3D mesh model.

[0062] In S530, to reduce the audio-visual misalignment caused by differences in computation time during multi-threaded parallel processing, the synchronous rendering output module 150 adopts a dynamic audio-video synchronization strategy based on the audio clock. The synchronous rendering output module 150 writes the audio waveform block data into the hardware buffer of the underlying audio playback component and obtains the current audio playback timestamp fed back by the underlying audio playback component. The synchronous rendering output module 150 compares the difference between the video frame timestamp generated by the current 3D mesh model update and the current audio playback timestamp. The video frame timestamp is obtained by converting the content presentation timestamp carried by the final fused action parameter tensor. The audio playback timestamp is obtained by converting the sampling point sequence number of the audio waveform block data and the audio sampling rate. Both the video frame timestamp and the audio playback timestamp are mapped to the same system monotonic clock reference. When the video frame timestamp lags behind the current audio playback timestamp and the difference between the video frame timestamp generated by the current 3D mesh model update and the current audio playback timestamp exceeds the theoretical rendering time threshold of one frame, the synchronous rendering output module 150 triggers the rendering frame dropping compensation mechanism. The theoretical rendering time threshold is set constantly based on the reciprocal of the target rendering frame rate. The synchronous rendering output module 150 forcibly discards the current video frame and accelerates the extraction of the final fusion action parameter tensor of the next frame for rendering. By discarding the visual coherence of local video frames, the temporal synchronization of speech production and lip movements is ensured. When the video frame timestamp is ahead of the current audio playback timestamp, the synchronous rendering output module 150 causes the 3D graphics rendering engine to enter a brief spin-wait state until the current audio playback timestamp catches up with the video frame timestamp. If the duration of the spin-wait state exceeds the preset maximum spin timeout threshold, the synchronous rendering output module 150 forcibly interrupts the spin-wait state and continues to execute the video frame rendering instructions. The maximum spin timeout threshold is set to 50ms to prevent algorithm rendering deadlock caused by underlying audio hardware abnormalities, thereby avoiding the risk of timing misalignment in the process of independent audio and video output.

[0063] After completing multimodal temporal alignment and updating the vertex coordinates of the 3D mesh model, the synchronous rendering output module 150 calls the underlying graphics application programming interface to perform vertex shading and fragment shading operations on the 3D mesh model, converting the 3D spatial data of the 3D mesh model after the spatial coordinate update into a 2D screen pixel array; for lighting calculation and shadow generation in the graphics pipeline, existing conventional real-time rendering technology can be used. The synchronous rendering output module 150 encodes the two-dimensional screen pixel array to generate video frames and encodes the audio waveform block data to generate digital audio frames. The synchronous rendering output module 150 performs multiplexing and encapsulation on the video frames and digital audio frames to generate a multimedia audio and video stream. In the server-side headless rendering mode, the multimedia audio and video stream is output to the client through the network transmission interface. In the terminal local presentation mode, the video frames are written to the display buffer and the audio waveform block data is written to the speaker playback buffer, thus completing all system control and rendering output tasks in a single streaming and cycle.

[0064] Application Examples: In this embodiment, a digital human for live-streaming sales deployed on a cloud server is selected as the physical operating carrier. The digital human identity identifier loaded in the system process is set as salesperson 01. The text stream parsing module 110 accesses the live-streaming sales script text data issued by an external large language model. The text stream parsing module 110 performs discretization segmentation on the live-streaming text data according to a fixed character length of 120, and outputs a text block sequence with independent boundaries. The text stream parsing module 110 calls the rule engine component to scan the local context window, hits the emphasis keywords set in the action mapping dictionary and extracts the corresponding micro-intent command stream. The micro-intent command stream includes an action classification label for pointing the right hand to the screen, an emotion polarity scalar with a value of 0.85, and an expected action cycle with a value of 1.5 seconds.

[0065] The dual-track concurrent generation module 120 receives the text block sequence and the micro-intent command stream. The acoustic feature submodule 121 synthesizes audio waveform block data with a duration of 1.82 seconds according to a sampling rate of 16kHz. The acoustic feature submodule 121 extracts the Mel spectrum and inputs it into the acoustically driven inference network, outputting a heterogeneous micro-motion parameter vector containing 51 continuous scalar values. The kinematics pre-reading submodule 122 generates hash key values ​​using action classification labels and digital human identity identifiers. The kinematics pre-reading submodule 122 addresses and matches the corresponding skeletal node rotation quaternion sequence in the action primitive hash table, outputting a heterogeneous macro-motion parameter vector. The above three sets of data are uniformly assigned a unique block identifier for the current text block and written to the delay queue submodule 123.

[0066] The timing alignment control module 130 reads the data in the delay queue submodule 123, extracts the acoustic peak time series of the audio waveform block data and the displacement key frame time series of the heterogeneous macroscopic motion parameter vector, calculates the local time deformation rate value, finds that the speech duration of 1.82 seconds is mismatched with the original motion period of 1.5 seconds, and the local time deformation rate value falls within the preset kinematic boundary threshold range. The timing alignment control module 130 performs a cubic spline interpolation resampling operation on the heterogeneous macroscopic motion parameter vector, so that the time axis of the heterogeneous macroscopic motion parameter vector is stretched to 1.82 seconds.

[0067] The spatial topology fusion module 140 calls the sparse mapping operator to perform isomorphic spatial remapping operation on the heterogeneous micro-motion parameter vector and the heterogeneous macro-motion parameter vector. The spatial topology fusion module 140 retrieves the persona joint priority weight tensor corresponding to the persona 01, assigns a weight value of 0.8 to the arm node according to the set persona joint priority weight tensor, performs nonlinear interpolation allocation, and generates a preliminary fused motion parameter tensor. The spatial topology fusion module 140 calculates the shortest spatial Euclidean distance between the digital persona's bounding sphere and bounding box in real time, determines that the shortest spatial Euclidean distance for the arm raising action is 8.2 cm, which is greater than the set safety interference physical threshold of 4.0 cm, and determines that there is no risk of overlapping interference in the current state, and outputs the final fused motion parameter tensor.

[0068] The synchronous rendering output module 150 uses the final fusion action parameter tensor to drive the loaded 3D mesh model, calls the linear blending skinning algorithm to update the vertex world coordinate vectors, compares the video frame timestamp with the current audio playback timestamp fed back by the underlying audio playback component, performs a forced alignment operation based on the audio clock, and calls the underlying graphics application programming interface to complete the fragment shading, generate multi-channel composite data frames and encode them into multimedia audio and video streams to be sent to the external live streaming push node.

[0069] Based on the physical operating platform of the above application embodiments, a comparative test experimental group was established. The traditional digital human generation algorithm with serial generation and no spatial interference detection mechanism was set as the control group, and the multimodal digital human concurrent generation subsystem and processing method provided by the present invention was set as the experimental group. A commercial test text data stream with a total duration of 72 hours was continuously injected into the two test systems using an automated script. The end-to-end processing latency, audio-visual time alignment error, mesh clipping interference frequency, and rendering pipeline frame drop ratio were synchronously recorded on the cloud rendering server node. The original performance matrix was constructed by extracting the system's underlying operation log and rendering status timestamp sequence. The original performance matrix was read and outlier filtering and statistical distribution fitting calculation were performed to generate an experimental feature dataset containing various quantitative evaluation indicators of the two systems.

[0070] Standardized parsing operations were performed on the extracted experimental feature dataset to calculate the arithmetic mean and cumulative failure frequency of each test evaluation dimension. The multidimensional discrete numerical values ​​were structured and imported into a two-dimensional data table to generate a performance comparison parameter set. The measured values ​​of the two systems under four evaluation dimensions were extracted to generate a quantitative comparison table of the experimental effects of the digital human content generation system, as shown in Table 1.

[0071] Table 1: Quantitative Comparison of Experimental Results of Digital Human Content Generation Systems

[0072] According to Table 1 and Figure 8 and Figure 9 Based on the data, the end-to-end response delay measured in the experimental group after introducing the dual-track and generation module 120 and the lockless ring buffer design was 284.16 ms, while the end-to-end response delay measured in the control group was 847.31 ms. The experimental group applied the timing alignment control module 130 to calculate the dynamic time warping path and combined it with the backpressure communication loop to perform timing feedback adjustment operations, resulting in a mean square error of audio-visual alignment of 18.39 ms, while the mean square error of audio-visual alignment in the control group was 136.85 ms. This demonstrates that the present invention eliminates the time dimension of speech and body movements. To investigate the phase misalignment phenomenon, the experimental group implemented a spatial topological collision truncation detection mechanism based on the shortest spatial Euclidean distance between the bounding sphere and the bounding box. The frequency of clipping through the 3D mesh was reduced to 0.15 times / hour, while the frequency of clipping through the 3D mesh in the unconstrained state of the control group was as high as 19.42 times / hour. This demonstrates that the joint computation mechanism of sparse mapping operator and collision truncation detection breaks the algorithm dead zone of traditional clipping rendering. The experimental group used the synchronous rendering output module 150 to establish a presentation timestamp synchronous encapsulation operation based on the audio clock, and the frame drop rate of the rendering pipeline was measured to be 0.

[0073] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating multimodal digital human content across the entire value chain based on a persona agent, characterized in that: The method includes: Acquire streaming text data and the current digital human identity identifier, perform discretization truncation and semantic feature extraction, and output a text block sequence and micro-intention instruction stream; On an independent data track, acoustic synthesis and kinematic lookup are performed in parallel according to the text block sequence and the micro-intent instruction stream to generate audio waveform block data, heterogeneous micro-motion parameter vectors and heterogeneous macro-motion parameter vectors, and write them into the delay queue submodule for queue caching. The acoustic peak time series and displacement key frame time series are extracted to calculate the local time deformation rate. When the local time deformation rate is determined to deviate from the preset kinematic boundary threshold, the timing back pressure signaling is triggered, and the timing alignment processing is performed on the heterogeneous macroscopic motion parameter vector. The heterogeneous micro-motion parameter vector and the heterogeneous macro-motion parameter vector after time alignment are subjected to isomorphic spatial remapping, merged using the character joint priority weight tensor, and spatial topological collision truncation detection is performed to generate the final fused motion parameter tensor. The final fused action parameter tensor and the audio waveform block data are subjected to a presentation timestamp synchronization encapsulation operation based on a common clock reference, which drives the rendering pipeline to update the mesh skeleton transformation matrix and output multimedia audio and video streams.

2. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The steps of outputting the text block sequence and micro-intent command stream include: The streaming text data is discretized and segmented into a sequence of text blocks containing explicit start and end bytes, and a unique block identifier is assigned. Control the text block sequence to enter the sliding text window component to establish a local context window; The character entity features contained in the current independent text block are input into the action mapping dictionary for query and comparison. Action classification labels, emotional polarity scalars, action amplitude parameters and expected action cycles with mapping relationships are extracted and encapsulated to generate the micro-intent command stream.

3. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The steps for generating audio waveform block data and heterogeneous micro-motion parameter vectors include: The text-to-speech engine is invoked to perform front-end text normalization and acoustic feature prediction on the text block sequence, and the audio waveform block data is synthesized. A two-dimensional feature matrix containing the Mel frequency band is generated by applying the short-time Fourier transform algorithm and input into the acoustically driven inference network to perform forward propagation operation; The emotional polarity scalar bound to the unique identifier of the current text block is read as an auxiliary control parameter to adjust the facial blending deformation weights and output the heterogeneous micro-motion parameter vector with constant dimension.

4. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The step of generating heterogeneous macroscopic action parameter vectors includes: The action category label in the micro-intent command stream is concatenated with the current digital human identity identifier to generate a hash key value; Address matching is performed within the preloaded action primitive hash table to obtain the sequence of rotation quaternions of the bound bone nodes. The time scale is pre-adjusted according to the expected action cycle and converted into Euler angle matrix format. The heterogeneous macroscopic action parameter vector is then output. When the hash key value does not match a valid storage address, the missing spatial motion data is replaced by a preset default standby skeleton pose constant matrix.

5. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The steps for caching the write delay queue include: The delay queue is constructed at the bottom layer using a data structure that combines a lock-free circular buffer and a timestamp indexed min-heap. Based on the audio sampling point location, action primitive frame sequence number, and rendering time step, generate content presentation timestamp tags; Using the unique block identifier and the content presentation timestamp label as the enqueue sorting basis weight, an ordered enqueue buffering operation is performed on the audio waveform block data, the heterogeneous micro-motion parameter vector, and the heterogeneous macro-motion parameter vector.

6. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The step of extracting the time series to calculate the local time deformation rate includes: The root mean square energy algorithm is used to extract the short-time energy envelope of the audio waveform block data, and the acoustic peak time series is generated by filtering the energy envelope threshold. Perform a first-order difference operation on the spatial Euler angle matrix of the heterogeneous macroscopic motion parameter vector, and filter the timestamp nodes where the rate of displacement change exceeds the kinematic extreme value to generate a displacement key frame time series. The dynamic time warping algorithm is invoked to find the optimal alignment path for the two sets of time series, and the local time deformation rate is calculated using the ratio of the adjacent time differences of the matching nodes on the optimal alignment path.

7. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The step of triggering the timing backpressure signaling includes: Calculate the difference between the local time deformation rate and the preset kinematic boundary threshold upper or lower limit, and generate the timing backpressure signaling carrying the deviance amount; When the local time deformation rate is greater than the upper limit of the kinematic boundary threshold, the input clock reference of the upstream unsynthesized text block sequence is delayed according to the timing backpressure signaling, and a mute compensation tag is inserted at the beginning of the text block sequence. When the local time deformation rate is less than the lower limit of the kinematic boundary threshold, the current macroscopic motion parameter vector is inserted into the motion end hold frame or transition posture frame according to the temporal back pressure signaling.

8. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The steps of performing isomorphic space remapping and merging include: The sparse mapping operator is invoked to fill zeros at the tensor positions corresponding to the limb bone nodes for the heterogeneous micro-motion parameter vector, and to fill zeros at the tensor positions corresponding to the facial deformation primitives for the heterogeneous macro-motion parameter vector, and to map out isomorphic micro-motion parameter vectors and isomorphic macro-motion parameter vectors. Retrieve the persona joint priority weight tensor corresponding to the current digital human identity identifier, perform nonlinear interpolation and Hadamard product allocation on the isomorphic micro-motion parameter vector and the isomorphic macro-motion parameter vector, and merge to generate a preliminary fused motion parameter tensor.

9. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 8, characterized in that, The steps for performing spatial topology collision truncation detection include: Based on the preliminary fusion motion parameter tensor, the three-dimensional spatial coordinates of the skeletal nodes are calculated, and a bounding sphere with the hand end effector as the center and a bounding box with the head node as the center are constructed. When the shortest spatial Euclidean distance between the enclosing sphere and the enclosing box is less than the safe interference physical threshold, a collision cutoff mechanism is triggered, and the incremental rotation angle of the hand bone that caused the distance to be shortened is reduced in reverse according to a set step size. If the distance is still less than the safe interference physical threshold after a specified number of consecutive reductions, the rotation increment that caused the interference will be forcibly discarded and the hand bone pose will be rolled back to the pose of the previous safe rendering frame.

10. The method for generating multimodal digital human content across the entire chain based on a persona agent according to claim 1, characterized in that, The step of synchronously encapsulating and driving rendering pipeline updates includes: Using the facial blending deformation scalar weights and skeleton rotation Euler angles carried in the final fusion action parameter tensor, the linear blending skinning algorithm is invoked to generate a global transformation matrix to update the vertex coordinates of the 3D mesh model; Compare the video frame timestamp with the current audio playback timestamp. When the video frame timestamp lags behind and the difference exceeds the theoretical rendering time threshold, discard the current video frame and accelerate the extraction of parameters for the next frame for rendering. When the video frame timestamp is ahead, the rendering engine enters a spin wait state until the current audio playback timestamp catches up with the video frame timestamp or reaches the preset maximum spin timeout threshold.

Images