Digital human smooth image generation method and system based on dynamic threshold triggering

Abstract

The invention provides a digital human smooth image generation method and system based on dynamic threshold triggering, and belongs to the technical field of digital human image generation. The method comprises the following steps: collecting a real-time audio stream, and extracting audio features; adding the audio features to a preset audio feature buffer area; dynamically adjusting a trigger frame number threshold value generated by the image according to the content complexity; judging whether the number of feature frames accumulated in the current audio feature buffer region meets a triggering condition set based on the triggering frame number threshold value or not; and if the triggering condition is met, extracting the latest multi-frame audio features of which the number is equal to the triggering frame number threshold from the audio feature buffer area, remodeling the multi-frame audio features into a model input tensor, and inputting the model input tensor into an image generation model to generate a corresponding face region image. Through a dynamic threshold mechanism, a generation strategy can be adaptively adjusted according to audio content complexity, on-demand allocation of computing resources is realized, and the resource utilization rate is optimized.

Description

Technical Field

[0001] This invention relates to the field of digital human image generation technology, specifically to a method and system for generating smooth digital human images based on dynamic threshold triggering. Background Technology

[0002] In the field of real-time rendering technology for digital humans, generating corresponding image frames in real time based on input audio features is the core task for realizing real-time interaction of digital humans. Currently, mainstream image generation methods typically employ a frame-by-frame synchronous processing mechanism, meaning that the image generation model is immediately invoked for calculation upon receiving each frame of audio features. While this approach ensures a low baseline latency, the dynamic characteristics of the audio stream, such as changes in speech rate and syllable density fluctuations, lead to an unbalanced computational load on the system. When the audio content is complex or drastically changing, the processing time for a single frame increases significantly; conversely, when the audio content is stable, computational resources are not fully utilized. This load imbalance causes significant latency fluctuations, resulting in an unstable frame rate and ultimately leading to stuttering, skipping, and other discontinuous phenomena in the rendered image sequence, severely impacting the user's visual experience in real-time interactive scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for generating smooth digital human images based on dynamic threshold triggering, which generates a face image by dynamically adjusting the threshold triggered by audio features and then fuses it with a full-body image to achieve smooth digital human image generation.

[0004] To achieve the above objectives, this invention provides a method for generating smooth digital human images based on dynamic threshold triggering, comprising: acquiring a real-time audio stream; processing the real-time audio stream data frame by frame and extracting audio features; adding the audio features to a preset audio feature buffer; calculating the dynamic change rate of the latest consecutive multi-frame audio features in the audio feature buffer as content complexity; dynamically adjusting the trigger frame number threshold for image generation according to the content complexity; determining whether the number of feature frames accumulated in the current audio feature buffer meets the trigger condition set based on the trigger frame number threshold; if the trigger condition is met, extracting the latest multi-frame audio features from the audio feature buffer, the number of which is equal to the trigger frame number threshold, and reshaping them into a model input tensor; inputting the model input tensor into an image generation model to generate a corresponding face region image; and fusing the face region image with a preset full-body image to obtain a complete digital human image.

[0005] Optionally, the audio feature extraction includes: performing fixed-time-window frame segmentation on the real-time audio stream to obtain a continuously arranged audio frame sequence; preprocessing each audio frame in the audio frame sequence, the preprocessing including pre-emphasis processing, windowing processing, and Fourier transform processing; inputting each preprocessed audio frame into a pre-trained audio feature extraction network, the audio feature extraction network outputting a fixed-dimensional audio feature tensor; adjusting the audio feature tensor to a feature vector that conforms to a preset dimension specification, and using this feature vector as the audio feature.

[0006] Optionally, calculating the dynamic change rate of the latest consecutive multiple frames of audio features in the audio feature buffer includes: obtaining the audio features of the latest consecutive preset number of frames from the audio feature buffer in time sequence, calculating the difference value between the audio feature vectors of adjacent frames to obtain an inter-frame difference value sequence; and obtaining the dynamic change rate by calculating the variance of the inter-frame difference value sequence.

[0007] Optionally, the dynamic adjustment of the trigger frame number threshold for image generation includes: predefining a negative correlation mapping rule between content complexity and trigger frame number threshold; obtaining the currently calculated content complexity value; inputting the content complexity value into the negative correlation mapping rule; and outputting the corresponding trigger frame number threshold.

[0008] Optionally, the predefined negative correlation mapping rule between content complexity and trigger frame number threshold includes: configuring a lower limit, an upper limit, and a default value for the trigger frame number threshold; configuring a high complexity threshold and a low complexity threshold for the content complexity value; and establishing a negative correlation mapping rule, including: setting the trigger frame number threshold to the lower limit when the content complexity value is greater than the high complexity threshold; setting the trigger frame number threshold to the upper limit when the content complexity value is less than the low complexity threshold; and setting the trigger frame number threshold to the default value when the content complexity value is greater than or equal to the low complexity threshold and less than or equal to the high complexity threshold.

[0009] Optionally, determining whether the number of accumulated feature frames in the current audio feature buffer meets the triggering condition set based on the triggering frame number threshold includes: monitoring the operating status of the audio feature buffer and obtaining the total number of feature frames currently stored in the audio feature buffer; comparing the total number of feature frames with the triggering frame number threshold dynamically determined based on content complexity; generating a triggering judgment signal based on the result of the numerical comparison; and generating a triggering judgment signal indicating that the triggering condition is met when the total number of feature frames reaches or exceeds the triggering frame number threshold.

[0010] Optionally, generating a trigger judgment signal based on the numerical comparison result includes: setting the critical judgment condition of the trigger condition judgment to be that the total number of feature frames currently stored in the audio feature buffer is greater than or equal to the trigger frame number threshold; when the critical judgment condition is met, generating a trigger execution signal immediately to trigger the execution of the subsequent image generation process; when the critical judgment condition is not met, controlling the audio feature buffer to maintain a waiting reception state and continuously receiving newly added audio feature frames.

[0011] Optionally, the step of extracting the latest multi-frame audio features from the audio feature buffer, the number of which is equal to the trigger frame number threshold, and reshaping them into a model input tensor includes: in response to a trigger judgment signal representing that the trigger condition is met, determining the number of audio feature frames to be extracted, the number of audio feature frames being equal to the currently effective trigger frame number threshold; extracting the latest corresponding number of consecutive audio feature frames from the end of the audio feature buffer cache; and concatenating and transforming the extracted multi-frame audio features along the feature dimension to generate a tensor format that meets the input requirements of the image generation model.

[0012] Optionally, the step of concatenating and transforming the extracted multi-frame audio features along the feature dimension includes: concatenating the extracted multi-frame audio features along the feature dimension in chronological order to obtain a concatenated feature tensor; performing dimension reshaping on the concatenated feature tensor to convert it into an input tensor preset by the image generation model; and during the dimension reshaping process, keeping the batch dimension of the concatenated feature tensor unchanged, and only adjusting the feature channel dimension and spatial dimension of the feature tensor.

[0013] On the other hand, the present invention provides a digital human smooth image generation system based on dynamic threshold triggering, for implementing a digital human smooth image generation method based on dynamic threshold triggering. The system includes a control module, the control module including a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the digital human smooth image generation method based on dynamic threshold triggering.

[0014] The aforementioned technical solution, by introducing a feature buffer and combining it with a dynamic threshold triggering mechanism, effectively eliminates latency fluctuations caused by uneven computational load in traditional frame-by-frame processing methods, achieving smooth and continuous image generation and fundamentally solving the problems of image stuttering and skipping. Furthermore, the dynamic threshold mechanism can adaptively adjust the generation strategy based on the complexity of the audio content, enabling on-demand allocation of computational resources and optimizing resource utilization. This method can improve the visual experience in real-time interactive applications, making the presentation of digital humans more natural and immersive, effectively enhancing user satisfaction.

[0015] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a digital human smoothing image generation method based on dynamic threshold triggering.

[0017] Figure 2 This is a flowchart of dynamic threshold control. Detailed Implementation

[0018] The following is in conjunction with the appendix Figure 1 -Appendix Figure 2 The specific implementation methods of the embodiments of the present invention will be described in detail below. It should be understood that the specific implementation methods described herein are only for illustrating and explaining the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0019] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0020] In the process of realizing this invention, the inventors of this application discovered that the existing frame-by-frame synchronization processing mechanism can cause uneven computational load due to dynamic changes in audio content, which in turn leads to latency fluctuations and causes discontinuous image generation.

[0021] Example 1 Reference Figures 1-2 This is the first embodiment of the present invention, which provides a method for generating smoothed digital human images based on dynamic threshold triggering, comprising: S100: Acquires real-time audio streams, processes the real-time audio stream data frame by frame, and extracts audio features.

[0022] In a preferred embodiment of this application, before the digital human image generation system based on multi-frame feature buffers is officially started and begins to process the real-time audio stream in a loop to generate a digital human image sequence, a one-time initialization step is performed: the feature buffer is initialized, and K=7 frames of audio features are preset. This operation is performed only once to establish the initial state for subsequent loops.

[0023] In the embodiments of this application, a fixed time window framing process is performed on the real-time audio stream to obtain a continuously arranged audio frame sequence; each audio frame in the audio frame sequence is preprocessed, including pre-emphasis processing, windowing processing, and Fourier transform processing; each preprocessed audio frame is input into a pre-trained audio feature extraction network, and the audio feature extraction network outputs a fixed-dimensional audio feature tensor; the audio feature tensor is adjusted to a feature vector that conforms to a preset dimension specification, and this feature vector is used as the audio feature.

[0024] In a preferred embodiment of this application, an audio stream signal is acquired in real time via an audio input device, and the continuous audio stream is segmented into frames according to a preset fixed time window (e.g., 20-40 milliseconds) to obtain a series of continuous audio frames. Then, for each frame of audio signal, a pre-trained audio feature extraction model based on deep learning networks such as CNN (Convolutional Neural Network) or RNN (Recurrent Neural Network) is used to extract high-level semantic features that can represent the speech content. The audio features output by the pre-trained audio feature extraction model for each frame are fixed-dimensional feature vectors. Next, the extracted audio feature tensor is normalized to a preset standardized dimension of [1, 16, 512] to adapt to the unified processing of the subsequent audio feature buffer and image generation model.

[0025] The above scheme, through fixed-time-window framing and pre-emphasis, windowing, and Fourier transform preprocessing, effectively improves the regularity of the audio signal and reduces noise interference, laying a solid foundation for subsequent feature extraction. Standardizing audio features into a unified standard dimension facilitates seamless adaptation with subsequent audio feature buffers and image generation models, ensuring smooth data compatibility across modules.

[0026] S200: Add audio features to the preset audio feature buffer.

[0027] In a preferred embodiment of this application, the buffer management module receives standardized audio features output from the feature extraction module through its data input interface and appends these audio features to the end of a preset audio feature buffer, increasing the number of feature frames stored in the audio buffer by one frame. During this process, the buffer management module synchronously updates the audio feature buffer status parameters, including the total number of currently stored feature frames. Finally, it outputs an audio feature buffer update completion signal.

[0028] It should be noted that the feature addition operation of the buffer management module and the subsequent feature removal step together constitute a complete first-in-first-out (FIFO) buffer management strategy.

[0029] The above scheme stores audio features according to the append-to-end rule, strictly preserving the temporal order of features. This ensures that the latest feature frames extracted subsequently accurately reflect the current audio context information, providing a temporal guarantee for the accurate generation of digital human images. The buffer management module synchronously updates status parameters such as the total number of currently stored feature frames, providing accurate and real-time data support for subsequent trigger condition judgments and content complexity calculations, ensuring the reliability of subsequent decision-making logic. The feature addition operation and the subsequent old frame removal step together constitute a complete FIFO management strategy, avoiding feature frame storage chaos and ensuring that the buffer always operates in an orderly manner, laying the rule foundation for subsequent feature extraction and cleanup operations.

[0030] S300: Calculate the dynamic rate of change of the latest consecutive multi-frame audio features in the audio feature buffer as the content complexity.

[0031] In the embodiments of this application, the audio features of the latest consecutive preset number of frames are obtained from the audio feature buffer in time sequence, the difference value between the audio feature vectors of adjacent frames is calculated, and the inter-frame difference value sequence is obtained; the dynamic change rate is obtained by calculating the variance of the inter-frame difference value sequence.

[0032] In a preferred embodiment of this application, the latest eight consecutive frames of audio features are extracted from the audio feature buffer in chronological order and used as a content complexity analysis window. Then, the Euclidean distance between the audio feature vectors of two adjacent frames within this analysis window is calculated to generate seven inter-frame difference values, and then an inter-frame difference value sequence is constructed.

[0033] Furthermore, variance is calculated on the inter-frame difference value sequence, and the calculated variance value is defined as the dynamic change rate, which is a quantitative indicator of content complexity.

[0034] The above scheme calculates the inter-frame difference value by measuring the Euclidean distance between the feature vectors of adjacent frames, and then transforms the abstract content complexity into a directly quantifiable indicator through variance calculation. This provides an objective and operable basis for subsequent dynamic adjustment of the trigger frame number threshold. The variance calculation method can accurately reflect the degree of speech change; the larger the variance, the more drastic the content change, and the smaller the variance, the more stable the content. This makes the complexity assessment results more consistent with the dynamic characteristics of actual speech expression, improving the targeting of subsequent threshold adjustments.

[0035] S400: Dynamically adjusts the threshold for triggering the number of frames generated for image generation based on the complexity of the content.

[0036] In the embodiments of this application, a negative correlation mapping rule between content complexity and trigger frame number threshold is predefined; the currently calculated content complexity value is obtained; the content complexity value is input into the negative correlation mapping rule, and the corresponding trigger frame number threshold is output.

[0037] In the embodiments of this application, a lower limit, an upper limit, and a default value are configured for the trigger frame count threshold; a high complexity threshold and a low complexity threshold are configured for the content complexity value; and a negative correlation mapping rule is established, including: when the content complexity value is greater than the high complexity threshold, the trigger frame count threshold is set to the lower limit; when the content complexity value is less than the low complexity threshold, the trigger frame count threshold is set to the upper limit; and when the content complexity value is greater than or equal to the low complexity threshold and less than or equal to the high complexity threshold, the trigger frame count threshold is set to the default value.

[0038] In a preferred embodiment of this application, the default value T0 of the trigger frame number threshold is set to 8, and the lower limit value is set to... =5, Upper limit value =12. Low and high complexity thresholds are determined by statistically analyzing the inter-frame difference variance of different speech datasets. A complexity calculation window (N=K+1=8) containing the features of the latest 8 frames is set, where K represents the baseline preset frame number (preset value is 7), and N represents the number of frames in the complexity calculation window (calculated result is 8). Then, a continuous loop process is entered. Each time, a new audio feature frame with dimensions [1, 16, 512] is received and added to the buffer to increment the frame number. The latest 8 frame features are then extracted from the buffer, and the Euclidean distance (or cosine similarity transformation distance) between the feature vectors of adjacent frames is calculated sequentially to obtain 7 difference values. The variance of these difference values ​​is calculated to characterize the content complexity (the larger the variance, the more drastic the change; the smaller the variance, the more stable the change). Next, the trigger frame number threshold T is dynamically mapped based on the comparison results of content complexity, low complexity threshold, and high complexity threshold.

[0039] Furthermore, when the content complexity value exceeds the high complexity threshold, the trigger frame count threshold T is set to the lower limit. =5, when the content complexity value is less than the low complexity threshold T, the trigger frame count threshold is set to the upper limit. =12. When the content complexity value is greater than or equal to the low complexity threshold and less than or equal to the high complexity threshold, the trigger frame threshold T will be set to the default value T0=8.

[0040] The above solution achieves intelligent adjustment of triggering based on demand by establishing a negative correlation mapping rule between content complexity and trigger frame threshold. In high-complexity scenarios, setting the threshold to the lower limit allows for rapid image generation, preventing the digital human image from lagging behind the speech content. In low-complexity scenarios, setting the threshold to the upper limit ensures smooth image generation. A default threshold is used for medium-complexity scenarios, balancing the risk of lag with the smoothing effect, comprehensively covering the needs of different speech scenarios. Furthermore, by configuring upper and lower limits for the trigger threshold and critical values ​​for complexity, the solution avoids unlimited fluctuations in the threshold, ensuring a stable image generation rhythm that avoids wasting system resources due to frequent triggering or causing content lag due to prolonged inactivity.

[0041] S500: Determine whether the number of feature frames accumulated in the current audio feature buffer meets the triggering condition set based on the trigger frame number threshold.

[0042] In the embodiments of this application, the operating status of the audio feature buffer is monitored to obtain the total number of feature frames currently stored in the audio feature buffer; the total number of feature frames is compared with a trigger frame number threshold dynamically determined based on content complexity; a trigger judgment signal is generated based on the result of the numerical comparison; when the total number of feature frames reaches or exceeds the trigger frame number threshold, a trigger judgment signal representing that the trigger condition is met is generated.

[0043] Furthermore, the critical condition for triggering the judgment is set to be that the total number of feature frames currently stored in the audio feature buffer is greater than or equal to the trigger frame number threshold. When the critical condition is met, a trigger execution signal is generated immediately to trigger the execution of the subsequent image generation process. When the critical condition is not met, the audio feature buffer is controlled to maintain a waiting state and continuously receive new audio feature frames.

[0044] In a preferred embodiment of this application, the operating status of the audio feature buffer is monitored in real time to accurately obtain the total number of feature frames accumulated in the current buffer (denoted as Current_Frame_Count, which is calculated by combining the number of newly received feature frames and the number of the oldest removed frames from the initial preset 7-frame baseline). Then, this total number of feature frames is directly compared with a dynamically determined trigger frame threshold T. The criterion is whether Current_Frame_Count is greater than or equal to T. Based on the comparison result, a corresponding trigger judgment signal is generated. If Current_Frame_Count ≥ T, meaning the critical condition is met, a trigger execution signal is generated immediately to trigger the execution of the subsequent S5 image generation process (including feature reshaping, U-Net model inference, image post-processing, and pasting back the full-body image, etc.). If Current_Frame_Count < T, meaning the critical condition is not met, a judgment signal indicating "not meeting the trigger condition" is generated. At this time, the audio feature buffer is controlled to remain in a waiting state, without performing any image generation operations, and continues to receive newly added audio feature frames until the total number of feature frames accumulated in the buffer reaches or exceeds the trigger frame count threshold T, and the critical condition is met.

[0045] The above scheme precisely controls the timing of image generation through clearly defined critical judgment conditions, avoiding both premature triggering (insufficient feature frames leading to poor image quality and missing details) and delayed triggering (rapid audio content progression causing image lag). When the triggering conditions are not met, the control buffer remains in a waiting state, continuously receiving new audio feature frames to ensure sufficient effective features are acquired for subsequent triggering, thus guaranteeing the continuity and integrity of digital human image generation.

[0046] S600: If the triggering condition is met, extract the latest multi-frame audio features from the audio feature buffer, the number of which is equal to the trigger frame number threshold, and reshape them into the model input tensor.

[0047] In the embodiments of this application, in response to a trigger judgment signal that represents the fulfillment of the trigger condition, the number of audio feature frames to be extracted is determined, and the number of audio feature frames is equal to the threshold of the number of trigger frames currently in effect; the latest number of consecutive audio feature frames are extracted from the end of the audio feature buffer; the extracted multi-frame audio features are spliced ​​and dimensionally transformed in the feature dimension to generate a tensor format that meets the input requirements of the image generation model.

[0048] In the embodiments of this application, the extracted multi-frame audio features are concatenated in chronological order along the feature dimension to obtain a concatenated feature tensor; the concatenated feature tensor is subjected to dimension reshaping to convert it into an input tensor preset by the image generation model; during the dimension reshaping process, the batch dimension of the concatenated feature tensor remains unchanged, and only the feature channel dimension and spatial dimension of the feature tensor are adjusted.

[0049] In a preferred embodiment of this application, a trigger judgment signal that represents the image generation trigger condition is responded to, and the number of audio feature frames to be extracted is determined. The number of audio feature frames to be extracted is consistent with the currently effective dynamic trigger frame number threshold T. The dynamic trigger frame number threshold T is a value obtained based on the dynamic mapping of audio content complexity, and its value ranges between the lower limit of 5 and the upper limit of 12. Based on the first-in-first-out (FIFO) queue storage rule adopted by the audio feature buffer, the latest T consecutive audio features are extracted from the tail of the audio feature buffer to ensure that the features of the input model can reflect the latest audio context information and avoid the lag in digital human image generation caused by old features.

[0050] Furthermore, the extracted T-frame audio features are concatenated along the feature dimension according to their chronological order of entry into the buffer, resulting in an initial concatenated feature tensor. Dimensional reshaping is then performed on this initial concatenated feature tensor, maintaining the batch dimension of the tensor while adjusting only the feature channel dimension and spatial dimension. This process ultimately converts the tensor into the input tensor format pre-defined by the pre-trained U-Net model. For example, when the dynamic trigger frame threshold T=8, the eight single-frame audio features with dimensions [1, 16, 512] are concatenated and reshaped to form a tensor of [1, 128, 16, 32] to meet the input requirements of the U-Net model.

[0051] In a preferred embodiment of this application, to address the issue of inconsistent intermediate tensor dimensions caused by the dynamically triggered frame number threshold T, an adaptation layer is added after dimension reshaping to perform unified channel number mapping. The specific design is as follows: 1. Adaptation layer structure: A 1×1 convolutional layer is used (only the number of channels is adjusted, without changing the spatial dimension). The number of convolutional kernels is set to a fixed value of 128. The activation function is ReLU (Rectified Linear Unit) to preserve the non-linear expressive ability of features. This layer shares training parameters with the subsequent U-Net model.

[0052] 2. Mapping logic: Regardless of the value of T between 5 and 12, the above intermediate tensors, after being processed by the adaptation layer, will uniformly output a model input tensor with a fixed dimension of [1, 128, 16, 32], ensuring that it fully matches the input requirements of the pre-trained U-Net model.

[0053] 3. Training and Adaptation: The adaptation layer can be trained end-to-end with the U-Net model (the training data needs to cover the entire scene from T=5 to 12), or it can be jointly fine-tuned based on the pre-trained U-Net model to ensure that the audio semantic features corresponding to different T values ​​do not lose key information after mapping.

[0054] Furthermore, after completing multi-frame audio feature extraction and model input tensor reshaping, the buffer management module simultaneously performs an old frame cleanup operation: based on the first-in-first-out (FIFO) storage rule, it retains the latest T-frame audio features in the audio feature buffer (i.e., the T-frame continuous features used to generate the image this time), and deletes all earlier feature frames stored before this T-frame. This cleanup operation, on the one hand, strictly controls the buffer's storage capacity, avoiding excessive resource consumption caused by the infinite accumulation of feature frames. On the other hand, it reserves basic feature frames for the next round of content complexity calculation. Subsequently, for each new audio feature frame received, the number of frames in the buffer gradually increases from T frames until it meets the complexity analysis window requirement of the latest 8 frames, ensuring the continuity of the next round of the process (for example, when T=5, the buffer retains 5 frames after cleanup, and receiving 3 more new features is enough to make 8 frames, without needing to wait for more frames).

[0055] The above scheme extracts the latest corresponding number of audio features from the tail of the audio feature buffer, ensuring that the features of the input image generation model can reflect the latest audio context information, effectively avoiding the lag problem in digital human image generation caused by old features. By concatenating multi-frame features in chronological order and adjusting the feature channels and spatial dimensions through dimensionality reshaping operations, the multi-frame audio features are transformed into a tensor format that meets the model's input requirements, providing adaptive input for subsequent image generation.

[0056] S700: The model is generated by taking the input tensor image and generating the corresponding face region image.

[0057] In a preferred embodiment of this application, the restructured model input tensor (e.g., with dimensions [1, 128, 16, 32], adapted to a preset dynamic trigger frame threshold T) is loaded into the input layer of the pre-trained U-Net model. This U-Net model employs an encoder-decoder architecture. Channel pruning reduces the number of encoder channels from 64 to 128 to 256 to decrease the total number of parameters. Simultaneously, the weights are quantized to 16-bit floating-point or 8-bit integers. Within a controllable accuracy range where the facial keypoint error is ≤2 pixels, the inference speed is increased by 2 to 4 times. The lightweight U-Net model uses depthwise separable convolutions instead of standard convolutions, reducing the computational cost to 1 / 8 to 1 / 10 of the original model, while retaining skip connections to fuse shallow textures and deep semantic features to avoid loss of facial details.

[0058] Furthermore, to reduce inference time, the lightweight U-Net model is adapted to the ONNX Runtime or TensorRT inference engine during the inference deployment phase. Optimization methods such as operator fusion and layer fusion are used to ensure that single-frame inference is controlled within 30ms. The model input tensor progressively compresses the feature map spatial dimension and increases the number of feature channels through the encoder path, then progressively restores the spatial resolution through transposed convolution through the decoder path. The final convolutional layer maps the number of feature channels to an RGB three-channel image, outputting a 160×160 pixel [1, 3, 160, 160] face region image tensor. Subsequently, fast numerical scaling is performed on this face region image tensor, linearly mapping pixel values ​​from the model output range of [-1, 1] or [0, 1] to the standard 8-bit image range of [0, 255], and converting the data type to 8-bit unsigned integer (uint8). Finally, a simplified version of the lightweight adaptive histogram equalization algorithm is used for post-processing to optimize the image visual effect without increasing the time consumption, ensuring that the accuracy requirements of subsequent fusion are met.

[0059] The above scheme, through optimization techniques such as channel pruning, weight quantization, and replacing standard convolution with depthwise separable convolution, significantly reduces the computational load of the model and significantly improves the inference speed while strictly controlling the accuracy constraints of facial key point errors within an acceptable range, perfectly balancing the generation quality and real-time requirements of digital facial images.

[0060] S800: The face region image is fused with a preset full-body image to obtain a complete digital human image.

[0061] In a preferred embodiment of this application, a lightweight real-time detection model, such as a simplified version of MediaPipe Face Mesh, is first used to perform facial landmark detection on a 160×160 pixel face region image, locating the pixel coordinates of key feature points such as the corners of the eyes, mouth, and nose. Then, a pre-stored full-body image template (containing fixed elements such as digital human body structure and background scene, reserving face region matching positions and having a resolution adapted to the preset requirements of the output video stream) is invoked, and the preset standard facial landmark coordinates in the template are obtained.

[0062] Furthermore, based on the corresponding keypoint coordinates of the generated face and the template, a simplified affine transformation matrix calculation method that retains only the core parameters of translation, rotation, and scaling is adopted to quickly obtain the optimal transformation matrix, effectively reducing computational complexity. Subsequently, the pre-generated soft-edge mask and the generated face image are synchronously subjected to affine transformation, and the mask edges are processed with a 3×3 simplified convolution kernel to perform fast Gaussian blurring to form smooth soft edges and avoid harsh fusion boundaries.

[0063] Furthermore, an optimized alpha fusion algorithm is employed, replacing the traditional pixel-by-pixel loop with parallel vector computation. The final image is calculated as (face image × mask) + (template image × (1 - mask)) to achieve a natural, gradual transition between the face and the template. Finally, the fused [512, 512, 3] complete digital human image is encapsulated in the output video stream format and sent to a thread-safe output video stream queue. This decouples the image from the audio processing thread, ensuring that the image generation task runs in parallel in the background without blocking audio stream processing. The maximum length of this output queue is set to 3. When the queue is full, the oldest task is automatically discarded, and the most recently generated image frame is retained first, ensuring the timeliness of the digital human image output. Ultimately, the complete transformation from audio features to a concrete digital human image frame is completed, resulting in a complete digital human image.

[0064] The present invention also provides a digital human smooth image generation system based on dynamic threshold triggering, for implementing a digital human smooth image generation method based on dynamic threshold triggering. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the digital human smooth image generation method based on dynamic threshold triggering.

[0065] This invention provides a storage medium storing a program that, when executed by a processor, implements a method for generating smooth digital human images based on dynamic threshold triggering.

[0066] This invention provides a processor for running a program, wherein the program executes a digital human smoothing image generation method based on dynamic threshold triggering.

[0067] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for generating smoothed digital human images based on dynamic threshold triggering. The device described herein can be a server, PC, tablet, mobile phone, etc.

[0068] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing a method for generating smooth images of digital humans based on dynamic threshold triggering.

[0069] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0074] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0075] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0076] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating a digital human smoothed image based on a dynamic threshold trigger, characterized in that, The method comprises the following steps: Collecting a real-time audio stream, processing the real-time audio stream data frame by frame, and extracting audio features; Adding the audio features to a preset audio feature buffer; Calculating the dynamic change rate of the latest continuous multiple frames of audio features in the audio feature buffer as the content complexity; Dynamically adjusting the trigger frame number threshold of image generation according to the content complexity; Judging whether the number of accumulated feature frames in the current audio feature buffer meets the trigger condition set based on the trigger frame number threshold; If the trigger condition is met, extracting the latest multiple frames of audio features equal to the trigger frame number threshold from the audio feature buffer, and reshaping them into a model input tensor; Inputting the model input tensor into an image generation model to generate a corresponding face region image; Fusing the face region image with a preset full-body image to obtain a complete digital human image.

2. The method of claim 1, wherein, The extraction of audio features comprises: Performing fixed time window frame processing on the real-time audio stream to obtain a continuous audio frame sequence; Respectively pre-processing each audio frame in the audio frame sequence, wherein the pre-processing comprises pre-emphasis processing, windowing processing, and Fourier transform processing; Respectively inputting each pre-processed audio frame into a pre-trained audio feature extraction network, wherein the audio feature extraction network outputs an audio feature tensor of a fixed dimension; Adjusting the audio feature tensor to a feature vector conforming to a preset dimension specification, and taking the feature vector as the audio feature.

3. The method of claim 1, wherein, The calculation of the dynamic change rate of the latest continuous multiple frames of audio features in the audio feature buffer comprises: Obtaining the latest continuous preset number of frames of audio features from the audio feature buffer in chronological order, calculating the difference values between adjacent frame audio feature vectors to obtain a frame difference value sequence; Calculating the variance of the frame difference value sequence to obtain the dynamic change rate.

4. The method of claim 1, wherein, The dynamic adjustment of the trigger frame number threshold of image generation comprises: Predefining a negative correlation mapping rule between the content complexity and the trigger frame number threshold; Obtaining the current calculated content complexity value; Inputting the content complexity value into the negative correlation mapping rule to output the corresponding trigger frame number threshold.

5. The method of claim 4, wherein, The predefining of the negative correlation mapping rule between the content complexity and the trigger frame number threshold comprises: Configuring a lower limit value, an upper limit value, and a default value for the trigger frame number threshold; Configuring a high complexity threshold and a low complexity threshold for the content complexity value; Establishing a negative correlation mapping rule, comprising: When the content complexity value is greater than the high complexity threshold, setting the trigger frame number threshold to the lower limit value; When the content complexity value is less than the low complexity threshold, setting the trigger frame number threshold to the upper limit value; When the content complexity value is greater than or equal to the low complexity threshold and less than or equal to the high complexity threshold, setting the trigger frame number threshold to the default value.

6. The method of claim 1, wherein, The judgment of whether the number of accumulated feature frames in the current audio feature buffer meets the trigger condition set based on the trigger frame number threshold comprises: Monitoring the running state of the audio feature buffer, obtaining the total number of feature frames currently stored in the audio feature buffer; Numerical comparison between the total number of feature frames and a trigger frame number threshold dynamically determined based on content complexity; Generating a trigger judgment signal based on the result of the numerical comparison, and generating a trigger judgment signal representing that the trigger condition is met when the total number of feature frames reaches or exceeds the trigger frame number threshold.

7. The method of claim 6, wherein, The generation of the trigger judgment signal based on the result of the numerical comparison includes: Setting the critical decision condition for the trigger condition judgment as the total number of feature frames currently stored in the audio feature buffer being greater than or equal to the trigger frame number threshold; When the critical decision condition is met, a trigger execution signal is generated immediately, triggering the subsequent image generation process after execution; When the critical decision condition is not met, the audio feature buffer is controlled to maintain a waiting receiving state, and the newly added audio feature frames are continuously received.

8. The method of claim 1, wherein, The extraction of the latest multiple frames of audio features equal to the trigger frame number threshold from the audio feature buffer, and the reshaping of the model input tensor, includes: In response to the trigger judgment signal representing that the trigger condition is met, determining the number of audio feature frames currently required to be extracted, which is equal to the currently effective trigger frame number threshold; Extracting the corresponding number of continuous audio feature frames newly stored from the tail of the audio feature buffer cache; Concatenating and dimension transforming the extracted multiple frames of audio features in the feature dimension to generate a tensor format conforming to the input requirements of the image generation model.

9. The method of claim 8, wherein, The concatenation and dimension transformation of the extracted multiple frames of audio features in the feature dimension includes: Concatenating the extracted multiple frames of audio features in the feature dimension in time sequence to obtain a concatenated feature tensor; Performing dimension reorganization processing on the concatenated feature tensor to convert it into an input tensor preset by the image generation model; During the dimension reorganization processing, the batch dimension of the concatenated feature tensor is kept unchanged, and only the feature channel dimension and the spatial dimension of the feature tensor are adjusted.

10. A dynamic threshold trigger based digital human smoothing image generation system, characterized in that, The system includes a control module, the control module includes a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the digital human smooth image generation method based on dynamic threshold triggering according to any one of claims 1-9.

Images