Digital human generation system and method based on language model

Abstract

The invention discloses a digital human generation system and method based on a language model, and relates to the technical field of artificial intelligence. A complete digital human pronunciation modeling path from voice signal acquisition, formant frequency extraction, deviation analysis and pronunciation stability evaluation to motion compensation control and three-dimensional animation fusion is realized. Compared with the existing mode of driving the digital human pattern only based on energy envelope or mouth shape classification, the method has the advantages that the multi-frame formant frequency of the user voice is extracted and processed, so that the digital human can generate dynamic lingual surface and tongue tip actions according to the sound channel change in the real pronunciation process of the user; therefore, the physical correspondence and linguistic consistency of the pronunciation actions of the digital human are improved. According to the method, the direct mapping between the pronunciation characteristics and the three-dimensional tongue actions is realized, and the authenticity of the pronunciation animation of the digital human and the guidance in a teaching scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a digital human generation system and method based on a language model. Background Technology

[0002] With the development of artificial intelligence technology, digital human generation technology has become an important component of multiple technical fields such as intelligent interaction, virtual teaching, and film and television production, belonging to the digital human-driven sub-field within the multimodal generation field of artificial intelligence. Among these, voice-driven digital human generation technology, which processes user-input voice signals to drive facial expressions, mouth movements, or simulated articulation points in digital humans, is a typical interactive generation application. Within this sub-field, the technology of constructing intraoral movements (including tongue surface, tongue tip, and jaw displacement) based on voice formant information is one of the important technical routes for achieving high-precision digital human pronunciation simulation. Its core task is to capture the vocal tract morphological changes during the user's actual pronunciation process and map them into corresponding three-dimensional tongue dynamic animation.

[0003] Currently, existing voice-driven digital lip-reading and tongue-gesture systems generally suffer from insufficient accuracy. Traditional technologies typically rely on energy envelopes, fundamental frequencies (F0), universal lip-reading templates, or classification models to drive lip movements, but they cannot effectively describe the subtle changes in the vocal tract structure, such as tongue height, tongue front-back position, and tongue tip movement, during pronunciation. Especially in applications requiring high-precision "articular movement reproduction," such as drama teaching and broadcasting training, existing solutions cannot capture the true "articular exertion point" based on formant changes during pronunciation. This results in coarse tongue-gesture performance in digital humans, insensitive tongue surface tracking, and an inability to reflect the differences between standard pronunciation samples and user pronunciation, thus affecting teaching effectiveness and visual presentation accuracy.

[0004] The aforementioned shortcomings primarily stem from the insufficient utilization of formant information in the existing system's articulation motion mapping. It cannot perform dynamic deviation analysis of the multi-frame formant frequency trajectories of the user's speech signal, nor can it perform fine-grained compensation adjustments to tongue movements based on the degree of deviation. When the user's pronunciation is unstable, tongue position is severely deviated, or the articulation points differ significantly from standard samples, the existing system cannot identify these deviations or perform targeted fine-tuning based on them. This results in abnormal behaviors in the digital human's oral animation, such as "tongue lag," "mouth shape misalignment," and "lack of tongue tip response." These abnormalities are particularly pronounced when simulating tongue tip movements or jaw coordination, severely impacting the digital human's pronunciation realism, instructional guidance, and interactive experience. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a digital human generation system and method based on a language model, which solves the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution, comprising the following steps: S1. Collect the user's voice signal, preprocess the collected voice signal to obtain the formant frequency information of each frame of voice, and transfer the formant frequency information to the voice data storage database. S2. Extract resonance frequency information and standard samples from the speech storage database, perform deviation analysis, obtain the deviation magnitude Dev, construct the pronunciation stability score StabScore, and then conduct a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. S3. Based on the preliminary articulation stability assessment, trigger the articulation part compensation drive, extract and find the keyframe index n* based on the deviation magnitude Dev, and compare it with the maximum allowable deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; S4. Based on the range of the Tongue Compensation Control Factor TongueAdjRate, the control level is divided, and the tongue control strategy is executed based on the control level division results. Then, the execution results of the tongue control strategy are digitally fused with the digital human generation module to drive the three-dimensional digital human oral animation system and output a tongue dynamic animation clip that is synchronized with the speech and tongue movements.

[0007] Preferably, S1 includes S11; S11. The voice signal emitted by the user in real time is collected by the microphone set in the digital human user terminal, and the voice signal is segmented into frames to obtain the framed voice signal. The framed voice signal is uploaded to a cloud-based voice processing server via a local communication interface. The cloud-based voice processing server includes a preset voice data storage database. The speech data storage database includes a speech signal storage table, a resonance data storage table, and a standard sample storage table.

[0008] Preferably, S1 further includes S12; S12. In the cloud-based speech processing server, extract the framed speech signal from the speech signal storage table of the speech data storage database, perform preprocessing, and perform formant analysis processing based on the preprocessed framed speech signal to extract the formant frequency information of each frame of speech. The formant frequency information includes the first formant frequency F1(n) of the nth frame, the second formant frequency F2(n) of the nth frame, and the third formant frequency F3(n) of the nth frame. The preprocessing includes spectral subtraction noise reduction, pre-emphasis processing, frame-segmentation windowing, and linear prediction cepstral analysis.

[0009] Preferably, S2 includes S21; S21, the cloud-based voice processing server reads user formant frequency information and standard sample formant frequency information from the resonance data storage table and standard sample storage table of the voice data storage database, respectively. Based on the three sets of formant frequency differences between the user and the standard sample at the same frame position, deviation analysis is performed to obtain the deviation magnitude Dev. The deviation magnitude Dev is calculated and output using the following algorithm formula: In the formula, Dev(n) represents the deviation amplitude of the nth frame, uF1 represents the first formant frequency F1 of the standard sample, uF2 represents the second formant frequency F2 of the standard sample, and uF3 represents the third formant frequency F3 of the standard sample.

[0010] Preferably, S2 further includes S22; S22. The cloud-based speech processing server aggregates the deviation magnitude Dev of all frames to obtain a deviation magnitude set, and calculates the average deviation magnitude Dev of all frames, combining it with the maximum allowable deviation magnitude Dev calculated from the preset training corpus. max Generate a pronunciation stability score, StabScore; The pronunciation stability score StabScore is specifically calculated and output using the following algorithm formula: In the formula, N represents the total number of frames, and the pronunciation stability score StabScore ranges from 0 to 100. A preliminary assessment of pronunciation stability is conducted based on the output of the StabScore, which is used to determine the stability of pronunciation. The specific assessment content is as follows: When the pronunciation stability score StabScore is ≥90, it is considered stable pronunciation and no compensation is required; When the pronunciation stability score StabScore < 90, it is judged as unstable pronunciation, triggering the articulation place compensation drive.

[0011] Preferably, S3 includes S31; S31. When the pronunciation part compensation driver is triggered, the cloud speech processing server traverses all deviation amplitudes Dev in the deviation amplitude set and compares the deviation amplitudes of all frames according to the preset frame interval [1,N]. The index n* of the key frame with the most severe deviation is determined by the maximum deviation search calculation logic. The keyframe index n* is determined by the following maximum deviation search calculation logic; In the formula, argmax represents the value of the independent variable when it reaches its maximum value.

[0012] Preferably, S3 further includes S32; S32. After determining the keyframe index n* with the most severe deviation, the cloud-based speech processing server uses the deviation magnitude Dev(n*) of the n*th keyframe index in the deviation magnitude set and the maximum allowable deviation magnitude Dev max The tongue surface compensation control factor TongueAdjRate is constructed using a normalized ratio calculation method. The calculation method for TongueAdjRate is as follows: TongueAdjRate = Dev(n*) / Dev max .

[0013] Preferably, S4 includes S41; S41. Based on the range of the tongue surface compensation control factor TongueAdjRate, the control level is divided, and then the corresponding tongue surface control strategy is executed according to the control level division results. The specific control level division is as follows: When the tongue surface compensation control factor TongueAdjRate∈[0.00-0.60], the control level is divided into the first-level compensation level, and the first-level tongue surface control strategy is executed. The first-level tongue surface control strategy increases the tongue surface forward displacement distance by 10% compared to the original tongue surface displacement, increases the tongue tip upward angle by 5°, and extends the duration of tongue movement by 10%. When the tongue surface compensation control factor TongueAdjRate∈[0.61-0.80], the control level is divided into a second-level compensation level, and a second-level tongue surface control strategy is executed. The second-level tongue surface control strategy increases the tongue tip forward movement distance by 25% and the tongue tip upward angle by 15°. When the tongue surface compensation control factor TongueAdjRate∈[0.81-1.00], the control level is divided into three compensation levels, and a three-level tongue surface control strategy is executed. The three-level tongue surface control strategy increases the tongue face forward distance to 40% and the tongue tip upward angle to 25°.

[0014] Preferably, S4 further includes S42; S42. After executing the corresponding tongue control strategy, the cloud-based speech processing server uses the execution result of the tongue control strategy as the action control parameter and injects it into the oral cavity three-dimensional action driving node in the digital human generation module for digital fusion. The digital fusion process maps the tongue tip forward movement distance to the tongue body forward and backward movement control curve of the digital human model, maps the tongue tip upward angle parameter to the tongue tip bone rotation control node, synchronizes the action duration parameter to the animation time axis controller, and transmits the visual assistance mode parameter to the oral cavity rendering module. The digital human generation module drives the 3D digital human oral cavity animation system based on the motion control parameters after digital fusion, generating a sequence of dynamic animation frames of the tongue surface that are consistent with the time axis of the user's voice signal, so as to achieve a coordinated and consistent dynamic animation clip of the tongue with the synchronous output of the voice signal and the tongue movement.

[0015] A language model-based digital human generation system includes a speech processing module, a pronunciation stabilization analysis module, a compensation driving module, and a control execution module; The voice processing module collects the user's voice signal, preprocesses the collected voice signal to obtain the formant frequency information of each frame of voice, and transfers the formant frequency information to the voice data storage database. The pronunciation stability analysis module extracts resonance frequency information and standard samples from the speech storage database, performs deviation analysis to obtain the deviation magnitude Dev, constructs a pronunciation stability score StabScore, and then performs a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. The compensation driving module triggers articulation part compensation based on a preliminary articulation stability assessment, extracts and finds keyframe indices n* based on the deviation magnitude Dev, and compares them with the maximum allowable deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; The control execution module divides the control level according to the range of the tongue surface compensation control factor TongueAdjRate, executes the tongue surface control strategy based on the control level division result, and then digitally fuses the tongue surface control strategy execution result with the digital human generation module to drive the three-dimensional digital human oral cavity animation system and output a tongue dynamic animation clip with synchronized speech and tongue surface movements.

[0016] This invention provides a digital human generation system and method based on a language model. It has the following beneficial effects: (1) This method, through a sequential processing flow from S1 to S4, realizes a complete digital human pronunciation modeling path, from speech signal acquisition, formant frequency extraction, deviation analysis, pronunciation stability assessment to motion compensation control and 3D animation fusion. Compared with existing methods that drive digital human speech patterns solely based on energy envelope or mouth shape classification, this invention extracts and processes multi-frame formant frequencies of user speech, enabling the digital human to generate dynamic tongue surface and tongue tip movements based on vocal tract changes during the user's actual pronunciation process, thereby improving the physical correspondence and linguistic consistency of the digital human's pronunciation movements. This method achieves a direct mapping between pronunciation features and 3D tongue movements, significantly improving the realism of digital human pronunciation animation and its guiding value in teaching scenarios.

[0017] (2) This method, through the Dev calculation and StabScore construction mechanism proposed in S2, enables the system to automatically assess the stability of the user's pronunciation from multi-frame formant trajectories. By comparing the differences between the user's pronunciation and the standard sample on the F1, F2, and F3 vocal tract formants, the system can accurately identify the degree of deviation of the tongue height, front-back position of the tongue, and tongue tip movement during the user's pronunciation. When the pronunciation is unstable, this invention obtains the most representative deviation keyframe index n* through maximum deviation search in S3, and constructs the tongue surface compensation control factor TongueAdjRate accordingly, so that the tongue movement compensation corresponds proportionally to the strength of the user's pronunciation deviation, thereby achieving precise correction of the tongue front displacement distance, tongue tip angle, and movement duration. This mechanism solves the shortcomings of existing technologies that cannot detect the deviation of the pronunciation position and cannot make fine adjustments to the movement for local problems, enabling digital humans to reflect the real differences in the "articular force point" at the movement level.

[0018] (3) This method utilizes the hierarchical control strategy of the tongue surface compensation control factor TongueAdjRate proposed in S4 to make motion compensation controllable, progressive, and stable. Different intervals of the tongue surface compensation control factor TongueAdjRate correspond to different tongue motion strategies, so that the compensation amplitude is no longer fixed or simply linearly adjusted, but dynamically determined by the user's pronunciation deviation. In the digital fusion stage, by accurately mapping parameters such as the tongue forward movement distance, the tongue tip upward angle, and the duration of the action to the skeletal nodes, control curves, and rendering units of the digital human generation module, deep collaboration between tongue movements and oral cavity 3D animation is achieved. The digital human oral cavity animation clips generated in this way have the characteristics of high synchronization, smooth movements, accurate tongue position, and natural visual display, avoiding abnormal problems such as "unresponsive tongue tip" and "incoherent oral cavity movements" in existing technologies, and significantly improving the expressive ability of the digital human model in teaching, training, and interactive scenarios. Attached Figure Description

[0019] Figure 1This is a schematic diagram illustrating the steps of a language model-based digital human generation method according to the present invention; Figure 2 This is a schematic diagram of the process of a language model-based digital human generation system according to the present invention; Figure 3 This is a schematic diagram of the connection relationship of the cloud-based voice processing server according to the present invention; Figure 4 A schematic diagram of injecting 3D animation nodes into the tongue surface control parameters. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0021] Please see Figure 1 This invention provides a digital human generation method based on a language model. To achieve the above objectives, this invention employs the following technical solution, comprising the following steps: S1. Collect the user's voice signal, preprocess the collected voice signal to obtain the formant frequency information of each frame of voice, and transfer the formant frequency information to the voice data storage database. S2. Extract resonance frequency information and standard samples from the speech storage database, perform deviation analysis, obtain the deviation magnitude Dev, construct the pronunciation stability score StabScore, and then conduct a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. S3. Based on the preliminary articulation stability assessment, trigger the articulation part compensation drive, extract and find the keyframe index n* based on the deviation magnitude Dev, and compare it with the maximum allowable deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; S4. Based on the range of the Tongue Compensation Control Factor TongueAdjRate, the control level is divided, and the tongue control strategy is executed based on the control level division results. Then, the execution results of the tongue control strategy are digitally fused with the digital human generation module to drive the three-dimensional digital human oral animation system and output a tongue dynamic animation clip that is synchronized with the speech and tongue movements.

[0022] In this embodiment, the method performs frame-based preprocessing on the user's speech signal and extracts the formant frequency information of each frame, enabling the system to maintain real-time perception of tongue height, front-back position of the tongue, and tongue tip shape during speech processes with rapid changes in the vocal tract. Directly analyzing the entire speech segment would result in the loss of fine-grained tongue movements, particularly in rapidly transitioning syllables, leading to problems such as "tongue position lag" or "mouth shape lag." Therefore, extracting formants on a frame-by-frame basis is physically necessary. S2 establishes the deviation magnitude Dev based on the differences between the user and standard samples on the three formants, and then constructs the pronunciation stability score StabScore, enabling the distinction between "overall deviation" and "local instability." Without a frame-level difference measure of the deviation magnitude Dev, even if the user deviates significantly in some key syllables, it cannot be recognized, resulting in the digital human's oral cavity movements failing to accurately simulate key points of force application. The design of the pronunciation stability score StabScore ensures that unnecessary compensation is not generated when the overall pronunciation is stable, while timely intervention is possible when there is significant local deviation. S3 locates the keyframe index n* through maximum deviation search and calculates the tongue surface compensation control factor TongueAdjRate, establishing a direct mapping relationship between compensation behavior and deviation intensity. Without locating the keyframe index n*, the system can only perform average compensation across all frames, which would cause overall tongue movement distortion, such as elongation of a syllable or "excessive tongue tip elevation." By mapping Dev(n*) to the maximum allowable deviation magnitude Dev... max The Tongue AdjRate, a normalized calculation method, strictly limits compensation to a physically reasonable range, ensuring adjustments occur only at the most critical tongue movement points and preventing overcompensation. S4 employs tiered control via Tongue AdjRate to achieve both responsiveness and stability in tongue adjustment. For example, excessive compensation for slight deviations can result in abrupt movement jumps, while insufficient compensation for severe deviations fails to eliminate visual distortions caused by incorrect pronunciation. Through a tiered strategy with range boundaries, the tongue's forward movement distance, tip elevation angle, and duration can be steadily adjusted according to different degrees of deviation. These adjustments are ultimately mapped digitally to 3D skeletal drive nodes, ensuring precise synchronization of tongue movements with the speech timeline and preventing common animation anomalies such as "speech precedes mouth shape" or "no tongue tip response." In summary, the present invention drives tongue micro-movements through acoustic parameters, enabling digital humans to possess realistic physical logic in terms of articulation force, tongue position adjustment, and tongue tip movements. This not only avoids the unnatural effects of traditional digital mouth shapes due to the lack of tongue involvement, but also significantly improves the accuracy, synchronization, and visual expression capabilities of pronunciation movements in teaching, training, and complex speech presentation scenarios. Example 2

[0023] Please see Figure 1 and Figure 3 Specifically: S1 includes S11; S11. The voice signal emitted by the user in real time is collected by the microphone set in the digital human user terminal, and the voice signal is segmented into frames to obtain the framed voice signal. The framed voice signal is uploaded to the cloud voice processing server through the local communication interface. The cloud voice processing server includes a preset voice data storage database. The speech data storage database includes a speech signal storage table, a resonance data storage table, and a standard sample storage table; Among them, the voice signal storage table is used to store the voice signals acquired in real time; The resonance data storage table is used to store the resonance peak frequency information obtained after preprocessing in real time; The standard sample storage table is used to store user-defined teaching audio standard samples in real time, perform formant analysis processing, extract the formant frequency information of the standard samples, and then store it.

[0024] S1 also includes S12; S12. In the cloud-based speech processing server, extract the framed speech signal from the speech signal storage table of the speech data storage database, perform preprocessing, and perform formant analysis processing based on the preprocessed framed speech signal to extract the formant frequency information of each frame of speech. The formant frequency information includes the first formant frequency F1(n) of the nth frame, the second formant frequency F2(n) of the nth frame, and the third formant frequency F3(n) of the nth frame. Formant analysis processing involves performing linear prediction analysis on each framed speech signal. A linear prediction model describing the vocal tract filtering characteristics is established using linear prediction modeling technology. The Levinson-Durbin recursive solution algorithm is used to obtain the linear prediction coefficient sequence. Based on the linear prediction coefficients in the linear prediction coefficient sequence, the linear prediction spectrum envelope of the corresponding frame is constructed to reflect the vocal tract frequency response characteristics of the current speech frame. Local convex peak regions are located in the linear prediction spectrum envelope as formant candidate regions. Subsequently, pole solving is performed on the polynomial corresponding to the linear prediction model. By obtaining the conjugate complex roots of the poles and analyzing their angular frequencies, the angular frequencies are converted into actual frequency values. The first three frequencies are selected in ascending order as the first formant frequency F1, the second formant frequency F2, and the third formant frequency F3. Among them, the first formant frequency F1 reflects the height of the tongue position (the height of the tongue). The higher the first formant frequency F1, the lower the tongue position; the lower the first formant frequency F1, the higher the tongue position. For example, the first formant frequency F1 of the high vowel [i] is very low; the first formant frequency F1 of the low vowel [a] is very high. In digital human control, the first formant frequency F1 directly corresponds to the overall up-down movement adjustment of the tongue. The second formant frequency F2 reflects the front-to-back position of the tongue; the higher the F2, the further forward the tongue is; the lower the F2, the further back the tongue is. For example, the front vowel [i] has a very high F2; the back vowel [u] has a very low F2. In digital human control, F2 determines the front-to-back movement of the tongue (DistanceAnterior parameter). The third formant frequency F3 primarily reflects the shape of the tongue tip, the shape of the oral cavity opening, and the special characteristics of some consonants; common uses include: distinguishing between retroflex and non-retroflex consonants; detecting the positional relationship between the tongue tip and the alveolar ridge; simulating apical phonemes such as voiced alveolar approximant [r], voiceless postalveolar affricate [tʃ], and voiceless retroflex fricative [ʂ]; in digital human control: F3 determines the tip elevation angle. Preprocessing includes spectral subtraction noise reduction, pre-emphasis processing, frame-segmentation windowing, and linear prediction cepstral analysis. The spectral subtraction noise reduction process is achieved through frequency domain noise estimation and amplitude spectrum compensation techniques. The frequency domain amplitude spectrum is obtained by performing a fast Fourier transform on the framed speech signal. After noise suppression is performed by combining the noise power spectrum estimation algorithm, the time domain denoised speech frame is constructed using inverse fast Fourier transform. Pre-emphasis processing enhances the high-frequency components of the speech signal using a high-pass digital filter. By performing digital filtering on the speech sampling sequence, the spectral resolution of the subsequent formant frequency extraction stage is improved. Frame-by-frame windowing is based on time window slicing and window function smoothing techniques. By performing sliding time window segmentation on continuous speech signals and applying a preset window function, framed speech has good short-term stability and reduces spectral leakage. Linear predictive cepstral analysis is achieved using vocal tract linear predictive modeling and prediction parameter solving techniques. A linear predictive model is established on the framed speech signal and a recursive solution algorithm is used to obtain the linear predictive coefficients. The coefficients are then converted into cepstral parameters to characterize the vocal tract resonance characteristics.

[0025] In this embodiment, the method uses a microphone on the digital human user terminal to frame the user's speech signal, thereby segmenting the originally continuous and rapidly changing speech process into short-term, approximately stable framed speech signals. Directly processing the entire speech segment as a whole not only makes it difficult to accurately capture changes in tongue position at any given moment, but also introduces "temporal ambiguity" during subsequent formant analysis, leading to problems such as overall tongue movement drift and inaccurate articulation in the digital human. By framing the speech and then uploading it to a cloud-based speech processing server, where it is categorized and stored in a speech signal storage table, a formant data storage table, and a standard sample storage table, each frame of speech and its corresponding standard sample has a clear data attribute, providing a clear data foundation and retrieval path for subsequent frame-by-frame comparison and compensation control.

[0026] In cloud-based speech processing servers, spectral subtraction for noise reduction, pre-emphasis, frame-by-frame windowing, and linear prediction cepstral analysis are performed on framed speech signals to minimize interference from noise, spectral leakage, and high-frequency attenuation before formant analysis. If these preprocessing steps are omitted and linear prediction modeling is directly applied to the original speech, the resulting linear prediction spectral envelope will be heavily contaminated by background noise and energy fluctuations. This leads to increased jitter in the localized formant positions and shifts in F1 / F2 / F3 frequency values, ultimately causing the digital human to misalign key movements such as tongue height, front-back position, and tongue tip angle, resulting in a visually unnatural appearance of "the mouth is moving, but the tongue is not." The aforementioned preprocessing significantly improves the stability and resolution of formant frequency extraction, making the digital human's detailed articulation movements more closely resemble real pronunciation. Linear predictive analysis combined with pole finding is used to extract the first formant frequency F1, the second formant frequency F2, and the third formant frequency F3, and directly correlate them with tongue height, tongue position (front and back), and tongue tip shape. This is to ensure the system has a clear physical correspondence, rather than simply driving the "auditory signal" in a coarse manner. Without considering the vocal tract model and using only simple energy or pitch-driven methods, the digital human cannot accurately reflect the essential differences in tongue position and oral cavity structure, such as high vowels [i], low vowels [a], front vowels [i], and back vowels [u]. The differences between teacher demonstrations and student pronunciation in teaching scenarios cannot be visually amplified through the digital human's tongue movements. Through this parameter extraction method based on vocal tract resonance characteristics, this embodiment not only visually recreates the true movement logic of the tongue but also provides quantifiable, interpretable, and adjustable basic variables for subsequent deviation analysis, compensation control, and level classification, resulting in a substantial improvement in the digital human's expressive power in pronunciation details compared to traditional solutions. Example 3

[0027] Please see Figure 1 Specifically: S2 includes S21; S21, the cloud-based voice processing server reads user formant frequency information and standard sample formant frequency information from the resonance data storage table and standard sample storage table of the voice data storage database, respectively. Based on the three sets of formant frequency differences between the user and the standard sample at the same frame position, deviation analysis is performed to obtain the deviation magnitude Dev; this is used to measure the degree to which the formant position of this frame deviates from the standard sample from the perspective of a single frame. Deviation magnitude (Dev) is calculated using the following algorithm formula: In the formula, Dev(n) represents the deviation amplitude of the nth frame, uF1 represents the first formant frequency F1 of the standard sample, uF2 represents the second formant frequency F2 of the standard sample, and uF3 represents the third formant frequency F3 of the standard sample. This formula is essentially derived from the Euclidean distance concept in analytic geometry and vector distance measurement. In this formula, the three-dimensional coordinates are replaced with points in the three-dimensional formant frequency space, and the classic Euclidean distance concept is limited and mapped to the "third-order formant frequency space of speech" to measure the difference between the user's pronunciation trajectory and the standard trajectory; it maintains the monotonicity of the deviation (the square does not change the magnitude comparison relationship); and it provides a direct basis for the deviation amount for subsequent normalization and pronunciation stability scoring StabScore.

[0028] S2 also includes S22; S22. The cloud-based speech processing server aggregates the deviation magnitude Dev of all frames to obtain a deviation magnitude set, and calculates the average deviation magnitude Dev of all frames, combining it with the maximum allowable deviation magnitude Dev calculated from the preset training corpus. max Generate a pronunciation stability score, StabScore; The pronunciation stability score, StabScore, is calculated and output using the following algorithm: In the formula, N represents the total number of frames, and the pronunciation stability score StabScore ranges from 0 to 100. A preliminary assessment of pronunciation stability is conducted based on the output of the StabScore, which is used to determine the stability of pronunciation. The specific assessment content is as follows: When the pronunciation stability score StabScore is ≥90, it is considered stable pronunciation and no compensation is required; When the pronunciation stability score StabScore < 90, it is judged as unstable pronunciation, triggering the articulation place compensation drive.

[0029] In this embodiment, the method calculates the deviation magnitude Dev using the user formant frequency information and the standard sample formant frequency information based on the same frame position in S21. This is to establish a "distance scale" similar to a three-dimensional coordinate for each frame of pronunciation in the "speech third-order formant frequency space". The larger the deviation magnitude Dev, the further the actual resonance state of the user's vocal tract in this frame differs from the vocal tract resonance state of the standard sample, equivalent to the two points being farther apart in three-dimensional space. If this frame-by-frame deviation analysis is not performed, but only an overall average or only a single formant frequency is considered, a situation will occur where "the pronunciation looks similar overall, but a certain key syllable is obviously off-target but is ignored," which can easily lead to subsequent pronunciation compensation drivers not being able to find the pronunciation segment that truly needs correction. This implementation describes the pronunciation difference from a single-frame perspective by using the deviation magnitude Dev, so that the system not only knows "whether there is a problem", but also knows "which specific frame and which pronunciation segment the problem is in".

[0030] The deviation magnitude Dev of all frames is summarized and statistically analyzed in S22, and the maximum allowable deviation magnitude Dev is introduced. max Normalizing the deviation magnitude Dev into a pronunciation stability score StabScore essentially transforms "distance information" into a "determinable and comparable" quantitative indicator of pronunciation stability within the 0-100 range. When the pronunciation stability score StabScore ≥ 90, it indicates that the deviation magnitude Dev is relatively small in most frames, and the user's overall pronunciation trajectory clusters around the standard sample, requiring no tongue surface compensation. When the pronunciation stability score StabScore < 90, it indicates that the deviation magnitude Dev is statistically large. If the pronunciation part compensation driver is not triggered at this point, the problem of "the digital human's mouth shape is very standard, but the tongue movement never responds to the user's incorrect pronunciation" will occur. By setting a StabScore threshold for pronunciation stability, a clear mapping relationship is established between the frame-level deviation magnitude Dev and the decision logic for whether to initiate compensation. On the one hand, this avoids the over-amplification of slight normal differences (preventing overcompensation), and on the other hand, it can promptly capture obvious pronunciation abnormalities (avoiding missed compensation). This achieves a reasonable closed loop from "acoustic differences in pronunciation" to "tongue movement correction behavior" in a real physical sense, significantly improving the precision and reliability of pronunciation correction for digital humans. Example 4

[0031] Please see Figure 1 and Figure 4 Specifically: S3 includes S31; S31. When the pronunciation part compensation driver is triggered, the cloud speech processing server traverses all deviation amplitudes Dev in the deviation amplitude set and compares the deviation amplitudes of all frames according to the preset frame interval [1,N]. The index n* of the key frame with the most severe deviation is determined by the maximum deviation search calculation logic. The keyframe index n* is determined by the following maximum deviation search calculation logic; In the formula, argmax represents the value of the independent variable when it reaches its maximum value. The keyframe n* with the most severe deviation represents: the moment when the "problem is most serious" during pronunciation; the frame with the greatest deviation in tongue position, oral cavity opening, or tongue tip position; and the key reference for subsequent compensation. This frame determines how to adjust the digital human's movements, such as whether to raise the angle of the tongue tip, whether to push the tongue forward, and whether to slow down the oral cavity movements for that syllable.

[0032] S3 also includes S32; S32. After determining the keyframe index n* with the most severe deviation, the cloud-based speech processing server uses the deviation magnitude Dev(n*) of the n*th keyframe index in the deviation magnitude set and the maximum allowable deviation magnitude Dev max The tongue surface compensation control factor TongueAdjRate is constructed using a normalized ratio calculation method. The calculation method for TongueAdjRate is as follows: TongueAdjRate = Dev(n*) / Dev max Where, Dev(n*) is the deviation magnitude of the n*th keyframe index, representing the formant deviation of the frame with the most severe deviation, and the maximum allowable deviation magnitude Dev max The normalized baseline value is used to constrain deviations from the upper limit; the tongue surface compensation control factor TongueAdjRate ranges from 0 to 1 and is used to drive the corresponding compensation adjustment of the forward and backward movement distance of the digital human tongue, the angle of tongue tip elevation, and the duration of the movement.

[0033] In this embodiment, the method searches for the keyframe index n* with the most severe deviation in the deviation amplitude set in S31 to pinpoint the time point with the "biggest problem" in the entire pronunciation, rather than treating all frames equally. If the keyframe index n* with the most severe deviation is not identified separately, and compensation is uniformly applied to the entire speech, even relatively normal frames will be over-adjusted, which will not only disrupt the naturalness of the overall movement but also weaken the targeted correction of the actual pronunciation error. By searching for the maximum deviation, the frame "farthest from the standard sample" is used as the main reference frame for subsequent compensation, allowing for targeted action enhancement at the most critical articulation moment. In S32, the deviation amplitude Dev(n) of the nth keyframe index and the maximum allowable deviation amplitude Dev are used. maxThe Tongue AdjRate control factor essentially converts the "deviation distance" into a compensation intensity proportionality coefficient ranging from 0 to 1: a small deviation results in a Tongue AdjRate close to 0, requiring only slight tongue adjustments; a large deviation results in a Tongue AdjRate close to 1, triggering significant tongue forward movement, tongue tip elevation, or prolonged movement duration. This design ensures that the compensation amplitude is proportional to the actual degree of deviation, avoiding "over-processing minor issues," and also prevents insufficient compensation for serious pronunciation errors. It establishes a continuous mapping relationship from "acoustic deviation magnitude to tongue movement intensity" in a true physical sense, thereby significantly improving the precision, stability, and visual persuasiveness of digital human pronunciation correction. Example 5

[0034] Please see Figure 1 Specifically: S4 includes S41; S41. Based on the range of the tongue surface compensation control factor TongueAdjRate, the control level is divided, and then the corresponding tongue surface control strategy is executed according to the control level division results. The specific control level division is as follows: When the tongue surface compensation control factor TongueAdjRate∈[0.00-0.60], the control level is divided into the first-level compensation level, and the first-level tongue surface control strategy is executed. The first-level tongue surface control strategy increases the tongue surface forward displacement distance by 10% compared to the original tongue surface displacement, increases the tongue tip upward angle by 5°, and extends the duration of tongue movement by 10%. When the tongue surface compensation control factor TongueAdjRate∈[0.61-0.80], the control level is divided into a second-level compensation level, and a second-level tongue surface control strategy is executed. The second-level tongue surface control strategy increases the tongue tip forward movement distance by 25% and the tongue tip upward angle by 15°. When the tongue surface compensation control factor TongueAdjRate∈[0.81-1.00], the control level is divided into three compensation levels, and a three-level tongue surface control strategy is implemented. The three-level tongue surface control strategy increases the tongue tip forward movement distance to 40% and the tongue tip upward angle to 25°.

[0035] S4 also includes S42; S42. After executing the corresponding tongue control strategy, the cloud-based speech processing server uses the execution result of the tongue control strategy as the action control parameter and injects it into the oral cavity three-dimensional action driving node in the digital human generation module for digital fusion. The digital fusion process maps the tongue tip forward movement distance to the tongue body forward and backward movement control curve of the digital human model, maps the tongue tip upward angle parameter to the tongue tip bone rotation control node, and synchronizes the action duration parameter to the animation time axis controller and transmits the visual assistance mode parameter to the oral cavity rendering module. The digital human generation module drives the 3D digital human oral cavity animation system based on the motion control parameters after digital fusion, generating a sequence of dynamic animation frames of the tongue surface that are consistent with the time axis of the user's voice signal, so as to achieve a coordinated and consistent dynamic animation clip of the tongue with the synchronous output of the voice signal and the tongue movement.

[0036] In this embodiment, the method divides the tongue surface compensation control factor TongueAdjRate into different intervals and corresponding to different levels of tongue surface control strategies through S41. The core purpose is to transform the "size of acoustic deviation" into "gradation of action intensity," rather than making linear, fragmented, and difficult-to-control continuous adjustments each time. If gradation is not performed and the tongue surface compensation control factor TongueAdjRate is adjusted infinitely according to its proportion, two typical problems will occur in practice: First, even slight fluctuations will cause small vibrations of the tongue surface, making the image appear "shaky, trembling, and unstable"; second, compensation is difficult to adjust, and slightly increasing the coefficient will cause the tongue to "tap" too forcefully in some syllables. By mapping the tongue surface compensation control factors TongueAdjRate∈Slight compensation [0.00–0.60], moderate compensation [0.61–0.80], and large compensation [0.81–1.00] to Level 1, Level 2, and Level 3 compensation levels respectively, and providing clear displacement ratios (10%, 25%, 40%) and tongue tip angle elevations (5°, 15°, 25°), it can be ensured that: slight deviations are only slightly corrected, moderate deviations are significantly but not excessively amplified, and only severe deviations trigger large-scale tongue forward movement and tongue tip elevation, thus establishing a stable segmented response in a true physical sense where "the more deviation, the more movement." In S42, the execution results of the tongue surface control strategy are injected as parameters into the tongue forward and backward movement control curve, tongue tip bone rotation node, and animation timeline controller in the digital human generation module to ensure that the acoustic layer, control layer, and 3D animation layer are aligned on the same time reference. If the skeleton is adjusted only at the 3D model level without being uniformly integrated based on the tongue surface compensation control factor TongueAdjRate and the predetermined displacement ratio, it is easy to encounter phenomena such as "the sound has already passed before the tongue starts to move" or "the tongue tip angle adjustment is misaligned with the timing of pronunciation." Through this digital fusion method, the tongue front displacement distance, the tongue tip upward angle, and the duration of the movement are all strictly synchronized with the speech timeline. This avoids the common problems in traditional systems such as asynchrony between speech and oral cavity movements and abrupt tongue deformation. Furthermore, it visually transforms pronunciation deviations into clearly visible dynamic tongue cues, significantly improving the interpretability and credibility of digital human articulation movements in teaching, training, or demonstration scenarios. Example 6

[0037] Please see Figure 1 and Figure 2 A digital human generation system based on a language model includes a speech processing module, a pronunciation stabilization analysis module, a compensation driving module, and a control execution module; The voice processing module collects the user's voice signal, preprocesses the collected voice signal to obtain the formant frequency information of each frame of voice, and transfers the formant frequency information to the voice data storage database. The pronunciation stability analysis module extracts resonance frequency information and standard samples from the speech storage database, performs deviation analysis, obtains the deviation magnitude Dev, constructs a pronunciation stability score StabScore, and then conducts a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. The compensation-driven module triggers articulation place compensation based on a preliminary articulation stability assessment. It then extracts and finds keyframe indices n* based on the deviation magnitude Dev, and compares these indices with the maximum permissible deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; The control execution module divides the control level according to the range of the Tongue Compensation Control Factor TongueAdjRate, executes the tongue control strategy based on the control level division results, and then digitally fuses the execution results of the tongue control strategy with the digital human generation module to drive the three-dimensional digital human oral animation system and output a tongue dynamic animation clip with synchronized speech and tongue movements.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for generating digital humans based on a language model, characterized in that: Includes the following steps: S1. Collect the user's voice signal, preprocess the collected voice signal to obtain the formant frequency information of each frame of voice, and transfer the formant frequency information to the voice data storage database. S2. Extract resonance frequency information and standard samples from the speech storage database, perform deviation analysis, obtain the deviation magnitude Dev, construct the pronunciation stability score StabScore, and then conduct a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. S3. Based on the preliminary articulation stability assessment, trigger the articulation part compensation drive, extract and find the keyframe index n* based on the deviation magnitude Dev, and compare it with the maximum allowable deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; S4. Based on the range of the Tongue Compensation Control Factor TongueAdjRate, the control level is divided, and the tongue control strategy is executed based on the control level division results. Then, the execution results of the tongue control strategy are digitally fused with the digital human generation module to drive the three-dimensional digital human oral animation system and output a tongue dynamic animation clip that is synchronized with the speech and tongue movements.

2. The digital human generation method based on a language model according to claim 1, characterized in that: S1 includes S11; S11. The voice signal emitted by the user in real time is collected by the microphone set in the digital human user terminal, and the voice signal is segmented into frames to obtain the framed voice signal. The framed voice signal is uploaded to a cloud-based voice processing server via a local communication interface. The cloud-based voice processing server includes a preset voice data storage database. The speech data storage database includes a speech signal storage table, a resonance data storage table, and a standard sample storage table.

3. The digital human generation method based on a language model according to claim 2, characterized in that: S1 further includes S12; S12. In the cloud-based speech processing server, extract the framed speech signal from the speech signal storage table of the speech data storage database, perform preprocessing, and perform formant analysis processing based on the preprocessed framed speech signal to extract the formant frequency information of each frame of speech. The formant frequency information includes the first formant frequency F1(n) of the nth frame, the second formant frequency F2(n) of the nth frame, and the third formant frequency F3(n) of the nth frame. The preprocessing includes spectral subtraction noise reduction, pre-emphasis processing, frame-segmentation windowing, and linear prediction cepstral analysis.

4. The digital human generation method based on a language model according to claim 3, characterized in that: S2 includes S21; S21, the cloud-based voice processing server reads user formant frequency information and standard sample formant frequency information from the resonance data storage table and standard sample storage table of the voice data storage database, respectively. Based on the three sets of formant frequency differences between the user and the standard sample at the same frame position, deviation analysis is performed to obtain the deviation magnitude Dev. The deviation magnitude Dev is calculated and output using the following algorithm formula: In the formula, Dev(n) represents the deviation amplitude of the nth frame, uF1 represents the first formant frequency F1 of the standard sample, uF2 represents the second formant frequency F2 of the standard sample, and uF3 represents the third formant frequency F3 of the standard sample.

5. The digital human generation method based on a language model according to claim 4, characterized in that: S2 further includes S22; S22. The cloud-based speech processing server aggregates the deviation magnitude Dev of all frames to obtain a deviation magnitude set, and calculates the average deviation magnitude Dev of all frames, combining it with the maximum allowable deviation magnitude Dev calculated from the preset training corpus. max Generate a pronunciation stability score, StabScore; The pronunciation stability score StabScore is specifically calculated and output using the following algorithm formula: In the formula, N represents the total number of frames, and the pronunciation stability score StabScore ranges from 0 to 100. A preliminary assessment of pronunciation stability is conducted based on the output of the StabScore, which is used to determine the stability of pronunciation. The specific assessment content is as follows: When the pronunciation stability score StabScore is ≥90, it is considered stable pronunciation and no compensation is required; When the pronunciation stability score StabScore < 90, it is judged as unstable pronunciation, triggering the articulation place compensation drive.

6. The digital human generation method based on a language model according to claim 5, characterized in that: S3 includes S31; S31. When the pronunciation part compensation driver is triggered, the cloud speech processing server traverses all deviation amplitudes Dev in the deviation amplitude set and compares the deviation amplitudes of all frames according to the preset frame interval [1,N]. The index n* of the key frame with the most severe deviation is determined by the maximum deviation search calculation logic. The keyframe index n* is determined by the following maximum deviation search calculation logic; In the formula, argmax represents the value of the independent variable when it reaches its maximum value.

7. The method for generating a digital human based on a language model according to claim 6, characterized in that: S3 further includes S32; S32. After determining the keyframe index n* with the most severe deviation, the cloud-based speech processing server uses the deviation magnitude Dev(n*) of the n*th keyframe index in the deviation magnitude set and the maximum allowable deviation magnitude Dev max The tongue surface compensation control factor TongueAdjRate is constructed using a normalized ratio calculation method. The calculation method for TongueAdjRate is as follows: TongueAdjRate = Dev(n*) / Dev max .

8. The digital human generation method based on a language model according to claim 7, characterized in that: S4 includes S41; S41. Based on the range of the tongue surface compensation control factor TongueAdjRate, the control level is divided, and then the corresponding tongue surface control strategy is executed according to the control level division results. The specific control level division is as follows: When the tongue surface compensation control factor TongueAdjRate∈[0.00-0.60], the control level is divided into the first-level compensation level, and the first-level tongue surface control strategy is executed. The first-level tongue surface control strategy increases the tongue surface forward displacement distance by 10% compared to the original tongue surface displacement, increases the tongue tip upward angle by 5°, and extends the duration of tongue movement by 10%. When the tongue surface compensation control factor TongueAdjRate∈[0.61-0.80], the control level is divided into a second-level compensation level, and a second-level tongue surface control strategy is executed. The second-level tongue surface control strategy increases the tongue tip forward movement distance by 25% and the tongue tip upward angle by 15°. When the tongue surface compensation control factor TongueAdjRate∈[0.81-1.00], the control level is divided into three compensation levels, and a three-level tongue surface control strategy is executed. The three-level tongue surface control strategy increases the tongue face forward distance to 40% and the tongue tip upward angle to 25°.

9. The method for generating a digital human based on a language model according to claim 8, characterized in that: S4 also includes S42; S42. After executing the corresponding tongue control strategy, the cloud-based speech processing server uses the execution result of the tongue control strategy as the action control parameter and injects it into the oral cavity three-dimensional action driving node in the digital human generation module for digital fusion. The digital fusion process maps the tongue tip forward movement distance to the tongue body forward and backward movement control curve of the digital human model, maps the tongue tip upward angle parameter to the tongue tip bone rotation control node, synchronizes the action duration parameter to the animation time axis controller, and transmits the visual assistance mode parameter to the oral cavity rendering module. The digital human generation module drives the 3D digital human oral cavity animation system based on the motion control parameters after digital fusion, generating a sequence of dynamic animation frames of the tongue surface that are consistent with the time axis of the user's voice signal, so as to achieve a coordinated and consistent dynamic animation clip of the tongue with the synchronous output of the voice signal and the tongue movement.

10. A language model-based digital human generation system, applied to the language model-based digital human generation method according to any one of claims 1-9, characterized in that: It includes a speech processing module, a pronunciation stabilization analysis module, a compensation driving module, and a control execution module; The voice processing module collects the user's voice signal, preprocesses the collected voice signal to obtain the formant frequency information of each frame of voice, and transfers the formant frequency information to the voice data storage database. The pronunciation stability analysis module extracts resonance frequency information and standard samples from the speech storage database, performs deviation analysis to obtain the deviation magnitude Dev, constructs a pronunciation stability score StabScore, and then performs a preliminary pronunciation stability assessment based on the pronunciation stability score StabScore to determine the stability of pronunciation. The compensation driving module triggers articulation part compensation based on a preliminary articulation stability assessment, extracts and finds keyframe indices n* based on the deviation magnitude Dev, and compares them with the maximum allowable deviation magnitude Dev. max Calculations were performed to obtain the tongue surface compensation control factor TongueAdjRate; The control execution module divides the control level according to the range of the tongue surface compensation control factor TongueAdjRate, executes the tongue surface control strategy based on the control level division result, and then digitally fuses the tongue surface control strategy execution result with the digital human generation module to drive the three-dimensional digital human oral cavity animation system and output a tongue dynamic animation clip with synchronized speech and tongue surface movements.

Images