Digital human voice synthesis and lip synchronization method and system

Abstract

The application discloses a digital human voice synthesis and lip synchronization method and system, relates to the technical field of voice synthesis, and comprises the following steps: collecting streaming text and constructing a pre-reading window, identifying word boundaries to generate candidate segmentation positions; combining prosodic break tendency to determine pronounceable boundaries; performing voice synthesis based on the boundaries, extracting phoneme duration and fundamental frequency to form prosodic states and cross-segment transmission; determining the cross-segment phoneme time corresponding relationship, setting the cooperative pronunciation transition interval to generate continuous lip shapes; and establishing rhythm mapping to drive lip shape changes under a unified time framework. Through the pre-reading window and prosodic state transmission mechanism, the application ensures that the text segmentation is consistent with the pronunciation boundary, guarantees the continuity of cross-segment speech speed and energy trend, and improves the coherence of the voice; meanwhile, through cooperative pronunciation transition and time mapping processing, the application realizes the synchronization of lip rhythm and phonemes, eliminates cross-segment lip break, and improves the naturalness of digital human performance.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, specifically to a method and system for digital human speech synthesis and lip-syncing. Background Technology

[0002] Digital human speech synthesis and lip-sync refers to the process in real-time digital human interaction scenarios where input text is streamed and segmented using a large language model, then synthesized by a text-to-speech module, simultaneously outputting phoneme duration or audio energy information. The edge device constructs lip-sync rhythm based on phoneme-level alignment results and drives the digital human's lip movements to synchronize with speech during the rendering phase. This process includes not only the acoustic conversion from text to speech but also prosodic parameter generation, phoneme boundary alignment, and lip-sync parameter mapping. Its core lies in establishing a temporal consistency between text, speech signals, and visual lip movements, ensuring that speech rhythm, phoneme structure, and lip-sync advance synchronously on the timeline. This achieves low-latency, continuous, and natural speech playback and facial movement presentation effects in streaming dialogues.

[0003] The existing technology has the following shortcomings:

[0004] In the process of digital human speech synthesis and lip-syncing, the main problem lies in the inconsistency between the streaming segmentation mechanism and the actual pronunciation boundaries. Specifically, large language models typically segment text according to punctuation or semantic phrases and push each segment to the text-to-speech module. The text-to-speech module, however, synthesizes each text segment independently and resets the prosodic state at the end of each segment. Due to the phenomena of connected speech, neutral tone, retroflexion, and polyphonic characters relying on contextual disambiguation in Chinese expression, when the preceding segment lacks contextual support from the following segment, unnatural pauses or prosodic resets are prone to occur at the end of the segment, causing phoneme duration to shift at the boundaries and a sudden drop in energy envelope. The mouth opening and closing rhythm inferred from the above information on the edge device exhibits visual breaks at the segment boundaries, such as premature closing and reopening of the mouth. The proportion of cross-segment silence increases abnormally, and the lip opening degree shows a non-semantic decline and then rise, thus disrupting the continuity and naturalness between speech and lip shape.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a method and system for digital human speech synthesis and lip-syncing to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, this application provides the following technical solution: a method for digital human speech synthesis and lip-syncing, comprising the following steps:

[0008] Collect text fragments and their corresponding arrival time information from the real-time output of the large language model. Construct a pre-reading window while continuously receiving text fragments. Perform joint analysis on the current text fragment and several subsequent characters to identify word boundaries and grammatical positions that can be broken into sentences, and generate a set of candidate segmentation positions.

[0009] Based on the candidate segmentation position set and combined with the text-to-speech front-end prediction information on prosodic breakage trend, the connection tendency and pause tendency of each candidate segmentation position are determined. The segmentation position is confirmed when the connection preservation condition or clear pause condition is met and the minimum synthesizable length is reached, the articulated boundary is obtained, and the text segment is segmented according to the articulated boundary.

[0010] Speech synthesis is performed on segmented text fragments using the articulated boundaries. After each speech synthesis is completed, the corresponding phoneme duration distribution data and fundamental frequency change trend data are extracted to form the current prosodic state. The current prosodic state is transmitted synchronously when processing the next text fragment to maintain the continuity of speech rate and energy trends across text fragments.

[0011] By combining the current prosodic state and the phoneme alignment information generated by speech synthesis, the temporal correspondence between the end phoneme of the previous text segment and the beginning phoneme of the next text segment is determined. A coarse articulation transition interval is set at the position corresponding to the articulated boundary. The lip opening and closing parameters are subjected to temporal remapping and smoothing mixing to obtain a continuous mouth shape transition result.

[0012] Based on the continuous lip-shape transition results and the continuously updated current prosodic state, a lip-shape opening and closing rhythm mapping relationship is established under a unified time progression framework. During speech playback, lip shape changes are driven according to the lip-shape opening and closing rhythm mapping relationship to achieve synchronous advancement of phoneme rhythm and lip-shape movement.

[0013] Preferably, obtaining the candidate segmentation position set includes the following steps:

[0014] It continuously receives text segments output by the large language model in a streaming manner, records the arrival time information corresponding to each text segment, appends the text segments to the continuous text cache sequence in character order, and adds an arrival time mark to each character. It keeps the text segments in the state of waiting to be analyzed before the segmentation boundary is determined, and forms a pre-read window based on the continuous text cache sequence.

[0015] Within the pre-read window, semantic parsing and grammatical structure analysis are performed on continuous text cache sequences to identify word boundaries and grammatical positions where sentences can be broken, and a correspondence is formed by combining character order and arrival time information;

[0016] The identified word boundaries and grammatical positions that can be broken down are organized, converted into character indices, and associated with arrival time information to form a set of candidate segmentation positions containing character indices and time stamps;

[0017] During the continuous reception of text segments, the pre-read window is dynamically adjusted, and the word boundaries and grammatical positions within the influence range of newly added characters are updated. The candidate segmentation position set is updated synchronously to serve as the segmentation basis for subsequent pronunciation continuity boundary determination.

[0018] Preferably, during the dynamic adjustment of the pre-read window, the character order and arrival time information in the continuous text cache sequence are used together as the basis for updating the candidate segmentation position set. The word boundaries and grammatical positions that can be broken into sentences due to semantic structure changes caused by the addition of new characters are re-identified, and the updated candidate segmentation position set is continuously used as the basis for segmentation to participate in the pronunciation continuity boundary determination.

[0019] Preferably, obtaining the phonable boundaries and segmenting the text fragments based on the phonable boundaries includes the following steps:

[0020] The candidate segmentation position set is arranged in character index order. For each candidate segmentation position, the continuous characters before the candidate segmentation position are extracted to form the text interval to be pronounced, and the adjacent characters are extracted to form the context continuation interval. The fundamental frequency change trend information, phoneme duration prediction information, and sentence ending intonation information are obtained by combining the text-to-speech front-end prediction information of prosodic breakage trend, and the corresponding relationship is established.

[0021] Based on the established correspondence, the candidate segmentation positions are judged for their tendency to connect and pause, the combination relationship between the preceding and following words and the semantic closure state are analyzed, and the candidate positions that meet the conditions for maintaining the connection or the conditions for a clear pause are determined in combination with the minimum possible synthesizable length.

[0022] From the candidate positions that meet the conditions for maintaining connected speech or clear pauses and reach the minimum synthesizable length, select the corresponding positions in character order as the articulated boundaries, and record the character index, arrival time information and predicted fundamental frequency trend;

[0023] Based on the pronunciation boundaries, the corresponding character intervals are extracted to form segmented text fragments. At the same time, the remaining characters are retained to generate a new set of candidate segmentation positions and continue to perform the processing of connected speech tendency determination and pause tendency determination.

[0024] Preferably, in the process of determining the tendency of connected speech and the tendency of pause, the fundamental frequency change trend information corresponding to the candidate segmentation position is correlated with the phoneme duration prediction information. When the fundamental frequency change shows a convergence trend and the duration of the last phoneme shows an extension trend and satisfies the semantic closure state, it is confirmed that the explicit pause condition is met; when the fundamental frequency change shows a continuous trend and the preceding and following words form a continuous expression relationship, it is confirmed that the connected speech maintenance condition is met.

[0025] Preferably, the process of extracting phoneme duration distribution data and fundamental frequency change trend data and forming the current prosodic state at each completion of speech synthesis includes the following steps:

[0026] Based on the phonable boundaries, the segmented text fragments are obtained and speech synthesis processing is performed to generate audio waveform data. At the same time, the phoneme sequence, phoneme time start point and time end point corresponding to the audio waveform data are output to form phoneme duration distribution data. The fundamental frequency value of the audio waveform on the time axis is extracted frame by frame to form fundamental frequency change trend data.

[0027] The phoneme duration distribution data and fundamental frequency change trend data are organized, the average phoneme duration and the last phoneme duration are statistically analyzed, the fundamental frequency change direction, the last fundamental frequency value and the last fundamental frequency change slope are extracted, and the current prosodic state is formed by combining the end time point of the segmented text segment.

[0028] When processing the next text segment, the current prosodic state is input into the speech synthesis processing flow, the duration of the last phoneme is used as the reference value of the duration of the starting phoneme, and the value of the last fundamental frequency and the slope of change are used as the reference for the generation of the starting fundamental frequency, so as to maintain the continuity of speech rate trend and energy trend between text segments.

[0029] Preferably, the current prosodic state includes the duration of the last phoneme in the phoneme duration distribution data and the value and slope of the last fundamental frequency in the fundamental frequency change trend data. When processing the next text segment, the duration of the last phoneme is used to limit the generation range of the duration of the starting phoneme, and the value and slope of the last fundamental frequency change are used to limit the generation direction of the starting fundamental frequency, thereby achieving a continuous connection of speech rate trend and energy trend between text segments.

[0030] Preferably, obtaining continuous mouth shape transition results includes the following steps:

[0031] Based on the current prosodic state and the phoneme alignment information generated by speech synthesis, the time interval of the last phoneme of the previous text segment and the time interval of the first phoneme of the next text segment are extracted. The time sequence of the last phoneme and the time sequence of the first phoneme are formed in chronological order. The time and prosodic joint connection record is established by combining the duration of the last phoneme and the slope of the last fundamental frequency change.

[0032] Based on the joint connection record of time and prosody, the time correspondence between the last phoneme of the previous text segment and the first phoneme of the next text segment is determined, and the time interval is adjusted according to the duration of the last phoneme to form a continuous time mapping relationship.

[0033] Based on the time correspondence, a co-pronunciation transition interval is set at the position corresponding to the articulated boundary, and the length of the co-pronunciation transition interval is determined according to the duration of the final phoneme;

[0034] Within the cophonic transition interval, extract the lip opening / closing parameter value sequence of the last phoneme of the previous text segment and the first phoneme of the next text segment. Perform time remapping and smoothing mixing on the lip opening / closing parameters to form the lip opening / closing parameter sequence.

[0035] By concatenating the lip opening and closing parameter sequence with the lip opening and closing parameter sequences of the previous and next text segments in chronological order, a continuous mouth shape transition result corresponding to the phoneme alignment information is obtained.

[0036] Preferably, establishing a mouth shape opening and closing rhythm mapping relationship based on continuous mouth shape transition results and continuously updated current prosodic state includes the following steps:

[0037] The audio playback timeline of the speech synthesis output is set as the reference timeline. The timestamps of the lip opening and closing parameters in the continuous mouth shape transition results are mapped to the audio playback timeline. The phoneme duration distribution data and fundamental frequency change trend data in the current prosodic state are marked on the same timeline to form a unified time progression framework.

[0038] Within a unified time-progression framework, phoneme rhythm intervals are divided based on phoneme alignment information. The lip opening and closing parameter change curves of the intervals corresponding to the continuous mouth shape transition results are read, and a mouth shape opening and closing rhythm mapping relationship is established in combination with the fundamental frequency change direction in the current prosodic state.

[0039] During voice playback, the lip opening and closing parameter values ​​corresponding to the lip opening and closing rhythm mapping relationship are read step by step according to a unified time progression framework, and the values ​​are switched and updated according to the phoneme rhythm interval to achieve synchronous advancement of phoneme rhythm and lip movement.

[0040] A digital human speech synthesis and lip-sync system includes a streaming pre-reading analysis module, a pronunciation boundary determination module, a prosodic state generation module, a cross-segment lip-sync transition module, and a rhythm mapping driving module;

[0041] The streaming pre-reading analysis module collects text fragments and their corresponding arrival time information from the real-time output of the large language model. During the continuous reception of text fragments, it constructs a pre-reading window, performs joint analysis on the current text fragment and several subsequent characters, identifies word boundaries and grammatical positions that can be broken down, and generates a set of candidate segmentation positions.

[0042] The pronunciation boundary determination module, based on the candidate segmentation position set and combined with the text-to-speech front-end prediction information on prosodic breakage trend, determines the tendency of connected speech and the tendency of pauses for each candidate segmentation position. Under the condition of meeting the condition of maintaining connected speech or the condition of clear pause and reaching the minimum synthesizable length, the segmentation position is confirmed and the pronunciation boundary is obtained.

[0043] The prosodic state generation module performs speech synthesis processing on the segmented text fragments using the articulated boundaries. After each speech synthesis is completed, it extracts the corresponding phoneme duration distribution data and fundamental frequency change trend data to form the current prosodic state, and transmits the current prosodic state synchronously when processing the next text fragment.

[0044] The cross-segment lip-shape transition module combines the current prosodic state and the phoneme alignment information generated by speech synthesis to determine the time correspondence between the end phoneme of the previous text segment and the beginning phoneme of the next text segment. It sets a co-pronunciation transition interval at the position corresponding to the articulated boundary and performs time remapping and smoothing mixing on the lip opening and closing parameters to obtain continuous lip-shape transition results.

[0045] The rhythm mapping driven module establishes a mouth shape opening and closing rhythm mapping relationship under a unified time progression framework based on the continuous mouth shape transition results and the continuously updated current prosodic state. During speech playback, it drives lip shape changes according to the mouth shape opening and closing rhythm mapping relationship.

[0046] The technical effects and advantages provided by this application in the above technical solution are as follows:

[0047] This application constructs a pre-reading window and introduces a prosodic break trend prediction mechanism to pre-control text segmentation boundaries, ensuring that the text segmentation results are consistent with the actual pronunciation boundaries, thereby reducing the phenomenon of end-of-segment prosodic reset and abrupt intonation changes from the source. The current prosodic state is transmitted during the speech synthesis stage, maintaining the continuity of speech rate and energy trends across text segments. This results in natural speech rhythm transitions during streaming playback, avoiding non-semantic pauses and abrupt pronunciation jumps between segments, and improving the overall coherence and naturalness of the speech output.

[0048] This application establishes a co-pronunciation transition interval at the articulated boundary and performs time remapping and smoothing mixing on the lip opening and closing parameters. Simultaneously, it establishes a lip opening and closing rhythm mapping relationship within a unified time progression framework, ensuring that phoneme rhythm and lip movements advance synchronously on the time axis. Through the synergistic effect of continuous lip transition results and continuously updated current prosodic state, it effectively eliminates premature lip closing and reopening phenomena at cross-text segments, resulting in a continuous and smooth visual effect for digital human facial movements and enhancing the realism and stability of speech and lip-sync performance. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This is a flowchart of the method in this application.

[0051] Figure 2 A flowchart for obtaining the articulated boundaries for this application and segmenting text fragments based on the articulated boundaries.

[0052] Figure 3 This is a flowchart for obtaining continuous mouth shape transition results for this application.

[0053] Figure 4 This is a schematic diagram of the modules in this application. Detailed Implementation

[0054] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0055] This application provides, as follows: Figures 1 to 3 The digital human speech synthesis and lip-syncing method shown includes the following steps:

[0056] Step 1: Collect the text fragments and corresponding arrival time information output in real time by the large language model. During the continuous reception of text fragments, construct a pre-reading window, perform joint analysis on the current text fragment and several subsequent characters, identify word boundaries and grammatical positions that can be broken into sentences, and generate a set of candidate segmentation positions.

[0057] The specific steps to obtain the candidate segmentation position set are as follows:

[0058] When the digital human performs real-time voice broadcasting, it continuously receives text segments output by a large language model in a streaming manner, and synchronously records the specific arrival time information of each text segment in the processing flow. Whenever a new text segment arrives, the corresponding text segment is appended to the end of the existing continuous text buffer sequence in character order, and the corresponding arrival time information is appended to each newly added character to form a character-level time mapping relationship. Before the segmentation boundary of the text segment is determined, the text segment remains in a state of pending analysis and is not invoked for speech synthesis.

[0059] As multiple text fragments arrive consecutively, a continuous character stream is formed within the cache sequence. Based on the current total character length, the most recent possible semantic closure position, and the most recent arrival time interval, a pre-read window is dynamically defined, covering the current text fragment and several subsequent characters. The left boundary of the pre-read window is always positioned at the earliest character position before segmentation, while the right boundary is positioned at the latest arriving character position or several characters before it. This ensures that the current text fragment receives contextual support from several subsequent characters during analysis. In this process, the character order and arrival time information form a dual index relationship, providing two parallel reference clues—semantic order and temporal order—for subsequent joint analysis.

[0060] It should be noted that:

[0061] Large language models are natural language processing models trained on large-scale corpora. They can generate continuous text content based on the input context and have been widely used in scenarios such as dialogue generation, text completion, and semantic understanding. In existing technologies, large language models typically adopt an autoregressive generation approach, progressively outputting text fragments in a streaming manner. During the generation process, word prediction and sentence structure construction are performed based on contextual semantic relationships, thereby achieving real-time text generation capabilities.

[0062] Within the established pre-reading window, continuous semantic analysis and grammatical structure analysis are performed on the current text fragment and several subsequent characters to identify word boundaries and grammatical positions where sentences can be broken. In practice, according to the rules of Chinese word formation, continuous characters are matched and identified according to two-character words, three-character words, fixed collocations, and common function word combinations, and character intervals that may form complete words are marked one by one. For verb-object combinations, adjective-noun combinations, and quantifier structures with connected reading relationships, the overall identification is performed based on their grammatical roles in the sentence to avoid misclassifying internal characters as independent boundaries.

[0063] While identifying word boundaries, the sentence's subject-predicate structure, verb-object structure, parallel structure, and adversative connection structure are analyzed segment by segment. When a pause corresponding to a punctuation mark, the end of a modal particle, or the end of a complete declarative sentence structure is found, it is recorded as a grammatical position that can be broken down. When there are incomplete parallel or adversative connections within the sentence, they are not marked as grammatical positions that can be broken down. The above identification process proceeds character by character within the entire pre-reading window. Whenever a few new subsequent characters are added, the semantic and grammatical relationships of the affected character intervals are reconstructed to ensure that word boundaries and grammatical positions that can be broken down always reflect the current complete context.

[0064] This process allows us to obtain a set of word boundaries and grammatical positions that conform to the continuity of language expression, without relying on the original segmentation results of a large language model.

[0065] After identifying word boundaries and grammatical breakable positions, all word boundaries and grammatical breakable positions are uniformly organized to generate a set of candidate segmentation positions. In practice, each word boundary position and grammatical breakable position is converted into a specific character index according to the order of characters in the continuous text cache sequence. The character index is then associated with the arrival time information of the corresponding character to form a candidate segmentation position entry containing the character index and time stamp.

[0066] Character positions that simultaneously satisfy the conditions of the end of a complete semantic unit and the absence of subsequent characters forming a new complete grammatical structure are classified as priority candidate segmentation positions; character positions that only satisfy word boundaries but whose subsequent semantics are not yet closed are retained as ordinary candidate segmentation positions. All candidate segmentation positions are then arranged in character index order, and their relative positions within the pre-read window are recorded.

[0067] For characters located at the end of the current text segment where the following characters are insufficient to form a complete grammatical unit, their positions are retained in the candidate segmentation position set but not immediately output. This allows for further joint analysis once more subsequent characters arrive. Through this process, the candidate segmentation position set no longer relies on a single punctuation mark but instead combines word boundaries and grammatical positions that allow for sentence segmentation to form a multi-candidate set.

[0068] After generating the candidate segmentation position set, the candidate segmentation position set is continuously synchronized with the dynamic update process of the pre-read window. When a new text fragment arrives and is appended to the continuous text cache sequence, the right boundary of the pre-read window is readjusted, and word boundaries and sentence breakable grammatical positions within the influence range of the newly added characters are re-identified, while the candidate segmentation position set is updated.

[0069] For candidate segmentation positions that have stabilized due to semantic closure, the position is maintained in the candidate segmentation position set if the characters following the segmentation position no longer change the grammatical attributes. For candidate segmentation positions whose grammatical structure is altered by adding subsequent characters, the candidate attributes are readjusted according to the new semantic structure. Through this continuous update method, the candidate segmentation position set remains consistent with the currently available text content, while simultaneously retaining both character index and arrival time information. This provides a unified input source with both semantic and temporal continuity for subsequent boundary determination based on pronunciation continuity.

[0070] Throughout the process, the text segments did not directly enter the speech synthesis stage according to the original output boundaries of the large language model. Instead, the candidate segmentation position set was used as the only segmentation basis that could be used for subsequent processing. The correspondence between the streaming segmentation and the actual pronunciation boundary was adjusted from the source, creating continuous contextual conditions for the confirmation of the pronunciation boundary in the subsequent speech synthesis stage.

[0071] Step 2: Based on the candidate segmentation position set and combined with the prediction information of prosodic breakage trend from text to speech front end, the connection tendency and pause tendency of each candidate segmentation position are determined. The segmentation position is confirmed when the connection maintenance condition or the clear pause condition is met and the minimum synthesizable length is reached, the articulated boundary is obtained, and the text segment is segmented according to the articulated boundary.

[0072] The steps for obtaining the articulated boundaries and segmenting the text fragment based on these boundaries are as follows:

[0073] After the candidate segmentation position set is generated, each candidate segmentation position is arranged according to the index order of the characters in the continuous text sequence. For each candidate segmentation position, all consecutive characters before the candidate segmentation position are extracted as the text interval to be pronounced. At the same time, at least three to five characters immediately after the candidate segmentation position are extracted as the context continuation interval.

[0074] For each text segment to be pronounced, the text-to-speech front-end performs pronunciation trend prediction processing on the segment, obtaining corresponding fundamental frequency change trend information, phoneme duration prediction information, and sentence-end intonation trend information. This prediction information is then mapped one-to-one with the character positions of the candidate segmentation locations. Specifically, if the last word before the candidate segmentation location shows a gradually decreasing fundamental frequency, prolonged final phoneme duration, and a complete intonation convergence in the prediction information, and the contextual continuation interval does not show subsequent semantic extension, then the corresponding candidate segmentation location is recorded as having a pause tendency. If the prediction information shows that the fundamental frequency has not yet completed its descent, the final phoneme duration remains at a normal length, the intonation trend is still rising or steadily continuing, and the characters in the contextual continuation interval form verb-object or adjective-noun combinations with the preceding text, then the corresponding candidate segmentation location is recorded as having a liaison tendency. Through this one-to-one mapping process, a specific correspondence is established between the set of candidate segmentation locations and the text-to-speech front-end's prediction information on prosodic breakage trends.

[0075] After establishing the above correspondence, each candidate segmentation position undergoes refined processing to determine its tendency to connect words and pauses. For candidate segmentation positions already recorded as having a tendency to connect words, further analysis is conducted to determine whether the words before and after the candidate segmentation position belong to fixed collocations, such as structures like "to process," "to achieve control," or "to complete the operation," or whether they belong to combinations of quantifiers and nouns, such as expressions like "three steps" or "two methods." If they belong to the above continuous expression forms, the candidate segmentation position is marked as meeting the condition for maintaining connecting words, and the candidate segmentation position is not confirmed as the final segmentation position.

[0076] For candidate segmentation positions already recorded as having a tendency to pause, further analysis is conducted to determine whether the text content preceding the candidate segmentation position has formed a complete declarative sentence, a complete interrogative sentence, or a complete imperative sentence, and whether there is no longer a grammatical dependency relationship between the end and subsequent characters. If the above semantic closure conditions are met, the candidate segmentation position is marked as meeting the explicit pause condition. Simultaneously, the total number of characters preceding the candidate segmentation position is counted. When the total number of characters is less than the preset minimum synthesizable length, even if the explicit pause condition is met, the corresponding candidate segmentation position is marked as temporarily unconfirmed, and the text fragment continues to remain in the continuous text cache sequence awaiting the addition of subsequent characters. When the total number of characters is greater than or equal to the minimum synthesizable length, the corresponding candidate segmentation position is included in the confirmable range. Through this comprehensive processing method that combines the conditions for maintaining connected speech, the explicit pause condition, and the minimum synthesizable length, the determination of connected speech tendency and pause tendency is specifically implemented at each candidate segmentation position.

[0077] After determining the tendency for connected speech and pauses at all candidate segmentation positions, the first candidate segmentation position that simultaneously meets the conditions for maintaining connected speech or clear pauses and reaches the minimum synthesizable length is selected from the sorted order and confirmed as the current articulated boundary. During the confirmation process, the character index corresponding to the candidate segmentation position, the arrival time information corresponding to the candidate segmentation position, and the corresponding predicted fundamental frequency trend are recorded as the attributes of the current articulated boundary. The continuous character interval before the candidate segmentation position is separated from the continuous text sequence as a complete segmented text fragment.

[0078] Candidate segmentation positions following the corresponding candidate segmentation position are retained in the candidate segmentation position set to participate in the new judgment process in the next round. When no candidate segmentation position that simultaneously meets the above conditions is found in the current continuous text sequence, the text fragments continue to accumulate. After new characters are added, the processes for determining liaison and pause tendencies are re-executed until a new articulated boundary is formed. Through this sequential selection method, the articulated boundary is always based on the combined effect of the specific candidate segmentation position set and the predicted information of the specific prosodic breakage trend.

[0079] After obtaining the articulated boundaries, the text segments are segmented according to these boundaries. Specifically, all characters in the continuous text sequence from the earliest unsegmented character position to the character position corresponding to the current articulated boundary are extracted into an independent text segment. This independent text segment is then marked as a pronunciation unit that can enter the speech synthesis stage. Simultaneously, the starting position of the continuous text buffer sequence is updated, and characters after the articulated boundaries are retained in the continuous text buffer sequence for subsequent processing.

[0080] As new characters continue to arrive, a new set of candidate segmentation positions is regenerated, and combined with the prediction information of prosodic breakage trend from the text-to-speech front end, the joint reading tendency determination and pause tendency determination are performed again to form a new articulated boundary, thereby achieving continuous segmentation of text segments.

[0081] Through the specific implementation of the above steps, the determination of the candidate segmentation position set can be completed dynamically during the arrival of the streaming text. The candidate segmentation position is confirmed when the conditions for maintaining continuous reading or clear pause conditions are met and the minimum synthesizable length is reached, thus obtaining the articulated boundary. The text segment is then segmented based on the articulated boundary, thereby establishing control logic consistent with the continuity of pronunciation during the segmentation stage.

[0082] Step 3: Use the articulated boundaries to perform speech synthesis processing on the segmented text segments. After each speech synthesis is completed, extract the corresponding phoneme duration distribution and fundamental frequency change trend to form the current prosodic state. When processing the next text segment, the current prosodic state is transmitted synchronously to maintain the continuity of speech rate and energy trends across text segments.

[0083] After each speech synthesis is completed, the corresponding phoneme duration distribution and fundamental frequency change trend are extracted to form the current prosodic state. The specific steps are as follows:

[0084] After segmenting the text based on its articulated boundaries, each segment is fed into the speech synthesis process as an independent articulation unit. During speech synthesis, the corresponding pinyin sequence is generated character by character in the text order, and further converted into a phoneme sequence. Each phoneme is assigned a specific start and end point in time, forming a phoneme timeline accurate to the millisecond level.

[0085] While outputting audio waveform data, the speech synthesis process simultaneously outputs phoneme duration distribution data, where each phoneme corresponds to a specific duration value, and records the transition time relationship between adjacent phonemes. Simultaneously, the fundamental frequency value of the audio waveform is extracted frame-by-frame on the time axis, forming a fundamental frequency change trend curve covering the entire segmented text segment. This curve includes the fundamental frequency value for each time frame and the direction of fundamental frequency change between adjacent time frames. At the end of the speech synthesis for the current segmented text segment, complete audio data, complete phoneme duration distribution data, and complete fundamental frequency change trend data are obtained, maintaining a one-to-one correspondence between the three on the same time axis, thus providing a detailed data source for subsequently forming the current prosodic state.

[0086] After obtaining the phoneme duration distribution data and fundamental frequency change trend data, the above data are organized item by item to form the current prosodic state. Specifically, the phoneme duration distribution data is first statistically analyzed. All phonemes within the currently segmented text segment are arranged in chronological order, and the duration of each phoneme is calculated. Furthermore, the average duration of three consecutive phonemes is calculated and recorded as the average phoneme duration, reflecting the current speech rate trend. Simultaneously, the duration data of the last three phonemes is extracted and recorded as the last phoneme duration. The last phoneme duration is compared with the average duration of the phonemes at the beginning of the segmented text segment to record whether the last phoneme exhibits any prolongation. Next, the fundamental frequency change trend is analyzed. Starting from the fundamental frequency value of the first frame corresponding to the initial phoneme, the direction of fundamental frequency change along the entire time axis is recorded and recorded as the fundamental frequency change direction, including the time intervals of the rising phase, the steady phase, and the falling phase. The fundamental frequency change slope within the last 50 milliseconds and the fundamental frequency value at the end of the last phoneme are specifically extracted and recorded as the last fundamental frequency change slope and the last fundamental frequency value, respectively.

[0087] The average phoneme duration, the duration of the final phoneme, the direction of fundamental frequency change, the value of the final fundamental frequency, and the slope of the final fundamental frequency change in the phoneme duration distribution are combined and organized to form the current prosodic state, which includes the characteristics of speech rate and energy trends. At the same time, the end time point of the text segment after the current segment is added to the current prosodic state to mark the termination position of the prosodic state on the timeline, so that subsequent processing can be connected based on temporal continuity.

[0088] It should be noted that:

[0089] The average duration of the three consecutive phonemes mentioned above refers to the statistical result obtained by averaging the durations of the three adjacent phonemes. It is used to characterize the speech rate characteristics within this local time range, and is not the average duration of a single phoneme. At the same time, the average phoneme duration is used to reflect the overall speech rate trend, while the duration of the final phoneme is used to characterize the local pronunciation characteristics at the end of the segment. The relationship between the two is a comparison between local statistics and local features, and is not a comparison between the three phonemes and another set of three phonemes.

[0090] After the current prosodic state is formed, it is synchronously transmitted when processing the next text segment. The phoneme duration distribution features and fundamental frequency change trend features contained in the current prosodic state are used as the initial reference for the speech synthesis processing of the next text segment. In specific implementation, when a new articulated boundary is generated and the next segmented text segment is obtained, the current prosodic state formed by the previous segmented text segment is input into the speech synthesis process. When generating the phoneme sequence of the next text segment, the average duration of the last three phonemes of the previous segmented text segment is used as the reference value for the duration of the first three phonemes of the next text segment. In the fundamental frequency generation stage, the fundamental frequency value at the end of the previous segmented text segment is used as the starting point of the fundamental frequency value of the next text segment, and the change trend in the same direction continues according to the fundamental frequency change slope at the end of the previous segmented text segment.

[0091] In this way, the speech synthesis processing of the next text segment establishes a continuous connection with the previous text segment in both phoneme duration distribution and fundamental frequency variation trend. During continuous playback, a new current prosodic state is re-formed after each speech synthesis and synchronously transmitted during the processing of the next text segment. This establishes a continuous chain of speech rate and energy trends along the entire playback timeline, ensuring that the pronunciation rhythm and intonation changes remain consistent across text segments.

[0092] It should be noted that:

[0093] The average duration of the last three phonemes of the text segment after the previous segment refers to the statistical result obtained by averaging the duration of each of the last three consecutive phonemes. It is used to characterize the local speech rate characteristics at the end of the segment, and is not the average duration of a single phoneme. This average duration is used as an overall reference value to constrain the time distribution of the starting phonemes of the next text segment, so that the speech rate connection between segments remains consistent.

[0094] Step 4: Combining the current prosodic state and the phoneme alignment information generated by speech synthesis, determine the time correspondence between the last phoneme of the previous text segment and the starting phoneme of the next text segment. Set a co-pronunciation transition interval at the position corresponding to the articulated boundary, and perform time remapping and smoothing mixing on the lip opening and closing parameters to obtain a continuous mouth shape transition result.

[0095] To obtain continuous mouth shape transition results, the specific steps are as follows:

[0096] Given that the current prosodic state has been generated and the phoneme alignment information generated by speech synthesis has been fully output, the start and end times of the last three phonemes of the previous text segment are extracted from the phoneme alignment information, and the start and end times of the first three phonemes of the next text segment are extracted simultaneously. Specifically, the time intervals of the third, second-to-last, and last phonemes of the previous text segment are read from the phoneme alignment information and arranged chronologically to form the end phoneme time sequence; simultaneously, the time intervals of the first, second, and third phonemes of the next text segment are read to form the start phoneme time sequence.

[0097] Subsequently, the end time of the last phoneme time series of the previous text segment is sequentially connected with the start time of the first phoneme time series of the next text segment. The time difference between the two is calculated, and combined with the duration of the last phoneme and the slope of the final fundamental frequency change recorded in the current prosodic state, a joint time and prosodic connection record across text segments is formed. Through this process, the actual interval length between the previous and next text segments and the final intonation trend can be clearly identified on the timeline.

[0098] After obtaining the joint temporal and prosodic records across text segments, the temporal correspondence between the last phoneme of the previous text segment and the first phoneme of the next text segment is specifically determined. In practice, the end time of the last phoneme of the previous text segment is set as the time reference point, and the start time of the first phoneme of the next text segment is set as the next time reference point. The duration of the last phoneme recorded in the current prosodic state is used as the transition time scale to segment the interval between the two time reference points.

[0099] When the interval is longer than the duration of the last phoneme of the previous text segment, the excess portion is included in the cophony preparation interval; when the interval is shorter than the duration of the last phoneme of the previous text segment, the start time of the first phoneme of the next text segment is shifted forward, ensuring a continuous connection between the end times of the last phoneme of the previous text segment and the end times of the last phoneme of the previous text segment. Through this process, a seamless, continuous correspondence is established between the last phoneme of the previous text segment and the first phoneme of the next text segment on the timeline, while maintaining consistency with the speech rate trend in the current prosodic state.

[0100] After the time correspondence has been determined, a co-pronunciation transition interval is set at the position corresponding to the articulated boundary. In specific implementation, the character position corresponding to the articulated boundary is used as the time dividing point. The latter 40% of the time period of the last phoneme of the previous text segment is overlapped with the former 40% of the time period of the first phoneme of the next text segment, thereby forming a co-pronunciation transition interval covering the two phonemes.

[0101] The length of the coarticulation transition interval on the time axis is dynamically determined based on the duration of the final phoneme in the current prosodic state. For example, when the duration of the final phoneme is 100 milliseconds, the coarticulation transition interval is set to 80 milliseconds, with the final phoneme of the previous text segment occupying 40 milliseconds and the starting phoneme of the next text segment occupying 40 milliseconds. Through this specific time ratio division, the coarticulation transition interval has a clear start and end point on the time axis and is consistent with the segment position corresponding to the articulated boundary.

[0102] Given a predefined coarticulation transition interval, the lip opening / closing parameters are subjected to temporal remapping and smoothing blending. Specifically, the sequence of lip opening / closing parameter values ​​for the last phoneme of the previous text segment within the coarticulation transition interval is first extracted; this sequence includes the opening / closing amplitude value for each frame. Then, the sequence of lip opening / closing parameter values ​​for the first phoneme of the next text segment within the coarticulation transition interval is extracted.

[0103] Then, the lip-opening parameter values ​​of the last phoneme of the previous text segment within the coarticulation transition interval are linearly compressed according to the time axis ratio, so that the opening and closing amplitude of the lip-opening parameter values ​​decreases frame by frame. At the same time, the lip-opening parameter values ​​of the first phoneme of the next text segment within the coarticulation transition interval are linearly expanded according to the time axis ratio, so that the opening and closing amplitude of the lip-opening parameter values ​​increases frame by frame. Subsequently, the two sets of lip-opening parameter values ​​are superimposed frame by frame within the coarticulation transition interval, and a new lip-opening parameter sequence is generated in chronological order, so that the opening and closing action of the last phoneme of the previous text segment smoothly transitions to the opening and closing action of the first phoneme of the next text segment.

[0104] Throughout the processing, the direction of change of the lip opening and closing parameters is consistent with the direction of change of the fundamental frequency in the current prosodic state, so that the mouth movements and the trend of speech energy form a unified time progression relationship.

[0105] After completing temporal remapping and smoothing blending, the new lip-opening parameter sequence generated within the coarticulation transition interval is concatenated chronologically with the lip-opening parameter sequences at the beginning of the previous text segment and the end of the next text segment, forming a continuous lip-shape transition result covering the entire process from the end of the previous text segment to the beginning of the next text segment. This continuous lip-shape transition result maintains a one-to-one correspondence with the phoneme alignment information generated by speech synthesis and drives lip-shape changes sequentially along the timeline during playback.

[0106] Through the above steps, a clear co-pronunciation transition interval can be established at the position corresponding to the articulated boundary, and a continuous lip shape transition result can be formed through time remapping and smoothing blending, thereby eliminating the lip shape break phenomenon between text segments and realizing the continuous connection of phoneme rhythm and lip shape movement on the time axis.

[0107] Step 5: Based on the continuous lip shape transition results and the continuously updated current prosodic state, establish a lip shape opening and closing rhythm mapping relationship under a unified time progression framework. During speech playback, drive lip shape changes according to the lip shape opening and closing rhythm mapping relationship to achieve synchronous advancement of phoneme rhythm and lip shape movement.

[0108] To establish a mapping relationship between lip-opening and closing rhythms within a unified time-based framework, the specific steps are as follows:

[0109] After the continuous lip-sync transition results are generated, the complete audio playback timeline of the speech synthesis output is set as the unique reference timeline, and the starting time zero of the corresponding timeline is used as the starting point of the overall time progression. Subsequently, the timestamp corresponding to each lip-sync parameter value in the continuous lip-sync transition results is mapped one by one to the audio playback timeline, so that each frame of lip-sync parameter value has a clear playback time position.

[0110] For example, when the audio playback timeline advances in 10-millisecond steps, the lip-sync parameter sequence in the continuous lip-sync transition results is resampled according to the same 10-millisecond time step, so that the lip-sync parameter values ​​form a discrete time sequence on the timeline that is perfectly aligned with the audio playback. Simultaneously, the phoneme duration distribution data recorded in the continuously updated current prosodic state is marked on the same timeline according to the phoneme start and end times, and the fundamental frequency change trend data is superimposed on the corresponding timeline in time frame order, so that each time step simultaneously corresponds to the audio waveform position, lip-sync parameter value, and fundamental frequency value. Through the above processing, a complete triple alignment relationship is formed on the timeline, thereby constructing a unified time progression framework, making audio playback, lip-sync, and prosodic changes form a strictly consistent mapping structure in the time dimension.

[0111] With the unified time-based framework already established, a lip-opening rhythm mapping relationship is created based on continuous lip-shape transition results and the continuously updated current prosodic state. In practice, the time axis is divided into multiple continuous phoneme rhythm intervals according to the phoneme start and end times recorded in the phoneme alignment information. For example, when the duration of a certain phoneme is 120 milliseconds, the corresponding 120-millisecond interval is marked on the time axis as a single phoneme rhythm interval, and the corresponding lip-opening parameter change curve from the continuous lip-shape transition results is read within the phoneme rhythm interval.

[0112] Subsequently, the lip opening / closing parameter variation curves within the phoneme rhythm intervals are correlated with the fundamental frequency variation direction of the corresponding phoneme in the current prosodic state. When the fundamental frequency increases within the phoneme rhythm interval, the lip opening / closing amplitude is set to gradually increase on the time axis; when the fundamental frequency decreases within the phoneme rhythm interval, the lip opening / closing amplitude is set to gradually decrease on the time axis; when the fundamental frequency remains stable, the lip opening / closing amplitude is maintained within the current value range and changes smoothly. For coarticulation transition intervals across text segments, the entire coarticulation transition interval is considered as a single continuous phoneme rhythm interval, and the lip opening / closing parameters after time remapping and smoothing in the continuous lip shape transition results are used as the unique driving sequence, ensuring that there are no rhythmic breaks within the phoneme rhythm intervals. Through the above division and correspondence of phoneme rhythm intervals, a complete lip opening / closing rhythm mapping relationship is formed, ensuring that each time interval has a clear opening / closing variation rule.

[0113] During speech playback, lip shape changes are driven step-by-step according to the lip-opening rhythm mapping relationship, achieving synchronous advancement of phoneme rhythm and lip movements. Specifically, as the audio playback timeline advances from zero, the lip-opening parameter value corresponding to the current time point is read within each ten-millisecond time step. This parameter value is directly used to control the lip-opening amplitude, ensuring the lip shape completes the corresponding amplitude change within the current time step. Simultaneously, within the same time step, the fundamental frequency value and phoneme duration interval marker at the corresponding time position in the current prosodic state are read to confirm the phoneme rhythm interval to which the current time step belongs. The lip movements are then further advanced according to the opening and closing rules of the phoneme rhythm interval.

[0114] When the time progresses to the start time of the next phoneme rhythm interval, it automatically switches to the new lip-opening parameter change curve and updates the lip shape state frame by frame in the subsequent time steps according to the lip-opening parameter change curve. Through this time-step synchronous reading and driving method, the audio playback time, phoneme rhythm interval, and lip-opening changes are always kept synchronized on the same time axis. This ensures that the phoneme rhythm and lip movements are continuously synchronized throughout the entire speech playback process, thereby guaranteeing continuous lip changes and stable rhythm at the junctions of text segments.

[0115] This application provides, as follows: Figure 4 The digital human speech synthesis and lip-sync system shown includes a streaming pre-reading analysis module, a pronunciation boundary determination module, a prosodic state generation module, a cross-segment lip-sync transition module, and a rhythm mapping driving module.

[0116] The streaming pre-reading analysis module collects text fragments and their corresponding arrival time information from the real-time output of the large language model. During the continuous reception of text fragments, it constructs a pre-reading window, performs joint analysis on the current text fragment and several subsequent characters, identifies word boundaries and grammatical positions that can be broken down, and generates a set of candidate segmentation positions.

[0117] The pronunciation boundary determination module, based on the candidate segmentation position set and combined with the text-to-speech front-end prediction information on prosodic breakage trend, determines the tendency of connected speech and the tendency of pauses for each candidate segmentation position. Under the condition of meeting the condition of maintaining connected speech or the condition of clear pause and reaching the minimum synthesizable length, the segmentation position is confirmed and the pronunciation boundary is obtained.

[0118] The prosodic state generation module performs speech synthesis processing on the segmented text fragments using the articulated boundaries. After each speech synthesis is completed, it extracts the corresponding phoneme duration distribution data and fundamental frequency change trend data to form the current prosodic state, and transmits the current prosodic state synchronously when processing the next text fragment.

[0119] The cross-segment lip-shape transition module combines the current prosodic state and the phoneme alignment information generated by speech synthesis to determine the time correspondence between the end phoneme of the previous text segment and the beginning phoneme of the next text segment. It sets a co-pronunciation transition interval at the position corresponding to the articulated boundary and performs time remapping and smoothing mixing on the lip opening and closing parameters to obtain continuous lip-shape transition results.

[0120] The rhythm mapping driven module establishes a mouth shape opening and closing rhythm mapping relationship under a unified time progression framework based on the continuous mouth shape transition results and the continuously updated current prosodic state. During speech playback, it drives lip shape changes according to the mouth shape opening and closing rhythm mapping relationship.

[0121] The digital human speech synthesis and lip-sync method provided in this application is implemented through the aforementioned digital human speech synthesis and lip-sync system. For details of the specific method and process of the digital human speech synthesis and lip-sync system, please refer to the aforementioned embodiment of the digital human speech synthesis and lip-sync method, which will not be repeated here.

[0122] The foregoing has only described certain exemplary embodiments of this application by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of this application. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of this application.

Claims

1. A method for digital human speech synthesis and lip-syncing, characterized in that, Includes the following steps: Collect text fragments and their corresponding arrival time information from the real-time output of the large language model, perform joint analysis on the current text fragment and several subsequent characters, and generate a set of candidate segmentation positions; Based on the candidate segmentation position set and combined with the text-to-speech front-end prediction information on prosodic breakage trend, the linking tendency and pause tendency of each candidate segmentation position are determined. The segmentation position is confirmed when the linking preservation condition or the clear pause condition is met and the minimum synthesizable length is reached, and the articulated boundary is obtained. The speech synthesis process is performed on the segmented text segments using the calligraphic boundaries. After each speech synthesis is completed, the corresponding phoneme duration distribution and fundamental frequency change trend are extracted to form the current prosodic state. The current prosodic state is then transmitted synchronously when processing the next text segment. When processing the next text segment, the current prosodic state is transmitted synchronously, and the phoneme duration distribution features and fundamental frequency change trend features contained in the current prosodic state are used as the initial reference for the speech synthesis processing of the next text segment. By combining the current prosodic state and the phoneme alignment information generated by speech synthesis, the temporal correspondence between the end phoneme of the previous text segment and the beginning phoneme of the next text segment is determined. A coarse articulation transition interval is set at the position corresponding to the articulated boundary. The lip opening and closing parameters are subjected to temporal remapping and smoothing mixing to obtain a continuous mouth shape transition result. Based on the continuous lip shape transition results and the continuously updated current prosodic state, a lip shape opening and closing rhythm mapping relationship is established, and lip shape changes are driven according to the lip shape opening and closing rhythm mapping relationship during speech playback.

2. The method for digital human speech synthesis and lip-syncing according to claim 1, characterized in that, Generating a set of candidate segmentation locations involves the following steps: It continuously receives text segments output by the large language model in a streaming manner, records the arrival time information of each text segment, appends the text segments to the continuous text cache sequence in character order, and forms a pre-read window based on the continuous text cache sequence; Within the pre-read window, semantic parsing and grammatical structure analysis are performed on continuous text cache sequences to identify word boundaries and grammatical positions where sentences can be broken, and a correspondence is formed by combining character order and arrival time information; The identified word boundaries and grammatical positions that can be broken down are organized, converted into character indices, and associated with arrival time information to form a set of candidate segmentation positions containing character indices and time stamps.

3. The method for digital human speech synthesis and lip-syncing according to claim 2, characterized in that, The pre-read window is dynamically adjusted. During the dynamic adjustment of the pre-read window, the character order and arrival time information in the continuous text cache sequence are used as the basis for updating the candidate segmentation position set. The word boundaries and sentence breakable grammatical positions that have changed semantic structure due to the addition of new characters are re-identified, and the updated candidate segmentation position set is continuously used as the basis for segmentation to participate in the pronunciation continuity boundary determination.

4. The method for digital human speech synthesis and lip-syncing according to claim 1, characterized in that, Obtaining the articulated boundaries and segmenting the text based on these boundaries includes the following steps: The candidate segmentation positions are arranged in character index order. For each candidate segmentation position, the continuous characters before the candidate segmentation position are extracted to form the text interval to be pronounced. The fundamental frequency change trend information, phoneme duration prediction information, and sentence ending intonation information are obtained by combining the text-to-speech front-end prediction information of prosodic breakage trend and establishing corresponding relationships. Based on the established correspondence, the candidate segmentation positions are judged for both continuous reading tendency and pause tendency, and the candidate positions that meet the continuous reading preservation condition or the explicit pause condition are determined in combination with the minimum synthesizable length. Select the corresponding positions from the candidate positions in character order as the pronounceable boundaries; Based on the pronunciation boundaries, the corresponding character intervals are extracted to form segmented text fragments. At the same time, the remaining characters are retained to generate a new set of candidate segmentation positions and continue to perform the processing of connected speech tendency determination and pause tendency determination.

5. The method for digital human speech synthesis and lip-syncing according to claim 4, characterized in that, In the process of determining the tendency of connected speech and the tendency of pause, the fundamental frequency change trend information corresponding to the candidate segmentation position is correlated with the phoneme duration prediction information. When the fundamental frequency change shows a convergence trend and the duration of the last phoneme shows an extension trend and meets the semantic closure state, it is confirmed that the explicit pause condition is met. When the fundamental frequency change shows a continuous trend and the preceding and following words form a continuous expression relationship, it is confirmed that the connected speech maintenance condition is met.

6. The method for digital human speech synthesis and lip-syncing according to claim 1, characterized in that, After speech synthesis is completed, extracting the corresponding phoneme duration distribution data and fundamental frequency change trend data to form the current prosodic state includes the following steps: Based on the phonable boundaries, the segmented text fragments are obtained and speech synthesis processing is performed to generate audio waveform data. At the same time, the corresponding phoneme sequence and the phoneme time start and time end are output to form phoneme duration distribution data. The fundamental frequency value is extracted frame by frame to form fundamental frequency change trend data. The phoneme duration distribution data and fundamental frequency change trend data are organized to obtain the average phoneme duration and the last phoneme duration. The fundamental frequency change direction, the last fundamental frequency value and the last fundamental frequency change slope are extracted and combined with the end time of the text segment to form the current prosodic state. When processing the next text segment, the current prosodic state is input into the speech synthesis processing flow, and the duration of the last phoneme and the value of the last fundamental frequency and the slope of change are used as the starting reference.

7. The method for digital human speech synthesis and lip-syncing according to claim 6, characterized in that, The current prosodic state includes the duration of the last phoneme in the phoneme duration distribution data and the value and slope of the last fundamental frequency in the fundamental frequency change trend data. When processing the next text segment, the duration of the last phoneme is used to limit the range of the duration of the starting phoneme, and the value and slope of the last fundamental frequency change are used to limit the direction of the starting fundamental frequency generation.

8. The method for digital human speech synthesis and lip-syncing according to claim 1, characterized in that, Obtaining a continuous mouth shape transition result involves the following steps: Based on the current prosodic state and the phoneme alignment information generated by speech synthesis, the time interval of the last phoneme of the previous text segment and the time interval of the first phoneme of the next text segment are extracted and a time sequence is formed. The time and prosodic joint connection record is established by combining the duration of the last phoneme and the slope of the last fundamental frequency change. Based on the joint connection record of time and prosody, the temporal correspondence between the last phoneme of the previous text segment and the first phoneme of the next text segment is determined, and the time interval is adjusted according to the duration of the last phoneme to form a continuous time mapping relationship. Based on the time correspondence, a co-pronunciation transition interval is set at the position corresponding to the articulated boundary, and the length of the transition interval is determined according to the duration of the final phoneme; The lip opening parameter value sequence is extracted within the coarse articulation transition interval and subjected to time remapping and smoothing mixing to form the lip opening parameter sequence. By concatenating the lip opening and closing parameter sequence with the corresponding text segment's lip opening and closing parameter sequence in chronological order, a continuous mouth shape transition result corresponding to the phoneme alignment information is obtained.

9. A method for digital human speech synthesis and lip-syncing according to claim 8, characterized in that, Based on the continuous mouth shape transition results and the continuously updated current prosodic state, the mouth shape opening and closing rhythm mapping relationship is established through the following steps: The audio playback timeline of the speech synthesis output is set as the reference timeline. The timestamps of the lip opening and closing parameters in the continuous mouth shape transition results are mapped to the audio playback timeline. The phoneme duration distribution data and fundamental frequency change trend data in the current prosodic state are marked on the same timeline to form a unified time progression framework. Within a unified time-progression framework, phoneme rhythm intervals are divided based on phoneme alignment information. The lip opening and closing parameter change curves of the corresponding phoneme rhythm intervals for continuous lip shape transition results are read, and a lip opening and closing rhythm mapping relationship is established in combination with the fundamental frequency change direction in the current prosodic state.

10. A digital human speech synthesis and lip-syncing system, used to implement the digital human speech synthesis and lip-syncing method according to any one of claims 1-9, characterized in that, It includes a streaming pre-reading analysis module, a pronunciation boundary determination module, a prosodic state generation module, a cross-segment mouth shape transition module, and a rhythm mapping driving module; The streaming pre-reading analysis module collects text fragments and their corresponding arrival time information from the real-time output of the large language model. During the continuous reception of text fragments, it constructs a pre-reading window, performs joint analysis on the current text fragment and several subsequent characters, identifies word boundaries and grammatical positions that can be broken down, and generates a set of candidate segmentation positions. The pronunciation boundary determination module, based on the candidate segmentation position set and combined with the text-to-speech front-end prediction information on prosodic breakage trend, determines the tendency of connected speech and the tendency of pauses for each candidate segmentation position. Under the condition of meeting the condition of maintaining connected speech or the condition of clear pause and reaching the minimum synthesizable length, the segmentation position is confirmed and the pronunciation boundary is obtained. The prosodic state generation module performs speech synthesis processing on the segmented text fragments using the articulated boundaries. After each speech synthesis is completed, it extracts the corresponding phoneme duration distribution and fundamental frequency change trend to form the current prosodic state, and transmits the current prosodic state synchronously when processing the next text fragment. When processing the next text segment, the current prosodic state is transmitted synchronously, and the phoneme duration distribution features and fundamental frequency change trend features contained in the current prosodic state are used as the initial reference for the speech synthesis processing of the next text segment. The cross-segment lip-shape transition module combines the current prosodic state and the phoneme alignment information generated by speech synthesis to determine the time correspondence between the end phoneme of the previous text segment and the beginning phoneme of the next text segment. It sets a co-pronunciation transition interval at the position corresponding to the articulated boundary and performs time remapping and smoothing mixing on the lip opening and closing parameters to obtain continuous lip-shape transition results. The rhythm mapping driven module establishes a mouth shape opening and closing rhythm mapping relationship under a unified time progression framework based on the continuous mouth shape transition results and the continuously updated current prosodic state. During speech playback, it drives lip shape changes according to the mouth shape opening and closing rhythm mapping relationship.

Images