Digital human rendering method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide a digital human rendering method and device, a storage medium and a program product. The digital human rendering method comprises the following steps: analyzing a target voice stream matched with a to-be-rendered digital human to obtain a phoneme sequence and voice rhythm characteristics; a mouth shape control parameter sequence corresponding to the phoneme sequence is determined, all mouth shape control parameters in the mouth shape control parameter sequence are in one-to-one correspondence with all phonemes in the phoneme sequence, and a mouth shape control curve is generated based on the mouth shape control parameter sequence; performing rhythm alignment processing on the mouth shape control curve according to the voice rhythm characteristics, executing rasterization conversion, and generating a mouth shape image block sequence synchronized with the target voice stream; and synthesizing each mouth shape image block in the mouth shape image block sequence with other image contents of the digital human to generate a digital human image frame sequence.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of digital media technology, and more particularly, to a digital human rendering method, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Amid the rapid development of digital media technology, the application of digital human technology has penetrated into a wide range of fields, including film and television production, animation generation, virtual customer service, virtual live broadcasting, and intelligent education. With the continuous advancement of technologies such as artificial intelligence, computer graphics, and big data analysis, digital humans are no longer just synonymous with virtual characters; they have become an indispensable tool in various industries.

[0003] For example, in film and television production and animation, digital humans not only play a vital role but also provide audiences with a more immersive experience through virtual reality and augmented reality technologies. Another example is in areas such as virtual customer service and live streaming, where digital humans, as carriers of intelligent interaction, can simulate real human speech and provide more vivid and personalized services.

[0004] Among them, lip synchronization technology, as a key technology to achieve natural interaction of digital humans, directly affects its rendering effect and the quality of user experience. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a digital human rendering method, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0007] According to a first aspect of one or more embodiments of this specification, a digital human rendering method is provided, comprising:

[0008] Parse the target speech stream that matches the digital human to be rendered to obtain the phoneme sequence and speech rhythm features;

[0009] determining a lip control parameter sequence corresponding to the phoneme sequence, wherein each lip control parameter in the lip control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence, and generating a lip control curve based on the lip control parameter sequence;

[0010] Performing rhythm alignment processing on the lip control curve according to the speech rhythm feature and performing rasterization conversion to generate a lip image block sequence synchronized with the target speech stream;

[0011] Each lip shape image block in the lip shape image block sequence is synthesized with other image contents of the digital human to generate a digital human image frame sequence.

[0012] According to a second aspect of the embodiments of this specification, an electronic device is provided, including:

[0013] processor;

[0014] a memory for storing processor-executable instructions;

[0015] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0016] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0018] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0019] In the embodiments of this specification, the target speech stream for the digital human to be rendered is first parsed to extract the phoneme sequence and speech rhythm features, providing a precise data foundation for subsequent lip-sync control. By mapping the phonemes in the phoneme sequence to corresponding lip-sync control parameters and generating a lip-sync control curve, each pronunciation is precisely aligned with the corresponding lip-sync movement, improving the accuracy and consistency of lip-sync changes. Next, the lip-sync control curve is rhythmically aligned based on the rhythmic characteristics of the speech, and a rasterization conversion is performed to generate a sequence of lip-sync image blocks. This ensures that the rhythm of the lip-sync opening and closing naturally matches the rhythmic beat of the speech, effectively preventing lip-sync and speech desynchronization. Subsequently, the generated lip-sync image blocks are synthesized with other image content of the digital human to construct a complete digital human image frame sequence, achieving a natural integration of lip-sync animation with the overall image. In summary, this method effectively improves the synchronization and realism between the digital human's mouth shape and speech, making the digital human's conversational performance more natural and smooth, significantly enhancing the user's immersive and interactive experience.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the architecture of a digital human rendering service system provided by an exemplary embodiment.

[0022] Figure 2 This is a flowchart of a digital human rendering method provided by an exemplary embodiment.

[0023] Figure 3 This is a flowchart of another digital human rendering method provided by an exemplary embodiment.

[0024] Figure 4 The present invention is a flowchart of a digital human image frame sequence display output process provided by an exemplary embodiment.

[0025] Figure 5 It is a structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0027] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0029] With the rapid development of digital media technology, digital humans are increasingly being used in film and television production, animation generation, virtual customer service, virtual live broadcasting, smart education, and other fields. Lip synchronization, a key technology for achieving natural digital human interaction, directly impacts user experience.

[0030] Figure 1 This is a schematic diagram of the architecture of a digital human rendering service system provided by an exemplary embodiment. Figure 1As shown, the system may include a server 11, a network 12, and several terminals, such as a PC (Personal Computer) 13, a mobile phone 14, a digital human-in-one machine 15, and the like.

[0031] The server 11 may be a physical server including an independent host, or a virtual server hosted by a host cluster. During operation, the server 11 may run the server-side program of the digital human rendering application to realize a corresponding digital human rendering service platform.

[0032] PC 13 , mobile phone 14 , and digital human-in-one machine 15 are just some types of terminals that users can use to perform human-computer interaction.

[0033] Among them, the all-in-one digital human machine is an intelligent terminal device that deeply integrates multiple core technologies (such as artificial intelligence, speech recognition, natural language processing, and 3D modeling) to achieve digital human interaction. Its core advantage lies in breaking the fragmentation of technical modules and allowing the digital human function to be fully operated in a single device through the coordinated optimization of hardware and software. In application scenarios, the all-in-one digital human machine can be directly implemented in retail shopping guides, government services, education and training, and other scenarios: for example, at bank branches, it can answer business inquiries through voice interaction, while enhancing communication affinity with natural lip shape and expression; on campus, it can serve as a virtual teacher to conduct interactive teaching, demonstrating experimental steps through 3D action simulation. This "integrated" design ultimately realizes the rapid implementation of digital human technology from "laboratory" to "real-world scenarios", allowing users to enjoy a smooth and natural digital human interaction experience without the need for complex technical reserves.

[0034] In fact, users can obviously also use terminals such as the following types: tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification are not limited to this. During operation, the terminal can run the client-side program of the digital human rendering application, and can be implemented as a client of the digital human rendering service. Among them, the client application of the above-mentioned digital human rendering service can be started and run on the terminal. The client-side program can be a native application installed on the terminal, or the client-side program can be a small program, a quick application or other similar forms. Of course, when using web page technologies such as HTML5 or similar, the relevant functions can be implemented through the page displayed by the browser. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0035] Regarding the network 12 for interaction between terminals such as PC 13, mobile phone 14, and digital human-in-one machine 15 and server 11, communication can be achieved using a wired or wireless network based on the communication methods supported by the terminals, and this specification does not limit this. For example, if PC 13 supports both wired and wireless communication, then communication can be achieved using either a wired or wireless network as needed, while mobile phone 14 generally only supports wireless communication and thus can achieve communication using a wireless network.

[0036] Among them, the digital human rendering method provided in the embodiment of this specification can be executed by the above-mentioned server or terminal (such as a digital human all-in-one machine); or, the digital human rendering method provided in the embodiment of this specification can be executed in part by the above-mentioned server and the other part by the terminal; this embodiment does not impose any restrictions on this.

[0037] See also Figure 2 , shows a flow chart of a digital human rendering method, which is executed by an electronic device (such as the above-mentioned server, terminal, or other device), and includes:

[0038] In S201 , the target speech stream that matches the digital human to be rendered is parsed to obtain a phoneme sequence and speech rhythm features.

[0039] The "target voice stream for the digital human to be rendered" refers to the voice content needed to drive the digital human to produce corresponding lip movements, facial expressions, and interactive actions. It is the core input signal in the digital human rendering process. Essentially, it is an audio data stream that carries language information. It can be either real-time generated speech (such as real-time voice input when a user is talking to a digital human, or a virtual anchor's real-time broadcast voice) or pre-recorded voice files (such as the digital human's preset explanation audio or scripted dialogue recordings).

[0040] Exemplarily, the electronic device may use a sliding time window of a preset length (such as 20ms or 25ms) to perform frame processing on the voice stream to obtain an audio frame sequence; then, perform noise reduction processing on each audio frame in the audio frame sequence to improve the quality of the audio signal; then, extract Mel-Frequency Cepstral Coefficients (MFCC) of the denoised audio frame, and identify the corresponding phoneme sequence and speech rhythm features based on the Mel-Frequency Cepstral Coefficients in combination with a lightweight acoustic model (such as a convolutional neural network), where the speech rhythm features include but are not limited to phoneme duration, phoneme change speed, pitch change, intensity change, pause duration, stress position, etc.

[0041] This step achieves highly robust phoneme recognition and speech rhythm extraction, providing accurate and structured speech driving data for subsequent lip generation.

[0042] In S202, a lip control parameter sequence corresponding to the phoneme sequence is determined, each lip control parameter in the lip control parameter sequence corresponds to each phoneme in the phoneme sequence, and a lip control curve is generated based on the lip control parameter sequence.

[0043] In one possible implementation, the electronic device determines the lip control parameters corresponding to each phoneme in the phoneme sequence based on a preset mapping relationship between phonemes and lip control parameters, thereby obtaining a lip control parameter sequence. For example, the mapping relationship can be manually set or statistically obtained in the training data, and saved in the form of a table or hash map in local or cloud storage. After parsing the speech to obtain the phoneme sequence, the electronic device can directly look up the table to obtain the corresponding lip control parameters, forming a continuous lip control parameter sequence for subsequent curve generation and image rendering. This method is simple to implement and has a fast response speed. It is suitable for scenarios with a relatively single language and relatively fixed lip change rules. The mapping process does not require additional computing overhead, which is conducive to deployment in terminal devices or low-resource scenarios with high latency requirements.

[0044] In another possible implementation, the speech rhythm features include phoneme-level rhythm features corresponding to each phoneme in the phoneme sequence. The electronic device can perform vectorization processing based on each phoneme in the phoneme sequence and the corresponding phoneme-level rhythm features to obtain a speech feature vector corresponding to each phoneme; obtain a pre-stored lip shape vector library, which is used to store the mapping relationship between the speech feature vectors and lip shape control parameters of phonemes in different languages ​​and / or different pronunciations; finally, based on the speech feature vectors of each phoneme in the phoneme sequence, retrieve the lip shape control parameter sequence from the lip shape vector library, and each lip shape control parameter in the lip shape control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence. This method has greater generalization capabilities, supports adaptive processing for different speaking speeds, languages, and accents, and significantly improves the accuracy of lip shape fitting for natural speech.

[0045] Exemplarily, the lip shape vector library includes an approximate nearest neighbor index structure constructed based on the mapping relationship between the speech feature vectors of phonemes of different languages ​​and / or different pronunciations and the lip shape control parameters. When the electronic device retrieves the lip shape control parameter sequence from the lip shape vector library, it can query the approximate nearest neighbor index structure based on the speech feature vector of each phoneme in the phoneme sequence to obtain multiple candidate speech feature vectors similar to the speech feature vector of the phoneme. Then, based on the similarity between the speech feature vector of the phoneme and each candidate speech feature vector, and the mapping relationship between each candidate speech feature vector and the lip shape control parameter, the lip shape control parameter corresponding to the phoneme is determined. This embodiment uses a pre-constructed approximate nearest neighbor (ANN) index structure to quickly locate multiple similar candidates in a high-dimensional speech feature vector space, thereby improving the efficiency and response speed of lip shape control parameter retrieval.

[0046] For example, the lip shape control parameters determine the shape, position, degree of opening and closing of the mouth at a specific time point. The lip shape control parameters include, but are not limited to, mouth opening and closing degree, mouth corner angles, lip shape parameters, lip closure, tongue position parameters, tooth visibility, etc.

[0047] After obtaining a lip control parameter sequence, the electronic device can extract the curve control points associated with each lip control parameter in the lip control parameter sequence from a pre-stored vector path library to generate lip control curve segments based on the curve control points associated with each lip control parameter. The electronic device then splices the lip control curve segments corresponding to each lip control parameter in the order of the phonemes in the phoneme sequence to generate a lip control curve. This lip control curve is used to represent a continuous curve of lip shape changes over time, defining the lip shape transformation process synchronized with speech in digital human facial animation. In this embodiment, by generating a lip control curve segment for each phoneme based on the curve control points corresponding to the lip control parameters and splicing them together to form a complete lip control curve, a natural transition of lip shape states between phonemes can be achieved, avoiding a "frame skipping" or sudden lip shape changes, making the digital human mouth shape more smooth and natural.

[0048] Moreover, the above process is lightweight and has low reliance on GPU (Graphics Processing Unit) computing power. The core of the entire process is to extract associated curve control points from the pre-stored vector path library, generate curve segments based on these control points, and then splice them into lip control curves in sequence. Since the generation of curve control points and curve segments depends on pre-stored resources, the main operations are simple extraction, splicing and sequential combination, without the need for complex real-time calculations or graphics rendering operations. The entire process is mainly based on data retrieval and basic splicing logic, with small amount of calculation and concentrated on lightweight data processing, which can be completed efficiently by ordinary hardware.

[0049] Exemplarily, the vector path library is used to store curve control points constructed based on actual language pronunciation training, such as Bezier curve control points for generating Bezier curves; or B-Spline curve (B-Spline) control points for generating B-Spline curves.

[0050] Exemplarily, in order to further improve the fluency of digital mouth shape, the electronic device can perform interpolation processing based on the mouth shape control curve segments corresponding to adjacent phonemes to generate connecting curve segments for smooth transition; for example, the interpolation processing can be performed using algorithms such as piecewise approximation algorithm, linear interpolation or spline interpolation, and this embodiment does not impose any restrictions on this; then the electronic device splices the mouth shape control curve segments corresponding to each phoneme in the phoneme sequence and the connecting curve segments between adjacent phonemes according to the arrangement order of the phonemes in the phoneme sequence to generate a mouth shape control curve. This embodiment can further improve the overall smoothness and continuity of the mouth shape control curve by introducing connecting curve segments for smooth filtering between the mouth shape control curve segments corresponding to adjacent phonemes. The introduction of connecting curve segments helps to simulate the gradual evolution of the mouth shape during the natural pronunciation process of humans, so that the generated mouth shape control curve is more consistent with the real speech expression in terms of shape and rhythm.

[0051] In S203 , rhythm alignment processing is performed on the lip control curve according to the speech rhythm characteristics, and rasterization conversion is performed to generate a lip image block sequence synchronized with the target speech stream.

[0052] Exemplarily, speech rhythm features include at least one of the duration and change rate of each phoneme in a phoneme sequence, but are not limited thereto. The duration of a phoneme refers to the duration of the actual pronunciation of a phoneme in speech. The change rate of a phoneme refers to the speed of switching between phonemes. A high change rate indicates a rapid transition between phonemes and a faster speaking rate. For example, it can be expressed as the frequency of phoneme switching per unit time or the inverse of the time interval between adjacent phonemes.

[0053] The electronic device may adjust the time span of the lip control curve segment corresponding to each phoneme in the lip control curve based on at least one of the phoneme duration and the phoneme change rate of each phoneme in the phoneme sequence. The time span of the curve segment refers to the length (time) occupied by the lip control curve segment corresponding to each phoneme on the time axis of the lip control curve.

[0054] The longer the phoneme duration, the more "stretched" the pronunciation of the phoneme in the speech, and the corresponding mouth shape should be maintained for a relatively longer time. Therefore, in the mouth shape control curve, the time span of the curve segment assigned to the phoneme should be extended accordingly to maintain the synchronization and stability of the mouth shape and speech. Conversely, if the duration is short, the span of the corresponding curve segment in the overall control curve will also be shortened accordingly. In other words, the phoneme duration of a phoneme is positively correlated with the time span of the corresponding mouth shape control curve segment, thereby reducing the sense of "fast-forward" and preventing the digital mouth shape from deforming or jumping prematurely during the pronunciation of long phonemes.

[0055] Faster phoneme changes mean a more rapid pronunciation rhythm, and the mouth shape should also transition more quickly. Therefore, the time span of the curve segment should be shortened to accommodate the rapidly changing rhythm and allow the mouth shape to complete the necessary deformation in a short period of time. In other words, the phoneme change speed of a phoneme is negatively correlated with the time span of the corresponding mouth shape control curve segment. This enhances the responsiveness of the digital human's speech rhythm and avoids a sense of lag or excessively slow mouth shape.

[0056] In practical applications, electronic devices can consider at least one of the two factors, duration and change speed. If a phoneme has a long duration and a slow change speed, the time span of the curve segment will be longer; if a phoneme has a short duration and a fast change speed, the time span of the curve segment will be shorter. If both are considered at the same time, a weighting coefficient can be set for comprehensive processing.

[0057] This embodiment controls the time span of the lip shape control curve segment based on the duration of the phoneme and / or the speed of phoneme change, so that the lip shape control curve is aligned with the speech rhythm in the time dimension, ensuring that the lip shape change is synchronized with the pronunciation process, and enhancing the timing consistency and naturalness of the animation. In fast or slow speech scenarios, this mechanism can dynamically adjust the time proportion of each curve segment to avoid the problem of "the lip shape cannot keep up at fast speech speeds and the lip shape is stiff at slow speech speeds". It has the ability to adapt to different speaking rhythms, thereby improving the realism and emotional expression of the entire digital human performance. In addition, the rhythm alignment processing based on the speech rhythm characteristics only requires the timing adjustment of the lip shape control curve based on the pre-extracted rhythm characteristics. It is also a lightweight logical operation that does not take up too much computing power.

[0058] After obtaining the aligned lip control curve, the electronic device can perform rasterization conversion on the aligned lip control curve. The rasterization conversion process is used to convert the aligned lip control curve from vector graphics to lip image blocks frame by frame according to the time axis, thereby generating a sequence of lip image blocks synchronized with the target voice stream. This embodiment can achieve frame-level driven image block output through rasterization processing to meet the needs of real-time animation generation. In addition, the rasterization conversion link targets the local area of ​​the lip image block, rather than the full rasterization of the entire digital human model. The calculation scope is greatly reduced, which can effectively reduce the use of GPU computing power.

[0059] In S204, each lip shape image block in the lip shape image block sequence is synthesized with other image contents of the digital human to generate a digital human image frame sequence.

[0060] Among them, other image content of the digital human refers to the areas in the overall image of the digital human except the mouth, such as the head, eyes, facial skin, hairstyle, background and other image parts, which are usually pre-rendered static or dynamic layers.

[0061] In this step, the electronic device synthesizes the generated lip-sync image block sequence frame by frame with the digital human's other image content (such as the head, face, and background), thereby forming a complete, continuously playable digital human image frame sequence for visual output. Furthermore, the synthesis of the lip-sync image block sequence and other image content is primarily based on coordinate-matching image overlay operations, eliminating the need for complex rendering calculations such as lighting and shading, further reducing the demand on GPU graphics processing capabilities.

[0062] The digital human rendering method provided in this embodiment first parses the target speech stream associated with the digital human to be rendered, extracting the phoneme sequence and speech rhythm features, providing a precise data foundation for subsequent lip-sync control. By mapping the phonemes in the phoneme sequence to corresponding lip-sync control parameters and generating a lip-sync control curve, each pronunciation is precisely aligned with the corresponding lip-sync movement, improving the accuracy and consistency of lip-sync changes. Next, the lip-sync control curve is rhythmically aligned based on the speech rhythm features and rasterized to generate a sequence of lip-sync image blocks. This ensures that the rhythm of the lip-sync opening and closing naturally matches the rhythmic beat of the speech, effectively preventing lip-sync and speech desynchronization. Subsequently, the generated lip-sync image blocks are synthesized with other image content of the digital human to construct a complete digital human image frame sequence, achieving a natural integration of lip-sync animation and the overall image. In summary, this method effectively improves the synchronization and realism between the digital human's mouth shape and speech, making the digital human's conversational performance more natural and smooth, significantly enhancing the user's immersive and interactive experience. Furthermore, it is lightweight and requires less GPU computing power.

[0063] In some embodiments, seeFigure 3 , this specification also provides another digital human rendering method, which includes:

[0064] In S301, the target speech stream that matches the digital human to be rendered is parsed to obtain the phoneme sequence and speech rhythm features.

[0065] In S302, a lip control parameter sequence corresponding to the phoneme sequence is determined, and each lip control parameter in the lip control parameter sequence corresponds to each phoneme in the phoneme sequence.

[0066] In S303 , based on the speech rhythm feature and / or the idle computing resources of the rendering device, a current rendering mode is determined from the real-time rendering mode and the non-real-time rendering mode.

[0067] During the digital human rendering process, in order to achieve a balance between rendering effect and rendering efficiency under different operating environments or speech rhythm conditions, this embodiment introduces a dynamic switching mechanism for rendering modes.

[0068] In one possible implementation, if the speech rhythm characteristics meet the first real-time rendering condition, the current rendering mode is determined to be real-time rendering mode to ensure the immediacy and consistency of the rendering response; otherwise, the current rendering mode is determined to be non-real-time rendering mode to ensure rendering efficiency. The speech rhythm characteristics include the speed of phoneme changes; the first real-time rendering condition includes: the phoneme change speed is greater than a preset speed threshold. And / or, the speech rhythm characteristics include the duration of a phoneme; the first real-time rendering condition includes: the phoneme duration is less than a preset duration.

[0069] That is, when real-time rendering conditions are not met, the phoneme changes slowly or lasts for a long time, which means that the rhythm of the speech is relatively stable and does not produce rapid changes in lip shape. In this case, by decompressing the pre-stored lip shape image blocks, the lip shape corresponding to the speech changes can be effectively rendered without the need for real-time calculation. Because the lip shape changes relatively slowly, the stored lip shape image blocks can better match the speech rhythm.

[0070] When real-time rendering is required, phonemes change quickly or for short durations, indicating a fast speech rhythm and frequent lip-syncing. Pre-stored image blocks struggle to capture all these subtle and rapid changes. Lip-sync image blocks are discrete, finite samples, making it difficult to accurately capture these rapidly changing lip movements. This can lead to lip-syncing and speech desynchronization or stuttering. Therefore, real-time calculation and updating of each phoneme change is necessary. See the following description for details.

[0071] In another possible implementation, if the idle computing resources of the rendering device meet the second real-time rendering condition, the current rendering mode is determined to be real-time rendering mode to ensure the immediacy and consistency of the rendering response; otherwise, the current rendering mode is determined to be non-real-time rendering mode to ensure rendering efficiency. The second real-time rendering condition includes: the idle computing resources of the rendering device (such as processor utilization, free memory, etc.) are greater than the resource threshold, and the resource threshold is positively correlated with the duration of the target voice stream, that is, the longer the target voice stream, the higher the resources required, and the electronic device accordingly raises the judgment threshold.

[0072] In another possible implementation, if the speech rhythm characteristics meet the first real-time rendering condition and the idle computing resources of the rendering device meet the second real-time rendering condition, the current rendering mode is determined to be the real-time rendering mode to ensure the immediacy and continuity of the rendering response; otherwise, the current rendering mode is determined to be the non-real-time rendering mode to ensure rendering efficiency.

[0073] This embodiment realizes dynamic switching of rendering modes by judging at least one of speech rhythm and computing resources, ensuring that electronic devices can operate stably under different speech complexities and device load conditions, effectively avoiding the forced execution of high-load tasks when resources are insufficient, and improving overall rendering efficiency. For example, for voice input with fast speaking speed and frequent rhythm changes, the real-time rendering mode is preferred, which can effectively improve the synchronization between lip shape and speech and the ability of instant feedback, and enhance the naturalness of interaction. Whether deployed on high-performance devices or resource-constrained mobile terminals, this solution can flexibly match the operating environment, lower the deployment threshold, and improve scalability.

[0074] In S304, if the current rendering mode is real-time, a lip control curve is generated based on the lip control parameter sequence. The lip control curve is rhythmically aligned according to the speech rhythm characteristics, and rasterization is performed to generate a sequence of lip image blocks synchronized with the target speech stream. This is described above and will not be repeated here.

[0075] In S305, if the current rendering mode is a non-real-time rendering mode, a pre-stored compressed image resource is obtained, the compressed image resource including compressed lip image blocks associated with different lip control parameters, based on the lip control parameters corresponding to each phoneme in the phoneme sequence, the compressed lip image blocks corresponding to each phoneme are obtained from the compressed image resource, the compressed lip image blocks are decompressed and arranged in the order of the phoneme sequence to obtain a lip image block sequence.

[0076] In this step, the electronic device reads a pre-stored compressed image resource. This compressed image resource is a set of image blocks generated based on common lip control parameters. The image compression process has been completed and the image is stored in a specific compression format (such as JPEG, WebP, PNG-8, or texture compression format). The compressed image resource includes compressed lip image blocks associated with different lip control parameters. Each lip control parameter is associated with at least one compressed lip image block, and the specific number can be set according to the actual application scenario. Based on the lip control parameter value corresponding to each phoneme in the phoneme sequence, the electronic device searches the compressed image resource for a matching or closest compressed lip image block. The matched compressed image blocks are decompressed one by one, restoring them to directly renderable lip image data (such as a bitmap or texture image). These blocks are then arranged in the order of the phoneme sequence to ultimately form a sequence of lip image blocks for rendering. This pre-generation and compressed storage approach reduces system burden and improves overall processing efficiency. The image block decompression cost is lower than real-time computational generation, and fast decoding through pre-loaded resources makes it suitable for deployment on terminal devices with limited performance.

[0077] In S306, each lip shape image block in the lip shape image block sequence is synthesized with other image contents of the digital human to generate a digital human image frame sequence.

[0078] In this step, the electronic device synthesizes the generated lip-sync image block sequence frame by frame with other image content of the digital human (such as head, face, background, etc.) at the image level, thereby forming a complete and continuously playable digital human image frame sequence to achieve visual output.

[0079] In this embodiment, the target speech stream for the digital human to be rendered is first parsed to extract the phoneme sequence and speech rhythm characteristics, providing a precise basis for subsequent lip control and further determining the corresponding lip control parameter sequence. Based on this, the rendering mode can be flexibly switched between real-time and non-real-time rendering modes based on the speech rhythm characteristics and the available computing resources of the rendering device.

[0080] In real-time rendering mode, the lip control curve is generated and rhythm alignment is performed based on the speech rhythm characteristics. Rasterization conversion is then performed to ensure that lip changes are highly synchronized with speech, presenting a natural and smooth visual effect. In non-real-time rendering mode, pre-stored compressed image resources are called upon to quickly retrieve and decompress the corresponding lip image blocks based on the lip control parameters. This effectively reduces the computational burden and ensures rendering efficiency when device resources are limited.

[0081] Finally, the generated lip-sync image blocks are synthesized with the rest of the digital human's image content to output a continuous sequence of digital human image frames. Overall, this method, through its dynamic adaptive rendering strategy, not only ensures precise synchronization and realistic representation of the digital human's mouth shape and voice in different application scenarios, but also improves resource utilization efficiency and terminal adaptability, providing users with a more stable and smooth digital human interaction experience.

[0082] In some embodiments, in order to adapt to the rendering pressure brought by long voice input and improve the flexibility and efficiency of rendering, when the duration of the target voice stream is greater than a preset threshold, the target voice stream is divided into at least two voice segments; after parsing each voice segment, the phoneme subsequence corresponding to each voice segment and the local voice rhythm features corresponding to each phoneme subsequence can be obtained, and the lip control parameter subsequence corresponding to each phoneme subsequence can be further determined.

[0083] On this basis, the electronic device can determine the rendering mode corresponding to each phoneme subsequence from the real-time rendering mode and the non-real-time rendering mode based on the local speech rhythm characteristics of each phoneme subsequence and / or the idle computing resources of the rendering device during the corresponding rendering period. This embodiment divides the speech input that exceeds the preset duration into at least two speech segments and determines the rendering mode for each segment separately, avoiding the problem of uneven resource allocation or system response delay caused by excessive speech during the overall rendering process, and significantly improving the system stability and processing controllability in long speech rendering scenarios.

[0084] After determining the rendering mode corresponding to each phoneme subsequence, the electronic device can render each phoneme subsequence separately based on the corresponding rendering mode. The rendering method can be executed with reference to the aforementioned technical paths of real-time rendering and non-real-time rendering. Through the above method, a digital human image frame sequence corresponding to each phoneme subsequence is generated respectively. Subsequently, the electronic device integrates the digital human image frame sequence corresponding to each phoneme subsequence in an orderly manner on the timeline, thereby splicing it into a set of continuous and seamless complete digital human image frame sequences, and realizing the final output of the digital human full-process lip rendering. Through the above method, dynamic scheduling and fine control of the segmented rendering mode are realized, thereby improving the adaptability to long voice input.

[0085] In some embodiments, see Figure 4During playback of a sequence of digital human image frames, the electronic device must sequentially write the image frames into a frame buffer for display. To improve rendering efficiency and reduce redundant computation, the electronic device can compare the current digital human image frame to be played with the previous frame before each frame is played to determine whether there are any areas where the image content has changed (S402). This identification of changed areas can be achieved through methods such as motion vector analysis and regional difference detection, thereby intelligently determining whether there are visual differences between the current digital human image frame to be played and the previous digital human image frame.

[0086] If the current digital human image frame to be played has a changed area compared to the previous digital human image frame, the electronic device updates the frame buffer based on the changed area (S403), that is, only performs image update operations on the changed area. For example, when updating the frame buffer, multi-layer Alpha blending operations are only performed on this area, and pixel-level processing can be accelerated in parallel through the ARM NEON instruction set.

[0087] If the current digital human image frame to be played does not have any changed region compared to the previous digital human image frame, the electronic device can directly reuse the previous digital human image frame (S404) without re-rendering, thus saving processing resources.

[0088] In the image display output (S405) stage, the system adopts a direct frame buffer write strategy and combines it with a multi-core scheduling mechanism to ensure that the image update task is precisely aligned with the display refresh cycle in time, thereby effectively avoiding the occurrence of screen tearing.

[0089] This embodiment updates changed areas on demand, avoiding full image redrawing for each frame. This significantly reduces pixel processing and blending overhead, particularly in consecutive frames with minimal image changes. On mobile or embedded devices, rendering and writing only to the changed areas effectively controls processor load and power consumption, extending battery life.

[0090] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0091] In some embodiments, an embodiment of this specification further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0092] Figure 5 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0093] In some embodiments, the digital human rendering method and apparatus can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this specification. The digital human rendering method and device may include:

[0094] The voice stream parsing module is used to parse the target voice stream that matches the digital human to be rendered to obtain the phoneme sequence and voice rhythm features.

[0095] The lip control module is used to determine a lip control parameter sequence corresponding to the phoneme sequence, wherein each lip control parameter in the lip control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence, and generate a lip control curve based on the lip control parameter sequence.

[0096] The lip-sync image block sequence generation module is used to perform rhythm alignment processing on the lip-sync control curve according to the speech rhythm characteristics, and perform rasterization conversion to generate a lip-sync image block sequence synchronized with the target speech stream.

[0097] The digital human image frame sequence generation module is used to synthesize each lip image block in the lip image block sequence with other image content of the digital human to generate a digital human image frame sequence.

[0098] In one implementation, the lip control module is specifically configured to: extract, from a pre-stored vector path library, curve control points associated with each lip control parameter in a lip control parameter sequence, to generate a lip control curve segment based on the curve control points associated with each lip control parameter; and splice the lip control curve segments corresponding to each lip control parameter according to the arrangement order of the phoneme sequence to generate a lip control curve.

[0099] In one implementation, the lip control module is specifically used to: perform interpolation processing based on the lip control curve segments corresponding to adjacent phonemes to generate connecting curve segments for smooth transition; and splice the lip control curve segments corresponding to each phoneme in the phoneme sequence and the connecting curve segments between adjacent phonemes according to the arrangement order of the phoneme sequence to generate a lip control curve.

[0100] In an implementation manner, the speech rhythm feature comprises a phoneme duration and / or a phoneme variation speed of each phoneme in the phoneme sequence.

[0101] The mouth shape image block sequence generation module is specifically configured to adjust a time span of the mouth shape control curve segment corresponding to each phoneme in the mouth shape control curve according to a phoneme duration and / or a phoneme variation speed of each phoneme in the phoneme sequence; wherein the phoneme duration of any phoneme is in a positive correlation with the time span of the mouth shape control curve segment corresponding to the phoneme; and the phoneme variation speed of any phoneme is in a negative correlation with the time span of the mouth shape control curve segment corresponding to the phoneme.

[0102] In an implementation manner, the rendering mode selection module is further configured to determine a current rendering mode from a real-time rendering mode and a non-real-time rendering mode based on the speech rhythm feature and / or idle computing resources of the rendering device. The mouth shape control curve is generated in a case that the current rendering mode is the real-time rendering mode.

[0103] In an implementation manner, the mouth shape image block sequence generation module is further configured to, if the current rendering mode is the non-real-time rendering mode, acquire a pre-stored compressed image resource, the compressed image resource comprising compressed mouth shape image blocks associated with different mouth shape control parameters; acquire, based on the mouth shape control parameters corresponding to each phoneme in the phoneme sequence, the compressed mouth shape image blocks corresponding to each phoneme from the compressed image resource; decompress the compressed mouth shape image blocks and arrange them in an arrangement order of the phoneme sequence to obtain the mouth shape image block sequence.

[0104] In an implementation manner, the rendering mode selection module is specifically configured to determine the current rendering mode as the real-time rendering mode if the speech rhythm feature satisfies a first real-time rendering condition and / or the idle computing resources of the rendering device satisfy a second real-time rendering condition; otherwise, determine the current rendering mode as the non-real-time rendering mode.

[0105] The speech rhythm feature comprises a phoneme variation speed; the first real-time rendering condition comprises that the phoneme variation speed is greater than a preset speed threshold; and / or the speech rhythm feature comprises a phoneme duration; and the first real-time rendering condition comprises that the phoneme duration is lower than a preset duration.

[0106] The second real-time rendering condition comprises that the idle computing resources of the rendering device are greater than a resource threshold, and the resource threshold is in a positive correlation with a duration of the target speech stream.

[0107] In one implementation, when the duration of the target speech stream is greater than a preset threshold, the target speech stream is divided into at least two speech segments; the phoneme sequence includes phoneme subsequences corresponding to each speech segment respectively; and the speech rhythm feature includes local speech rhythm features corresponding to each phoneme subsequence respectively.

[0108] The rendering mode selection module is specifically used to determine the rendering mode corresponding to each phoneme subsequence from the real-time rendering mode and the non-real-time rendering mode based on the local speech rhythm characteristics of each phoneme subsequence and / or the idle computing resources of the rendering device in the corresponding rendering period.

[0109] In one implementation, the system further includes a playback module configured to update the frame buffer based on the changed region if the current digital human image frame to be played has a changed region relative to the previous digital human image frame during the playback of the digital human image frame sequence; otherwise, the previous digital human image frame is reused.

[0110] In one implementation, the lip shape control parameter corresponding to each phoneme in the phoneme sequence is determined by a preset mapping relationship between the phoneme and the lip shape control parameter.

[0111] In one implementation, the speech rhythm features include phoneme-level rhythm features corresponding to each phoneme in the phoneme sequence. The lip shape control module is specifically configured to perform vectorization processing based on each phoneme in the phoneme sequence and the corresponding phoneme-level rhythm features to obtain a speech feature vector corresponding to each phoneme; obtain a pre-stored lip shape vector library, which is configured to store mappings between speech feature vectors and lip shape control parameters for phonemes in different languages ​​and / or with different pronunciations; and retrieve a lip shape control parameter sequence from the lip shape vector library based on the speech feature vectors of each phoneme in the phoneme sequence, wherein each lip shape control parameter in the lip shape control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence.

[0112] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0113] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0114] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0115] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0116] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A digital human rendering method, comprising: Parse the target speech stream that matches the digital human to be rendered to obtain the phoneme sequence and speech rhythm features; determining a lip control parameter sequence corresponding to the phoneme sequence, wherein each lip control parameter in the lip control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence, and generating a lip control curve based on the lip control parameter sequence; Performing rhythm alignment processing on the lip control curve according to the speech rhythm feature and performing rasterization conversion to generate a lip image block sequence synchronized with the target speech stream; Each lip shape image block in the lip shape image block sequence is synthesized with other image contents of the digital human to generate a digital human image frame sequence.

2. The method according to claim 1, wherein generating a lip control curve based on the lip control parameter sequence comprises: Extracting, from a pre-stored vector path library, curve control points associated with each lip control parameter in the lip control parameter sequence, so as to generate a lip control curve segment based on the curve control points associated with each lip control parameter; Interpolation processing is performed based on the mouth shape control curve segments corresponding to adjacent phonemes to generate connecting curve segments for smooth transition; According to the arrangement order of the phoneme sequence, the mouth shape control curve segments corresponding to the respective phonemes in the phoneme sequence and the connecting curve segments between adjacent phonemes are spliced ​​to generate the mouth shape control curve.

3. The method according to claim 2, wherein the speech rhythm feature comprises a phoneme duration and / or a phoneme change speed of each phoneme in the phoneme sequence; The performing rhythm alignment processing on the lip shape control curve according to the speech rhythm feature includes: adjusting the time span of the lip shape control curve segment corresponding to each phoneme in the lip shape control curve according to the phoneme duration and / or phoneme change speed of each phoneme in the phoneme sequence; Among them, the phoneme duration of any phoneme is positively correlated with the time span of the mouth shape control curve segment corresponding to the phoneme; The phoneme change speed of any phoneme is negatively correlated with the time span of the mouth shape control curve segment corresponding to the phoneme.

4. The method according to claim 1, before generating the lip control curve based on the lip control parameter sequence, further comprising: Determining a current rendering mode from a real-time rendering mode and a non-real-time rendering mode based on the speech rhythm feature and / or idle computing resources of a rendering device; wherein the lip control curve is generated when the current rendering mode is the real-time rendering mode; If the current rendering mode is the non-real-time rendering mode, obtain a pre-stored compressed image resource, wherein the compressed image resource includes compressed lip image blocks associated with different lip control parameters, and based on the lip control parameters corresponding to each phoneme in the phoneme sequence, obtain the compressed lip image block corresponding to each phoneme from the compressed image resource, decompress the compressed lip image block and arrange it according to the arrangement order of the phoneme sequence to obtain the lip image block sequence.

5. The method according to claim 4, wherein determining the current rendering mode from the real-time rendering mode and the non-real-time rendering mode based on the speech rhythm feature and / or the idle computing resources of the rendering device comprises: If the speech rhythm feature satisfies a first real-time rendering condition and / or the idle computing resources of the rendering device meet a second real-time rendering condition, determining that the current rendering mode is the real-time rendering mode; Otherwise, determining that the current rendering mode is the non-real-time rendering mode; The speech rhythm feature includes a phoneme change speed; the first real-time rendering condition includes: the phoneme change speed is greater than a preset speed threshold; and / or the speech rhythm feature includes a phoneme duration; the first real-time rendering condition includes: the phoneme duration is less than a preset duration; The second real-time rendering condition includes: idle computing resources of the rendering device are greater than a resource threshold, and the resource threshold is positively correlated with the duration of the target voice stream.

6. The method according to claim 1, further comprising: During the process of playing the digital human image frame sequence, if a digital human image frame to be played currently has a changed region relative to a previous digital human image frame, updating the frame buffer based on the changed region; Otherwise, reuse the previous digital human image frame.

7. The method according to claim 1, wherein the lip shape control parameter corresponding to each phoneme in the phoneme sequence is determined by a preset mapping relationship between the phoneme and the lip shape control parameter; or, The speech rhythm feature includes a phoneme-level rhythm feature corresponding to each phoneme in the phoneme sequence; The step of determining a lip control parameter sequence corresponding to the phoneme sequence comprises: Performing vectorization processing based on each phoneme in the phoneme sequence and the corresponding phoneme-level rhythmic features to obtain a speech feature vector corresponding to each phoneme; Obtaining a pre-stored lip shape vector library, wherein the lip shape vector library is used to store mapping relationships between speech feature vectors of phonemes in different languages ​​and / or with different pronunciations and lip shape control parameters; Based on the speech feature vector of each phoneme in the phoneme sequence, a lip shape control parameter sequence is retrieved from the lip shape vector library, and each lip shape control parameter in the lip shape control parameter sequence corresponds one-to-one to each phoneme in the phoneme sequence.

8. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images