3D digital human voice and mouth shape action synchronization method, device, equipment and medium

Abstract

The embodiment of the invention relates to the field of computer technology application, and discloses a 3D digital human voice and mouth shape action synchronization method and device, equipment and a medium, and the method comprises the steps: obtaining voice data, mouth shape data and limb data, so as to obtain a voice data queue, a mouth shape data queue and a limb data queue; according to the frame information in the mouth shape data queue and the frame information in the limb data queue, adjusting the volume of voice data sent by the voice data queue at a time; playing each frame of data in the mouth shape data queue and each frame of data in the limb data queue frame by frame; when the total amount of the voice data sent by the voice data queue is greater than or equal to a preset voice data volume threshold value, playing voice corresponding to the voice data sent by the voice data queue; according to the 3D digital human voice and mouth shape motion synchronization method disclosed by the invention, the technical problem of low authenticity caused by asynchronous motion and voice of the digital human is solved, so that the motion and voice of the digital human are synchronized, and the authenticity of the digital human is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology applications, and in particular to a method, device, equipment and medium for synchronizing 3D digital human voice and lip movements. Background Art

[0002] The emergence of the metaverse has triggered speculation about the next generation of the Internet era, providing a new track and opportunity for scientific and technological innovation and the development of information technology. As one of the important carriers of the metaverse, virtual digital humans are increasingly attracting the attention of relevant companies and researchers, mainly involving multiple technical fields such as computer graphics, computer vision, and machine learning. Virtual digital humans simulate real human images through algorithms and have various intelligent and interactive performances. They can be used in multiple fields such as games, film and television animation, advertising marketing, and digital human customer service. Whether the virtual digital human simulates the real image depends largely on whether the digital human can talk like a real person, whether the voice can be aligned with the lip shape and body movements, and whether the trigger is natural and coherent. The existing digital human lip synchronization technology does not do much additional processing. It simply plugs the data into the engine for rendering. This cannot achieve strict alignment, and is affected by the network environment. The fluctuation is relatively large, which will lead to the so-called "sound is faster than the mouth" or "mouth is before the sound" and other strange phenomena, which seriously affect the authenticity of the digital human. Summary of the invention

[0003] The purpose of the present invention is to at least provide a method, device, equipment and medium for synchronizing 3D digital human voice and lip movements, which can at least solve the technical problem that the digital human's movements and voice are not synchronized, resulting in low authenticity, and at least ensure that the digital human's movements and voice are synchronized, thereby improving the authenticity of the digital human.

[0004] To solve the above technical problems, at least one embodiment of the present invention provides a method for synchronizing 3D digital human voice and lip movements, comprising:

[0005] Acquire voice data, lip shape data and body shape data to obtain a voice data queue, a lip shape data queue and a body shape data queue;

[0006] According to the frame information in the lip data queue and the frame information in the body data queue, adjusting the amount of voice data sent at a time in the voice data queue;

[0007] Play each frame of data in the lip data queue and each frame of data in the limb data queue frame by frame;

[0008] When the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, the voice corresponding to the voice data sent by the voice data queue is played.

[0009] In some embodiments, adjusting the amount of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the limb data queue includes:

[0010] Extract frames from the lip data queue and the body data queue;

[0011] A preset field is added to each frame of data obtained by frame extraction; wherein the preset field is used to identify the frame sequence number, so as to identify the amount of data currently sent by the lip data queue or the limb data queue through the frame sequence number;

[0012] According to the amount of data currently sent by the lip data queue or the body data queue, the amount of voice data sent at a time by the voice data queue is adjusted.

[0013] In some embodiments, the acquiring of voice data, lip shape data and body shape data includes:

[0014] Three threads are enabled to asynchronously receive voice data, lip data and body data respectively;

[0015] Use asynchronous queues to cache received voice data, lip shape data, and body shape data.

[0016] In some embodiments, when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, playing the voice corresponding to the voice data sent by the voice data queue includes:

[0017] Use the preset audio buffer callback function to monitor the audio data in the buffer in real time;

[0018] If the amount of audio data in the buffer is greater than or equal to the preset voice data amount threshold, the voice data corresponding to the voice sent by the voice data queue is played.

[0019] In some embodiments, the voice data sent by the voice playing data queue corresponds to voice, including:

[0020] Synthesize the target voice with the voice data sent by the voice data queue;

[0021] Play the target voice.

[0022] In some embodiments, the preset audio buffer callback function includes: a PCMReadCallBack callback function.

[0023] In some embodiments, the voice data is byte stream data.

[0024] At least one embodiment of the present invention further provides a 3D digital human voice and lip motion synchronization device, comprising:

[0025] A data acquisition module, used for acquiring voice data, lip shape data and body shape data, so as to obtain a voice data queue, a lip shape data queue and a body shape data queue;

[0026] A data volume adjustment module, used for adjusting the volume of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the body data queue;

[0027] The first data playing module is used to play each frame of data in the lip data queue and each frame of data in the limb data queue frame by frame;

[0028] The second data playing module is used to play the voice corresponding to the voice data sent by the voice data queue when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold.

[0029] At least one embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned 3D digital human voice and lip movement synchronization method.

[0030] At least one embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned method for synchronizing 3D digital human speech and lip movements.

[0031] The embodiments of the present invention provide a method, device, equipment and medium for synchronizing 3D digital human voice and lip movements, which obtain voice data, lip movement data and limb data to obtain a voice data queue, a lip movement data queue and a limb data queue; adjust the amount of voice data sent by the voice data queue at a time according to the frame information in the lip movement data queue and the frame information in the limb data queue; play each frame of data in the lip movement data queue and each frame of data in the limb data queue frame by frame; when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, play the voice corresponding to the voice data sent by the voice data queue; the synchronization method has good generalization, as long as the skeleton of the digital human model meets the CS standard and the shape key meets the ARKit standard, the method can be used for alignment control and driving; it can ensure that the action and voice of the digital human are not synchronized, thereby improving the authenticity of the digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0033] Figure 1is a flow chart of a method for synchronizing 3D digital human speech and lip movements provided by an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of an overall framework provided by another embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of an implementation flow provided by another embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of a 3D digital human voice and lip motion synchronization device provided by another embodiment of the present invention;

[0037] Figure 5 It is a schematic structural diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present invention can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0039] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by technicians in the relevant technical field without creative work are within the scope of protection of the present invention.

[0040] It is worth noting that in the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the present invention.

[0041] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. It should also be understood that when the term "and / or" is used in the present specification and the appended claims, it refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0042] In the description of the present specification and the appended claims, if the term "if" is used, it can be interpreted as "in the case of" or "when" or "upon" or "in response to determining" or "in response to detecting" according to the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determination" or "upon detection of [the described condition or event]" or "in response to detecting [the described condition or event]" according to the context.

[0043] In the description of the present specification and the appended claims, the terms "first", "second", "third", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance, nor are they used to describe a specific order or sequence.

[0044] In order to facilitate understanding of the embodiments of the present invention, relevant contents about 3D digital human are first introduced here.

[0045] The emergence of the metaverse has triggered speculation about the next generation of the Internet era, providing a new track and opportunity for scientific and technological innovation and the development of information technology. As one of the important carriers of the metaverse, virtual digital humans are increasingly attracting the attention of relevant companies and researchers, mainly involving multiple technical fields such as computer graphics, computer vision, and machine learning. Virtual digital humans simulate real human images through algorithms and have various intelligent and interactive performances. They can be used in multiple fields such as games, film and television animation, advertising marketing, and digital human customer service. Whether the virtual digital human simulates the real image depends largely on whether the digital human can talk like a real person, whether the voice can be aligned with the lip shape and body movements, and whether the trigger is natural and coherent. The existing digital human lip synchronization technology does not do much additional processing. It simply plugs the data into the engine for rendering. This cannot achieve strict alignment, and is affected by the network environment. The fluctuation is relatively large, which will lead to the so-called "sound is faster than the mouth" or "mouth is before the sound" and other strange phenomena, which seriously affect the authenticity of the digital human.

[0046] In order to solve the above-mentioned technical problem of the 3D digital human mouth shape, body movements and audio being out of sync, the present invention proposes a method for synchronizing 3D digital human voice and lip movements. The implementation details of the 3D digital human voice and lip movement synchronization method of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0047] Embodiment 1:

[0048] The 3D digital human voice and lip motion synchronization method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 As shown, including:

[0049] Step 100, acquiring voice data, lip shape data and body shape data to obtain a voice data queue, a lip shape data queue and a body shape data queue.

[0050] Specifically, in a streaming network environment, a 3D digital human voice and lip-syncing synchronization method is developed to align the digital human voice with lip-syncing and body movements in real time. The synchronization method mainly includes two modules: a streaming data processing framework and a synchronization mechanism based on frame alignment. The overall framework is as follows: Figure 2 shown.

[0051] Module 1, the streaming data processing framework, is mainly used to process the solution data transmitted from the upstream in a streaming manner in a real-time network environment, and use asynchronous queues to cache to ensure that the data will not be lost. For the data from the three input sources, namely, voice data, lip data, and body data, three threads are enabled to receive them asynchronously to ensure that the data processing will not be delayed or waited due to network delays.

[0052] The streaming network transmission framework can avoid sending all data at once, which will cause long processing time and cause front-end lag. It can also dynamically adjust the chunksize of the sent data according to the consumption of the data buffer in real time. It is one of the essential frameworks for 3D real-time interactive digital humans.

[0053] Chunksize refers to a strategy for dividing data into fixed-size chunks for processing during data transmission or processing. Its application scenarios are as follows:

[0054] Data storage: In cloud storage or local storage systems, chunksize is used to split large files into smaller chunks for storage. This allows for efficient management and storage of large-scale data, while facilitating incremental backup and recovery of data. For example, Tencent Cloud's object storage (COS) provides a chunk upload function, and users can set the appropriate chunksize as needed.

[0055] Data transmission: In network transmission, chunksize is used to split large files into multiple small chunks for transmission to improve transmission speed and stability. This method can avoid the problem of having to retransmit the entire file when a single large file fails to be transmitted.

[0056] Data processing: In big data processing, especially when using data processing libraries such as Pandas, chunksize can be used to split large data sets into small chunks for parallel processing or batch processing to improve processing efficiency and reduce computing costs.

[0057] The advantages are as follows:

[0058] Improve processing efficiency: By dividing large data sets into small blocks for processing, you can make full use of distributed computing resources and speed up data processing.

[0059] Reduce memory consumption: When processing large-scale data, directly operating on the entire data set may lead to insufficient memory. Using the chunksize parameter can process only a small piece of data at a time, thus effectively reducing memory consumption.

[0060] Enhanced system stability: If an error or interruption occurs during network transmission or data storage, only the erroneous block needs to be retransmitted or processed, without reprocessing the entire data set.

[0061] The setting principles are as follows:

[0062] Set according to the data size: For larger data sets, you can set a larger chunksize to speed up processing; for smaller data sets, you can appropriately reduce the chunksize to avoid unnecessary memory consumption.

[0063] Consider system resources: When setting the chunksize, you need to fully consider the system's memory, CPU and other resources to ensure that the system can operate normally and process data.

[0064] Adjust according to business needs: Different business needs have different requirements for the speed and accuracy of data processing. Therefore, when setting the chunksize, you need to adjust and optimize it according to specific business needs.

[0065] Step 200, adjusting the amount of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the body data queue.

[0066] Specifically, in module 2, the synchronization mechanism based on frame alignment, the Audio2Face and Audio2Gesture resolved data are both data segments, which contain all the data after a text segment is resolved, while the voice data is a byte stream buffer. How to align these two data of different dimensions is the key issue to be solved in this module. To this end, in this embodiment, a synchronization mechanism based on frame alignment is designed. Specifically, the resolved data segment can be framed, and a meta field is added to each frame to identify the frame number, and the frame number is used to identify the amount of data currently sent. Because the frame rate is fixed, the number of frames and the single frame time can be used to deduce the amount of data currently sent, and the chunksize of the audio buffer sent at a time can be dynamically adjusted based on this result.

[0067] In computer science, "buffer" is often used to describe an area in memory that is used to temporarily store data to reduce the number of accesses to the hard disk or other I / O devices and improve system efficiency. This buffer is usually used for temporary storage of data transmission to ensure stable data transmission and integrity.

[0068] Step 300, playing each frame of data in the lip data queue and each frame of data in the body data queue frame by frame.

[0069] Specifically, each frame of data in the lip data queue and each frame of data in the limb data queue can be played frame by frame.

[0070] Step 400: When the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, play the voice corresponding to the voice data sent by the voice data queue.

[0071] Specifically, at the data receiving end, for audio buffer data, in this embodiment, a PCM data format audio buffer callback function is designed to monitor the audio data in the buffer in real time and process it. For the resolved Audio2Face and Audio2Gesture data, the resolved data is still processed in a similar way to the audio data, except that the object of the operation is now an asynchronous queue. For voice, Audio2Face lip data, and Audio2Gesture body data, the frame number can be used to determine the playback order and timing to maintain audio and video synchronization.

[0072] In this embodiment, voice data, lip data and limb data are obtained to obtain a voice data queue, a lip data queue and a limb data queue; the amount of voice data sent by the voice data queue at a time is adjusted according to the frame information in the lip data queue and the frame information in the limb data queue; each frame of data in the lip data queue and each frame of data in the limb data queue are played frame by frame; when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, the voice corresponding to the voice data sent by the voice data queue is played; the synchronization method has good generalization, as long as the skeleton of the digital human model meets the CS standard and the shape key meets the ARKit standard, the method can be used for alignment control and driving; it can ensure that the movements of the digital human are synchronized with the voice, thereby improving the authenticity of the digital human.

[0073] ARKit standards cover technical foundations and functional features, usability and development support, performance and optimization, application scenarios and cases, and user experience design principles. These standards together constitute the core competitiveness of ARKit and provide developers with powerful tools to create engaging AR experiences.

[0074] In some embodiments, adjusting the amount of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the body data queue includes:

[0075] Extract frames from the lip data queue and the limb data queue.

[0076] A preset field is added to each frame of data obtained by frame extraction; wherein the preset field is used to identify the frame sequence number, so as to identify the amount of data currently sent by the lip data queue or the limb data queue through the frame sequence number.

[0077] According to the amount of data currently sent by the lip data queue or the body data queue, the amount of voice data sent at a time by the voice data queue is adjusted.

[0078] In some embodiments, obtaining voice data, lip shape data, and body shape data includes:

[0079] Three threads are enabled to asynchronously receive voice data, lip data and body data respectively.

[0080] Use asynchronous queues to cache received voice data, lip shape data, and body shape data.

[0081] In some embodiments, when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, playing the voice corresponding to the voice data sent by the voice data queue includes:

[0082] Use the preset audio buffer callback function to monitor the audio data in the buffer in real time.

[0083] If the amount of audio data in the buffer is greater than or equal to the preset voice data amount threshold, the voice data corresponding to the voice sent by the voice data queue is played.

[0084] In some embodiments, playing the voice data corresponding to the voice sent by the voice data queue includes:

[0085] Synthesize the voice data sent by the voice data queue into the target voice.

[0086] Play the target voice.

[0087] Specifically, in the above embodiments, Figure 2 The framework shown is an abstract outline of the present invention. The Unity engine needs to be used for front-end rendering and presentation during the entire process.

[0088] like Figure 3 As shown, the specific implementation steps are as follows:

[0089] First, write a script in Unity to build a socket communication framework for real-time receiving voice, lip shape and motion data sent from the cloud. Data from three different sources can be processed using three different threads to avoid congestion to the greatest extent. At the same time, use asynchronous queues to save data in separate queues to prevent loss.

[0090] Second, write a script in Unity to synchronously process the received three-track data and render it to the front-end display. Specifically, first, set certain limits on the frame rate of the entire process, and keep it consistent with the audio2face frame rate, so that the entire process is the most stable; in the received data stream, only the voice data is a byte stream buffer, and the other data are essentially float-type floating-point numbers, so the voice data needs to be processed separately. We put it in a callback function called PCMReadCallBack, which monitors the voice buffer in real time. When the data in it meets a certain chunksize, it will read the data, synthesize the voice, and then play the sound. Source; For audio2face data and audio2gesture data, the entire logic processing is in the entire large while monitoring loop, ensuring that when data comes in, it can be processed in time. When the entire program exits, the while monitoring loop will terminate; The logic of the while monitoring loop is actually very simple. Within a given deltatime time interval, the received data can be rendered frame by frame, and the voice broadcast is played according to chunsize. We only need to set the size of chunsize to the amount of voice data that can be processed within deltatime according to the actual sampling rate of the voice, so that the audio and video can be synchronized. On this basis, in order to ensure that the program runs more stably, we have also added some synchronization variables and boundary condition processing logic.

[0091] This embodiment provides a method and synchronization strategy for aligning the three-track data of 3D real-time interactive digital human voice, lip shape and body movement in a streaming network space. The method can also be applied to offline scenarios, such as digital human broadcasting. The method mainly includes two modules: a data streaming processing module and a data alignment control module. The former can be divided into three small parts, which respectively process the voice, lip shape and body movement data transmitted frame by frame in the network space. The voice data comes from TTS, the lip shape data comes from the solution of Audio2Face, and the body movement data comes from our pre-built digital human body movement library (Audio2Gesture solution); the latter mainly performs real-time alignment and dynamic frame control for the incoming three-track data. The synchronization method has good generalization. As long as the skeleton of the digital human model meets the CS standard and the shape key meets the ARKit standard, the method can be used for alignment control and driving.

[0092] Embodiment 2:

[0093] Another embodiment of the present invention relates to a 3D digital human voice and lip-syncing device. The implementation details of the 3D digital human voice and lip-syncing device of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details, which is not necessary for the implementation of this solution. The schematic diagram of the 3D digital human voice and lip-syncing device of this embodiment can be as follows: Figure 4 As shown, it includes a data acquisition module 401, a data amount adjustment module 402, a first data playback module 403 and a second data playback module 404.

[0094] The data acquisition module 401 is used to acquire voice data, lip shape data and body shape data to obtain a voice data queue, a lip shape data queue and a body shape data queue.

[0095] The data volume adjustment module 402 is used to adjust the volume of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the body data queue;

[0096] The first data playing module 403 is used to play each frame of data in the lip data queue and each frame of data in the body data queue frame by frame;

[0097] The second data playing module 404 is configured to play the voice corresponding to the voice data sent by the voice data queue when the total amount of the voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold.

[0098] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0099] Embodiment three:

[0100] Another embodiment of the present invention relates to an electronic device, such as Figure 5 As shown, it includes: at least one processor 501; and a memory 502 that is communicatively connected to the at least one processor 501; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 so that the at least one processor 501 can execute the 3D digital human voice and lip movement synchronization method in the above-mentioned embodiments.

[0101] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0102] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0103] In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with a wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer, a television, a server, etc.

[0104] Exemplarily, an embodiment of the present invention provides a feasible structure of an electronic device. For example, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface (hereinafter referred to as the USB interface), a charging management module, a power management module, a battery, a first antenna, a second antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera, a display screen, and a subscriber identification module (SIM) card interface (hereinafter referred to as the SIM card interface), etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0105] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present invention, the electronic device may include more or fewer components than the above structure, or combine some components, or split some components, or arrange the components differently. The above components may be implemented in hardware, software, or a combination of software and hardware.

[0106] The processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0107] Among them, the controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and execution.

[0108] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves the efficiency of the system.

[0109] Embodiment 4:

[0110] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0111] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0112] The above-mentioned non-transitory computer-readable storage medium can be set in an electronic device. Specifically, the electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention. The processor is a processor in any of the above-mentioned electronic devices.

[0113] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for synchronizing 3D digital human speech and lip movements, characterized in that: include: Acquire voice data, lip shape data and body shape data to obtain a voice data queue, a lip shape data queue and a body shape data queue; According to the frame information in the lip data queue and the frame information in the body data queue, adjusting the amount of voice data sent at a time in the voice data queue; Play each frame of data in the lip data queue and each frame of data in the limb data queue frame by frame; When the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, the voice corresponding to the voice data sent by the voice data queue is played.

2. The method for synchronizing 3D digital human voice and lip movements according to claim 1, characterized in that: The method of adjusting the amount of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the limb data queue comprises: Extract frames from the lip data queue and the body data queue; A preset field is added to each frame of data obtained by frame extraction; wherein the preset field is used to identify the frame sequence number, so as to identify the amount of data currently sent by the lip data queue or the limb data queue through the frame sequence number; According to the amount of data currently sent by the lip data queue or the body data queue, the amount of voice data sent at a time by the voice data queue is adjusted.

3. The method for synchronizing 3D digital human voice and lip movements according to claim 1, characterized in that: The acquiring of voice data, lip shape data and body shape data comprises: Three threads are enabled to asynchronously receive voice data, lip data and body data respectively; Use asynchronous queues to cache received voice data, lip shape data, and body shape data.

4. The method for synchronizing 3D digital human voice and lip movements according to claim 1, characterized in that: When the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold, playing the voice corresponding to the voice data sent by the voice data queue includes: Use the preset audio buffer callback function to monitor the audio data in the buffer in real time; If the amount of audio data in the buffer is greater than or equal to the preset voice data amount threshold, the voice data corresponding to the voice sent by the voice data queue is played.

5. The method for synchronizing 3D digital human voice and lip movements according to claim 4, characterized in that: The voice data corresponding to the voice sent by the voice playing data queue includes: Synthesize the target voice with the voice data sent by the voice data queue; Play the target voice.

6. The method for synchronizing 3D digital human voice and lip movements according to claim 4, characterized in that: The preset audio buffer callback function includes: a PCMReadCallBack callback function.

7. The method for synchronizing 3D digital human voice and lip movements according to claim 1, characterized in that: The voice data is byte stream data.

8. A 3D digital human voice and lip motion synchronization device, characterized in that: include: A data acquisition module, used for acquiring voice data, lip shape data and body shape data, so as to obtain a voice data queue, a lip shape data queue and a body shape data queue; A data volume adjustment module, used for adjusting the volume of voice data sent at a time by the voice data queue according to the frame information in the lip data queue and the frame information in the body data queue; The first data playing module is used to play each frame of data in the lip data queue and each frame of data in the limb data queue frame by frame; The second data playing module is used to play the voice corresponding to the voice data sent by the voice data queue when the total amount of voice data sent by the voice data queue is greater than or equal to a preset voice data amount threshold.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the 3D digital human voice and lip movement synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for synchronizing the speech and lip movements of a 3D digital human according to any one of claims 1 to 7 is implemented.

Images