Voice-based 3D digital population type driving method and device, equipment and medium

Abstract

The invention discloses a voice-based 3D digital population type driving method and device, equipment and a medium, and the method comprises the steps: obtaining target voice data; inputting the target voice data into a target analysis model to obtain target driving sequence data; the target analysis model is used for determining the target driving sequence data based on the target voice data, and the feature dimension corresponding to the target driving sequence data corresponds to the mouth feature dimension of the 3D digital human; and based on the target driving sequence data, carrying out rendering driving on the 3D digital human, so that the accuracy of the 3D digital human shape can be effectively improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing technology, and in particular to a voice-based 3D digital population driving method, device, equipment and medium. Background Art

[0002] The metaverse describes a colorful virtual world for people. As the master of this virtual world, digital people are receiving more and more attention. Digital people, as digital characters created by digital technology and close to human image and behavior, will replace humans to complete life production in the virtual world. At present, the voice model, which is the soul of digital people, has made rapid progress, such as GPT-4, Ernie 3.0Titan and ERNIE Bot, which have given digital people powerful language comprehension ability and put forward higher requirements on how to express knowledge content more accurately and elegantly. Speaking is a basic ability of digital people, and their lip shape must be accurately matched to better communicate life in the virtual world. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a voice-based 3D digital population driving method, device, equipment and medium, which can effectively improve the accuracy of the 3D digital population.

[0004] In a first aspect, an embodiment of the present application provides a voice-based 3D digital population driving method, comprising:

[0005] Obtain target voice data;

[0006] The target speech data is input into a target analysis model to obtain target drive sequence data; the target analysis model is used to determine the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the mouth feature dimension of the 3D digital human;

[0007] Based on the target driving sequence data, the 3D digital human is rendered and driven.

[0008] In some embodiments, before inputting the target speech data into the target analysis model to obtain the target driving sequence data, the method further includes:

[0009] Obtain sampled voice data;

[0010] Extracting speech features from the sampled speech data to obtain sampled speech features and corresponding sampled driving sequence data;

[0011] The target analysis model is trained based on the corresponding frame data of the sampled drive sequence data and the sampled voice data to obtain the target analysis model.

[0012] In some embodiments, the extracting speech features from the sampled speech data to obtain the sampled speech features and the corresponding sampled driving sequence data includes:

[0013] converting the sampled speech data into original drive sequence data, and extracting the sampled drive sequence data from the original drive sequence; or

[0014] The sampled speech data is sequentially input into a feature extraction layer, a linear interpolation layer, a Transformer layer and an MLP layer to obtain the sampled driving sequence data.

[0015] In some embodiments, the training of the target analysis model based on the corresponding frame data of the sampled driving sequence data and the sampled speech data includes:

[0016] Acquire the corresponding frame data of the sampled voice data;

[0017] extracting standard drive sequence data corresponding to the number of dimensions of the sampled drive sequence data from the corresponding frame data;

[0018] Based on the sampled driving sequence data and the standard driving sequence data, a correction sequence is performed on the target analysis model.

[0019] In some embodiments, the obtaining of frame data corresponding to the sampled speech data includes:

[0020] Performing facial image rendering based on the original drive sequence data to obtain corresponding frame data of the sampled voice data; or

[0021] The video data corresponding to the sampled voice data is obtained, and the video data is frame-aligned with the sampled voice data to obtain the frame data corresponding to the sampled voice data.

[0022] In some embodiments, the target drive sequence data are mouth feature points, and the number of dimensions of the target drive sequence data is 37; the original drive sequence data are facial feature points, and the number of dimensions of the original drive sequence data is 52.

[0023] In a second aspect, an embodiment of the present application provides a voice-based 3D digital population-type driving device, comprising:

[0024] An acquisition module, used to acquire target voice data;

[0025] An analysis module, used for inputting the target speech data into a target analysis model to obtain target drive sequence data; the target analysis model is used for determining the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the feature dimension of the mouth of the 3D digital human;

[0026] A rendering module is used to drive the 3D digital human based on the target driving sequence data.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the program.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the embodiment of the present application.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method described in the embodiment of the present application.

[0030] The speech-based 3D digital human mouth shape driving method proposed in the embodiment of the present application can use the target analysis model to determine the target driving sequence data required for the mouth shape that matches the target speech data to be "spoken" by the 3D digital human, and then render and drive the 3D digital human according to the target driving sequence data, so that the 3D digital human can have a more standard mouth shape and a more natural posture when "speaking" the target speech data.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0033] Figure 1 A schematic diagram of the process of a voice-based 3D digital population driving method provided in an embodiment of the present application is shown;

[0034] Figure 2 A schematic diagram of a process of a voice-based 3D digital population driving method provided in another embodiment of the present application is shown;

[0035] Figure 3 A schematic diagram of the structure of a target analysis model provided by an embodiment of the present application is shown;

[0036] Figure 4 A flowchart of obtaining sampling drive sequence data by the Audio2Face tool provided in an embodiment of the present application is shown;

[0037] Figure 5 A block diagram of a voice-based 3D digital population-type driving device provided in an embodiment of the present application is shown;

[0038] Figure 6 A schematic diagram of the structure of a computer system of an electronic device or server suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] In terms of 3D digital population driving, the relevant technology provides two processing methods. One is based on phoneme driving, which requires the production of viseme assets. These viseme assets need to be designed in advance in combination with language characteristics. There is no unified standard for production, and developers need to have a certain level of artistic skills. The requirements for developers are high, and it is difficult to achieve the generalization of 3D digital human applications. The other is based on facial expression driving. Currently, facial expression driving is more widely used in the industry. However, based on the current facial driving, 3D digital humans do not do much additional processing. They simply plug the data into the engine for rendering. This usually makes the 3D digital human's mouth shape weird when "speaking", which will cause the mouth to open too wide, the mouth shape cannot be closed, and other strange phenomena, which seriously affect the authenticity of the digital human.

[0042] In order to further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation instruction steps shown in the following embodiments or drawings, more or less operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method may be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the device is executed.

[0043] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for driving a 3D digital population based on voice according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0044] Step 101, obtaining target voice data.

[0045] It should be noted that the target voice data is the voice information to be "spoken" by the 3D digital human. In the embodiment of the present application, the corresponding lip-driven data is determined for the 3D digital human based on the target voice data, so that the 3D digital human's form is more accurate and natural when "speaking".

[0046] Step 102, input the target speech data into the target analysis model to obtain the target drive sequence data.

[0047] Among them, the target analysis model is used to determine the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the mouth feature dimension of the 3D digital human.

[0048] That is to say, in the embodiment of the present application, the trained target analysis model is used to convert the target voice data to be played into the target driving sequence data for driving the 3D digital population.

[0049] It should be noted that the target driving sequence data is a sequence of feature values ​​corresponding to at least one feature point of the 3D digital human body in a preset order, including but not limited to the position information corresponding to the feature parts such as the corners of the mouth and the peaks of the lips, or other information data that can indicate the feature parts when the image is rendered. Compared with the traditional facial blending animation (Blendshape), it effectively reduces the amount of data used to drive the 3D digital human to "speak", and the use of the target analysis model for analysis can provide more accurate driving features for the mouth, thereby improving the accuracy of the 3D digital human body.

[0050] Step 103: Rendering and driving the 3D digital human based on the target driving sequence data.

[0051] It should be noted that the target drive sequence data is the mouth feature value corresponding to when the 3D digital human "speaks" the target voice data. Therefore, by rendering the image of the mouth area of ​​the 3D digital human according to the target drive sequence data, the 3D digital human can "speak" the target voice data using the mouth shape that is suitable for the target voice data.

[0052] Specifically, the video compilation tool can render the facial information of the 3D digital human based on the target drive sequence data, and generate the final 3D digital human video by combining the target voice data. For example, the ffmpeg tool can be used to obtain the 3D digital human video containing voice and image.

[0053] Therefore, the speech-based 3D digital human mouth shape driving method proposed in the embodiment of the present application can use the target analysis model to determine the target driving sequence data required for the mouth shape to match the target speech data to be "spoken" by the 3D digital human, and then render and drive the 3D digital human according to the target driving sequence data, so that the 3D digital human's mouth shape will be more standard and the posture will be more natural when "speaking" the target speech data.

[0054] In a feasible embodiment, before inputting the target speech data into the target analysis model to obtain the target driving sequence, the target analysis model needs to be trained, such as Figure 2 As shown, specifically including:

[0055] Step 201, obtaining sampled voice data.

[0056] It should be noted that the sampled speech data used to train the target analysis model is noise-free speech data. Preferably, it is speech data with standard pronunciation, such as audio data of news broadcasts, and speech data generated by a trained speech synthesis tool (such as a TTS tool).

[0057] Optionally, in order to further ensure that the number of samples for training the target analysis model is sufficient, the total duration of the sampled speech data is greater than 2 hours.

[0058] In a specific embodiment, about 3 hours of news broadcast video is obtained, voice data in the news broadcast video is extracted, and human voice detection and denoising are performed on the voice data to obtain sampled voice data. In addition, the text of Chinese textbooks in primary and secondary schools is converted into sampled voice data using a TTS tool. A total of 7.5 hours of sampled voice data is obtained.

[0059] Step 202: extract speech features from the sampled speech data to obtain the sampled speech features and their corresponding sampled driving sequence data.

[0060] In the embodiment of the present application, the purpose of extracting speech features from sampled speech data is to understand the contextual relationship between speech data, that is, through the analysis of speech features, to fully understand the correlation between long-term audio contexts, so as to improve the coherence and accuracy of lip shape features.

[0061] In a possible embodiment, if Figure 3As shown, the sampled speech data is sequentially input into the feature extractor layer, the linear interpolation layer, the Transformer Encoder layer and the MLP layer to obtain the sampled driving sequence data.

[0062] Among them, the feature extraction (Feature Extractor) layer is used to extract specific speech features from the sampled speech data, the linear interpolation (Linear) layer is used to calculate the frame rate (fps) of the sampled speech data per second, so as to determine the number of facial images corresponding to the audio per second, and then determine the sampled driving sequence data corresponding to each image, and the Transformer Encoder layer and the MLP layer are used to comprehensively analyze the semantic association relationship and emotional association relationship between the sampled speech data context. After the sampled speech data passes through the above model structure, the 37-dimensional sampled driving sequence data can be obtained.

[0063] In a specific embodiment, feature extraction can use model structures such as Hubert, MFCC, FBank and DeepSpeech, among which the features extracted using the Hubert model structure have better robustness and stability. The loss function used in the training process is soft-DTW. At the same time, in order to improve the reliability of training, auxiliary training functions such as Droput, VelLoss and AccLoss can also be added.

[0064] In another specific embodiment, Figure 4 As shown, the above process can be completed using an automated script. For example, open the Omniverse Audio2Face tool, click the claire_solved_arkit project, set the Playback FPS to 25.0 in the A2F Data Conversion column, select / World / audio2face / BlendshapeSolve, and click Export Weights to output a json file. The content of the file is the 52-dimensional original drive sequence data corresponding to the input voice. Then, extract the 37-dimensional sampled drive sequence data from the 52-dimensional original drive sequence data.

[0065] It should be noted that both the sampled drive sequence data and the target drive sequence data are 37-dimensional, that is, the number of dimensions corresponding to the regional features of the 3D digital population. Specifically, Table 1 is a comparison table of 37-dimensional sampled drive sequence data identifiers and 52-dimensional original drive sequence data identifiers.

[0066] Table 1

[0067]

[0068] 37-dimensional sampled drive sequence data identification 52-dimensional original drive sequence data identification

[0069] Step 203, based on the corresponding frame data of the sampled driving sequence data and the sampled speech data, the target analysis model is trained to obtain the target analysis model.

[0070] It should be noted that the corresponding frame data of the sampled voice data is the face video data corresponding to the voice data.

[0071] In some embodiments, face rendering can be performed based on the original drive sequence data to obtain the corresponding frame data of the sampled voice data. Specifically, after obtaining the frame rate per second fps, full face rendering can be performed based on the original drive sequence data corresponding to at least one frame per second to obtain multiple corresponding frame data corresponding to the sampled voice data; or, the Audio2Face tool can be directly used to render the corresponding image frame. Preferably, it can be used to generate the corresponding frame data of Chinese texts for primary and secondary schools using the TTS tool.

[0072] In other embodiments, the video data corresponding to the sampled voice data is obtained, and the video data and the sampled voice data are frame aligned to obtain the corresponding frame data of the sampled voice data. Preferably, it can be used to generate the corresponding frame data of the news broadcast.

[0073] After obtaining the corresponding frame data of the sampled voice data, the standard drive sequence data corresponding to the number of dimensions of the sampled drive sequence data is extracted from the corresponding frame data. When the corresponding frame data is obtained using the original drive sequence data, the standard drive sequence data can be directly extracted from the original drive sequence data; when the corresponding frame data is obtained using the video data, it is necessary to perform frame-by-frame video analysis on the video data to obtain the standard drive sequence data corresponding to each corresponding frame data.

[0074] Then, based on the sampled drive sequence data and the standard drive sequence data, the target analysis model is trained, that is, the gap between the sampled drive sequence data and the standard drive sequence data is gradually reduced through iteration until the target analysis model is obtained.

[0075] Therefore, the speech-based 3D digital population shape driving method proposed in the embodiment of the present application can use real video data to train the target analysis model to ensure the accuracy and reliability of the target driving sequence data output by the target analysis model, or use the corresponding frame data generated based on the 52-dimensional feature information of the whole face to train the 37-dimensional mouth features, so that the target analysis model can still output highly accurate and reliable target driving sequence data that conforms to the lip shape rules when the feature dimension is reduced.

[0076] To sum up, the speech-based 3D digital human mouth shape driving method proposed in the embodiment of the present application can use the target analysis model to determine the target driving sequence data required for the mouth shape that matches the target speech data to be "spoken" by the 3D digital human, and then render and drive the 3D digital human according to the target driving sequence data, so that the 3D digital human's mouth shape will be more standard and the posture will be more natural when "speaking" the target speech data.

[0077] It should be noted that although the operations of the method of the present invention are described in a specific order in the drawings,

[0078] However, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed, to achieve desirable results.

[0079] Figure 5 A block diagram of a voice-based 3D digital population-type driving device provided in an embodiment of the present application is shown.

[0080] like Figure 5 As shown, the speech-based 3D digital population-type driving device 10 comprises:

[0081] An acquisition module 11 is used to acquire target voice data;

[0082] The analysis module 12 is used to input the target speech data into a target analysis model to obtain target drive sequence data; the target analysis model is used to determine the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the mouth feature dimension of the 3D digital human;

[0083] The rendering module 13 is used to drive the 3D digital human based on the target driving sequence data.

[0084] In some embodiments, the analysis module 12 is further configured to:

[0085] Obtain sampled voice data;

[0086] Extracting speech features from the sampled speech data to obtain sampled speech features and corresponding sampled driving sequence data;

[0087] The target analysis model is trained based on the corresponding frame data of the sampled drive sequence data and the sampled voice data to obtain the target analysis model.

[0088] In some embodiments, the analysis module 12 is further configured to:

[0089] converting the sampled speech data into original drive sequence data, and extracting the sampled drive sequence data from the original drive sequence; or

[0090] The sampled speech data is sequentially input into a feature extraction layer, a linear interpolation layer, a Transformer layer and an MLP layer to obtain the sampled driving sequence data.

[0091] In some embodiments, the analysis module 12 is further configured to:

[0092] Acquire the corresponding frame data of the sampled voice data;

[0093] extracting standard drive sequence data corresponding to the number of dimensions of the sampled drive sequence data from the corresponding frame data;

[0094] Based on the sampled driving sequence data and the standard driving sequence data, a correction sequence is performed on the target analysis model.

[0095] In some embodiments, the analysis module 12 is further configured to:

[0096] Performing facial image rendering based on the original driving sequence data to obtain corresponding frame data of the sampled voice data; or

[0097] The video data corresponding to the sampled voice data is obtained, and the video data is frame-aligned with the sampled voice data to obtain the frame data corresponding to the sampled voice data.

[0098] In some embodiments, the target drive sequence data are mouth feature points, and the number of dimensions of the target drive sequence data is 37; the original drive sequence data are facial feature points, and the number of dimensions of the original drive sequence data is 52.

[0099] It should be understood that the modules or modules described in the speech-based 3D digital population type driving device 10 are similar to those described in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the voice-based 3D digital population-type driving device 10 and the modules contained therein, and will not be repeated here. The voice-based 3D digital population-type driving device 10 can be pre-implemented in the browser or other security applications of the electronic device, and can also be loaded into the browser or its security application of the electronic device by downloading or the like. The corresponding modules in the voice-based 3D digital population-type driving device 10 can cooperate with the modules in the electronic device to implement the solution of the embodiment of the present application.

[0100] For the several modules or units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0101] Reference below Figure 5 , Figure 5 A schematic diagram of the structure of a computer system of an electronic device or server suitable for implementing an embodiment of the present application is shown.

[0102] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation instructions of the system are also stored. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0103] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0104] In particular, according to an embodiment of the present application, the above reference flow chart Figure 2 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0105] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0106] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The units or modules involved in the embodiments of the present application may be implemented by software or hardware. The units or modules described may also be provided in a processor, for example, may be described as: a processor includes an acquisition module, an analysis module, and a rendering module. The names of these units or modules do not, in certain cases, constitute limitations on the units or modules themselves, for example, an acquisition module may also be described as "acquiring target voice data".

[0108] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the voice-based 3D digital population driving method described in the present application.

[0109] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A 3D digital population driving method based on speech, characterized in that: include: Obtain target voice data; Inputting the target speech data into a target analysis model to obtain target drive sequence data; The target analysis model is used to determine the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the mouth feature dimension of the 3D digital human; Based on the target driving sequence data, the 3D digital human is rendered and driven.

2. The voice-based 3D digital population driving method according to claim 1, characterized in that: Before inputting the target speech data into the target analysis model to obtain the target driving sequence data, the method further includes: Obtain sampled voice data; Extracting speech features from the sampled speech data to obtain sampled speech features and corresponding sampled driving sequence data; The target analysis model is trained based on the corresponding frame data of the sampled drive sequence data and the sampled voice data to obtain the target analysis model.

3. The voice-based 3D digital population driving method according to claim 2, characterized in that: The extracting speech features from the sampled speech data to obtain the sampled speech features and the corresponding sampled driving sequence data includes: converting the sampled speech data into original drive sequence data, and extracting the sampled drive sequence data from the original drive sequence; or The sampled speech data is sequentially input into a feature extraction layer, a linear interpolation layer, a Transformer layer and an MLP layer to obtain the sampled driving sequence data.

4. The voice-based 3D digital population driving method according to claim 2 or 3, characterized in that: The training of the target analysis model based on the corresponding frame data of the sampled driving sequence data and the sampled speech data comprises: Acquire the corresponding frame data of the sampled voice data; extracting standard drive sequence data corresponding to the number of dimensions of the sampled drive sequence data from the corresponding frame data; Based on the sampled driving sequence data and the standard driving sequence data, a correction sequence is performed on the target analysis model.

5. The voice-based 3D digital population driving method according to claim 4, characterized in that: The obtaining of the corresponding frame data of the sampled voice data comprises: Performing facial image rendering based on the original driving sequence data to obtain corresponding frame data of the sampled voice data; or The video data corresponding to the sampled voice data is obtained, and the video data is frame-aligned with the sampled voice data to obtain the frame data corresponding to the sampled voice data.

6. The voice-based 3D digital population driving method according to claim 3, characterized in that: The target drive sequence data are mouth feature points, and the number of dimensions of the target drive sequence data is 37; the original drive sequence data are facial feature points, and the number of dimensions of the original drive sequence data is 52.

7. A 3D digital population-type driving device based on speech, characterized in that: include: An acquisition module, used to acquire target voice data; An analysis module, used for inputting the target speech data into a target analysis model to obtain target drive sequence data; The target analysis model is used to determine the target drive sequence data based on the target speech data, and the feature dimension corresponding to the target drive sequence data corresponds to the mouth feature dimension of the 3D digital human; A rendering module is used to drive the 3D digital human based on the target driving sequence data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the voice-based 3D digital population driving method as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the voice-based 3D digital population driving method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the voice-based 3D digital population driving method described in any one of claims 1 to 6 is implemented.

Images