Voice driving method for facial expression of digital curling player

Abstract

The invention provides a voice driving method for facial expressions of digital curling players. According to the method, digital human images of curling players are established based on curling match videos, a personalized emotion parameter library of the curling players is established, curling match audios are converted into personalized expression parameters through a neural network model, the personalized expression parameters are converted into three-dimensional face animations, and personalized digital human voice driving is achieved. In the modeling process of the digital curling player, personalized features such as facial features and expression features of the player are considered, meanwhile, only competition video modeling is used in digital human development, online driving based on voice is achieved, a developer can be helped to rapidly achieve personalized customization in the digital human development, and the requirements of different users or scenes are met.

Description

Technical Field

[0001] The present invention relates to the fields of metaverse, digital human and artificial intelligence technology, and in particular to a voice-driven method for facial expressions of a digital curling player. Background Art

[0002] With the development of 3D modeling technology, personalized digital human technology has become a hot research direction in the field of virtual sports competitions. By creating a 3D digital human image that is highly similar to real athletes, as well as voice and action drive, the digital human can simulate the behavior and expression of real athletes, forming a supplementary perspective to bring the audience a multi-angle immersive virtual viewing experience.

[0003] In the field of curling, personalized digital human modeling based on curling players' game videos has become an innovative and efficient method. Through deep learning algorithms, a speech-expression mapping system is developed to accurately identify the athlete's voice and convert it into facial expression control parameters, combined with the expression features in the curling game video to achieve real simulation. This method not only improves the efficiency of digital human creation, but also provides new opportunities for curling players' virtual training and game simulation, bringing a richer and more realistic experience. However, existing methods cannot achieve personalized customization, and the digital human modeling driving technology is cumbersome and cannot meet the needs of different users or scenarios. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art and propose a voice-driven method for facial expressions of digital curling players. The method establishes a digital human image of a curling player based on a curling game video, establishes a personalized emotional parameter library of the curling player, uses a neural network model to convert the curling game audio into personalized expression parameters, and converts the personalized expression parameters into three-dimensional facial animation to achieve personalized digital human voice driving.

[0005] The present invention is implemented by the following technical scheme. The present invention proposes a voice-driven method for facial expressions of digital curling players, and the method comprises the following steps:

[0006] Step 1: Create a digital human image of a curling player based on a curling game video; obtain a 3D facial model of the curling player with texture mapping through a multi-view 3D reconstruction method, and create a digital human template with complete skeleton and facial control parameters, align it with the facial 3D model, calibrate the skeleton points and facial control parameters, and obtain a digital facial image of the curling player with his own characteristics;

[0007] Step 2: Establish a personalized emotional parameter library of curling players through curling game videos; learn emotional vectors from the curling players' shouts based on the neural network model, learn and output the emotional vectors in the neural network model, and extract, train and infer the personalized expression features and emotional features of the curling players when expressing and shouting separately;

[0008] Step 3: Use a neural network model to learn expression parameters from the curling game audio, combine the emotion coefficient obtained in step 2 with the data in the personalized emotion parameter library, and superimpose it with the expression parameters to convert the curling game audio into personalized expression parameters to realize the online voice-driven method of facial expressions, and convert the personalized expression parameters into three-dimensional facial animation to realize personalized digital human voice drive.

[0009] Furthermore, the step 1 is specifically as follows:

[0010] Step 1.1, collecting a series of multi-view facial images of curling players with overlapping perspectives from the curling competition video, and using 3D modeling software to generate a 3D facial model of the curling player with texture mapping based on a multi-view stereo 3D reconstruction method;

[0011] Step 1.2, establish a digital human template with complete skeleton and facial control parameters, including male and female templates of various face shapes; select the face shape most similar to the athlete in the curling competition video from the digital human template, and align it with the obtained curling athlete's facial 3D model, that is, fit its vertices with the reconstructed facial 3D model, but do not change the original topological structure and number of vertices;

[0012] Step 1.3, calibrate the skeleton points and facial control parameters of the digital human template after mesh vertex alignment, convert the texture map of the facial 3D model into a texture material, replace the facial skin of the digital human template, modify the hairstyle, and add glasses and accessories to obtain a digital curling athlete's facial image with his own characteristics.

[0013] Furthermore, in step 2, based on the curling game video, emotional pictures of "happy", "sad", "afraid", "terrified", "surprised", "angry" and "disgusted" in six situations where curling players shouted "wipe the ice quickly", "stop wiping the ice", "a few seconds", "off the line", "top line" and "whisper" were collected, and each emotion was divided into two categories: "soothing" and "intense".

[0014] Furthermore, in step 2, a personalized emotion parameter library is established by detecting 68 key point sets {P1, P2, P3...P68} for each frame of the curling game video, reconstructing the facial expressions of the curling players using the eos library, fitting the key points to obtain the Blendshape parameter values ​​of the facial expressions, taking the average of the Blendshape parameter values ​​of each emotion, which is the parameter of the curling player under this emotion, saving all the emotion parameters to obtain a personalized emotion parameter library.

[0015] Furthermore, in step 2, a neural network model is constructed to learn and output an emotion vector as follows: a personalized emotion parameter library is used to design a neural network model, where the input layer receives the audio of curling athletes when they are shouting, giving commands, or otherwise communicating; in the hidden layer, a convolutional neural network is used to extract deep features, and an attention mechanism is introduced to focus on key areas; an emotion vector learning module is added to learn and output an emotion vector from the features, representing the weights of the emotions contained in the curling game audio at this time in terms of happiness, sadness, fear, terror, surprise, anger, and disgust, to obtain a 1*7 vector; the error between the predicted and true emotion vectors is minimized by training the model, and personalized emotion features are extracted from the curling athletes' shouts and expressions in real time.

[0016] Furthermore, in step 3, a neural network model is used to extract audio features from the curling game audio clip and perform feature representation; an expression network is used to analyze expression features; and a fully connected layer expands the expression parameter vector into a BlendShape parameter weight.

[0017] Furthermore, in step 3, the emotion vector obtained in step 2 is multiplied by the BlendShape parameter stored in the personalized emotion parameter library to obtain the emotion base; the emotion base is weightedly added to the expression parameters parsed by the neural network from the curling game audio to obtain the personalized expression parameters with emotion; the overall loss function value Loss is designed as:

[0018]

[0019] Among them, loss P Indicates the minimum mean square error between the predicted BlendShape parameters and the true value, loss M Indicates the coherence stability value between the previous and next frames; n is the number of channels of the BlendShape parameter, y (i) (x) is the network output, is the expected output, Δy (i) (x) is the difference in network output for the corresponding frame, is the difference in expected output for the corresponding frames.

[0020] Furthermore, in step 3, the BlendShape parameters are sent to the 3D modeling software, and the BlendShape parameters are received in real time through LiveLink, and the facial expressions of the digital curling player are controlled to render the animation in real time; after receiving the data, the parameter data is converted into an animation of the facial expressions of the digital curling player using a blueprint, thereby driving the digital curling player to make corresponding facial movements in coordination with the voice, thereby realizing online voice drive of the facial expressions of the digital curling player.

[0021] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the voice-driven method for facial expressions of a digital curling player when executing the computer program.

[0022] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the voice-driven method for the facial expressions of a digital curling player.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention provides a voice-driven method for facial expressions of digital curling athletes, which can enhance the user's immersion and participation through realistic simulation and interaction of digital curling athletes, allowing users to experience a more real and natural interactive experience. The modeling process of digital curling athletes takes into account the personalized characteristics of athletes, such as body shape, movement habits, expression characteristics, etc., which helps developers to better realize personalized customization in the development of digital humans and meet the needs of different users or scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0026] Figure 1 This is a flow chart of a voice-driven method for facial expressions of a digital curling player described in the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Combination Figure 1 The present invention proposes a voice-driven method for facial expressions of digital curling players, the method comprising the following steps:

[0029] Step 1: Create a digital human image of a curling player based on a curling game video; obtain a 3D facial model of the curling player with texture mapping through a multi-view 3D reconstruction method, and create a digital human template with complete skeleton and facial control parameters, align it with the facial 3D model, calibrate the skeleton points and facial control parameters, and obtain a digital facial image of the curling player with his own characteristics;

[0030] The step 1 is specifically as follows:

[0031] Step 1.1, collecting a series of multi-view facial images of curling players with overlapping perspectives from the curling competition video, and using 3D modeling software to generate a 3D facial model of the curling player with texture mapping based on a multi-view stereo 3D reconstruction method;

[0032] Step 1.2, establish a digital human template with complete skeleton and facial control parameters, including male and female templates of various face shapes; select the face shape that is most similar to the athlete in the curling competition video from the digital human template, and align it with the obtained curling athlete's facial 3D model, that is, fit its vertices with the reconstructed facial 3D model, but do not change the original topological structure and number of vertices;

[0033] Step 1.3, calibrate the skeleton points and facial control parameters of the digital human template after mesh vertex alignment, convert the texture map of the facial 3D model into a texture material, replace the facial skin of the digital human template, modify the hairstyle, and add glasses and accessories to obtain a digital curling athlete's facial image with his own characteristics.

[0034] Step 2: Establish a personalized emotional parameter library of curling players through curling game videos; learn emotional vectors from the curling players' shouts based on the neural network model, learn and output the emotional vectors in the neural network model, and extract, train and infer the personalized expression features and emotional features of the curling players when expressing and shouting separately;

[0035] In step 2, based on the curling game video, emotional pictures of "happy", "sad", "afraid", "terrified", "surprised", "angry" and "disgusted" in six situations where curling players shouted "wipe the ice quickly", "stop wiping the ice", "a few seconds", "off the line", "top line" and "whisper" were collected, and each emotion was divided into two categories: "soothing" and "intense".

[0036] In step 2, a personalized emotion parameter library is established as follows: 68 key point sets {P1, P2, P3...P68} are detected for each frame of the curling game video, the facial expressions of the curling players are reconstructed using the eos library, the key points are fitted to obtain the Blendshape parameter values ​​of the facial expressions, the Blendshape parameter values ​​of each emotion are averaged, which is the parameter of the curling player under this emotion, all the emotion parameters are saved to obtain a personalized emotion parameter library.

[0037] In step 2, a neural network model is constructed to learn and output the emotion vector as follows: a personalized emotion parameter library is used to design a neural network model, where the input layer receives the audio of curling athletes when they are shouting, giving orders, and communicating in other ways; in the hidden layer, a convolutional neural network is used to extract deep features, and an attention mechanism is introduced to focus on key areas; an emotion vector learning module is added to learn and output the emotion vector from the features, representing the weights of the emotions contained in the curling game audio at this time in terms of happiness, sadness, fear, terror, surprise, anger, and disgust, and a 1*7 vector is obtained; the error between the predicted and true emotion vectors is minimized by training the model, and personalized emotion features are extracted from the curling athletes' shouts and expressions in real time.

[0038] Step 3: Use a neural network model to learn expression parameters from the curling game audio, combine the emotion vector obtained in step 2 with the data in the personalized emotion parameter library, and superimpose it with the expression parameters to convert the curling game audio into personalized expression parameters to implement an online voice-driven method for facial expressions, convert personalized expression parameters into three-dimensional facial animation, and realize personalized digital human voice drive.

[0039] In step 3, a neural network model is used to extract audio features from the curling game audio clip and perform feature representation; an expression network is used to analyze expression features; and a fully connected layer expands the expression parameter vector into a BlendShape parameter weight.

[0040] In step 3, the emotion coefficient obtained in step 2 is multiplied by the BlendShape parameter saved in the personalized emotion parameter library to obtain the emotion base; the emotion base is weightedly added with the expression parameters parsed by the neural network from the curling game audio to obtain the personalized expression parameters with emotion; the overall loss function value Loss is designed as:

[0041]

[0042] Among them, loss P Indicates the minimum mean square error between the predicted BlendShape parameters and the true value, loss M Indicates the coherence stability value between the previous and next frames; n is the number of channels of the BlendShape parameter, y (i) (x) is the network output, is the desired output, Δy (i) (x) is the difference in network output for the corresponding frame, is the difference in expected output for the corresponding frames.

[0043] In step 3, the BlendShape parameters are sent to the 3D modeling software, which receives the BlendShape parameters in real time through LiveLink, controls the facial expressions of the digital curling player, and renders the animation in real time; after receiving the data, the parameter data is converted into an animation of the digital curling player's facial expressions using Blueprint, thereby driving the digital curling player to make corresponding facial movements in coordination with the voice, thereby realizing online voice drive of the digital curling player's facial expressions.

[0044] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the voice-driven method for facial expressions of a digital curling player when executing the computer program.

[0045] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the voice-driven method for the facial expressions of a digital curling player.

[0046] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0047] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0048] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0049] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0050] The above is a detailed introduction to the voice-driven method for facial expressions of a digital curling player proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A voice-driven method for facial expressions of digital curling players, characterized in that: The method comprises the following steps: Step 1: Create a digital human image of the curling athlete based on a curling competition video. A textured 3D facial model of the curling athlete is obtained through multi-view 3D reconstruction. A digital human template with complete skeleton and facial control parameters is created. This template is aligned with the 3D facial model, and the skeleton points and facial control parameters are calibrated to obtain a digital facial image of the curling athlete with the athlete's unique characteristics. Step 2: Build a personalized emotional parameter library for curlers using curling competition videos. Use a neural network model to learn emotional vectors from the curlers' calls. This process is then used to learn and output emotional vectors within the neural network model. The personalized facial and emotional features of the curlers during their expressions and calls are then extracted, trained, and inferred separately. Step 3: Use a neural network model to learn expression parameters from the curling game audio, combine the emotion vector obtained in step 2 with the data in the personalized emotion parameter library, and superimpose it with the expression parameters to convert the curling game audio into personalized expression parameters, realizing an online voice-driven method for facial expressions, and converting personalized expression parameters into three-dimensional facial animation to realize personalized digital human voice-driven.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: Step 1.1: Collect a series of overlapping multi-view facial images of curling athletes from a curling competition video. Based on a multi-view stereoscopic 3D reconstruction method, use 3D modeling software to generate a 3D facial model of the curling athlete with texture mapping. Step 1.2: Create digital human templates with complete skeleton and facial control parameters, including male and female templates with various face shapes. Select the face shape most similar to the curling athlete in the curling competition video from the digital human templates and align it with the obtained 3D facial model of the curling athlete. This means fitting its vertices to the reconstructed 3D facial model without changing the original topology or number of vertices. Step 1.3: Calibrate the skeleton points and facial control parameters of the digital human template after mesh vertex alignment. Convert the texture map of the 3D facial model into a texture material, replace the facial skin of the digital human template, modify the hairstyle, and add glasses and accessories to obtain a digital curling athlete's facial image with his or her own characteristics.

3. The method according to claim 2, characterized in that In step 2, based on the curling game video, emotional images of "happy", "sad", "fear", "horror", "surprise", "anger" and "disgust" in six situations of curling players shouting "quickly wipe the ice", "stop wiping the ice", "a few seconds", "off the line", "top line" and "whisper" are collected, and each emotion is divided into two categories: "soothing" and "intense".

4. The method according to claim 3, characterized in that In step 2, a personalized emotion parameter library is established as follows: 68 key point sets {P1, P2, P3...P68} are detected for each frame of the curling game video, the facial expressions of the curling players are reconstructed using the eos library, the key points are fitted to obtain the Blendshape parameter values ​​of the facial expressions, the Blendshape parameter values ​​of each emotion are averaged, which is the parameter of the curling player under this emotion, and all the emotion parameters are saved to obtain a personalized emotion parameter library.

5. The method according to claim 4, characterized in that In step 2, a neural network model is constructed to learn and output emotion vectors. Specifically, the following steps are performed: a personalized emotion parameter library is used to design a neural network model, where the input layer receives audio of curling athletes shouting, giving commands, and performing other communications; in the hidden layer, a convolutional neural network is used to extract deep features, and an attention mechanism is introduced to focus on key areas; an emotion vector learning module is added to learn and output emotion vectors from the features, representing the weights of the emotions contained in the curling game audio at this time in terms of happiness, sadness, fear, terror, surprise, anger, and disgust, to obtain a 1*7 vector; the model is trained to minimize the error between the predicted and true emotion vectors, and personalized emotion features are extracted from the curling athletes' shouts and expressions in real time.

6. The method according to claim 5, characterized in that In step 3, a neural network model is used to extract audio features from the curling game audio clip and perform feature representation; an expression network is used to analyze expression features; and a fully connected layer expands the expression parameter vector into BlendShape parameter weights.

7. The method according to claim 6, characterized in that In step 3, the emotion vector obtained in step 2 is multiplied by the BlendShape parameter stored in the personalized emotion parameter library to obtain the emotion base. The emotion base is weightedly added to the expression parameters parsed by the neural network from the curling game audio to obtain the personalized expression parameters with emotion. The overall loss function value of the design is: Among them, loss P Indicates the minimum mean square error between the predicted BlendShape parameters and the true value, loss M Indicates the coherence stability value between the previous and next frames; n is the number of channels of the BlendShape parameter, y (i) (x) is the network output, is the desired output, Δy (i) (x) is the difference in network output for the corresponding frame, is the difference in expected output for the corresponding frame.

8. The method according to claim 7, characterized in that In step 3, the BlendShape parameters are sent to the 3D modeling software, which receives the BlendShape parameters in real time via LiveLink, controls the facial expressions of the digital curling player, and renders the animation in real time. After receiving the data, the parameter data is converted into an animation of the digital curling player's facial expressions using Blueprint, thereby driving the digital curling player to make corresponding facial movements in coordination with the voice, thus realizing online voice-driven facial expressions of the digital curling player.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images