Virtual human action generation method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of virtual reality, and discloses a virtual human action generation method and device, equipment, a storage medium and a product, and the method comprises the steps: carrying out the emotion recognition of a target face image through an emotion recognition model, obtaining basic emotion data, and enabling the target face image to be a human face image collected in real time or a human face image designated by a user; determining character action data according to the basic emotion data, wherein the character action data comprises action data of a plurality of character parts; and performing animation synthesis based on the character action data to generate virtual human action data. Because the facial expression is a direct carrier of emotion expression, the emotion recognition is performed on the facial image, the facial expression features are captured and analyzed, and the virtual human action is constructed based on the recognized emotion, so that the emotion state of the human can be more accurately understood and simulated, and the more natural and emotional virtual human action is generated.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to methods, devices, equipment, storage media, and products for generating virtual human motion. Background Technology

[0002] In the field of Virtual Reality (VR), the generation of virtual human movements is a complex process involving multiple dimensions and technologies, encompassing various innovative methods and technical approaches. Currently, the generation methods mainly fall into two categories: text-driven generation and voice-driven generation.

[0003] Text-driven generation focuses on using natural language processing (NLP) technology to parse and understand text content, thereby guiding the actions and expressions of virtual characters. However, text-driven methods struggle to capture subtle changes in human emotions, making the movements of virtual characters appear mechanical and stiff.

[0004] Speech-driven generation focuses on using speech recognition and synthesis technologies to capture the speaker's emotions and intentions in real time by utilizing information such as rhythm, intonation, and speech rate in the sound signal. This allows the system to analyze the speech content and translate the user's voice commands into virtual character actions. Although speech contains rich emotional and rhythmic information, existing advanced technologies such as diffusion models often struggle to effectively capture and express the complex relationships between tone, rhythm, and movement during the process of converting this information into an implicit spatial representation through vector quantization. This results in even the most advanced speech-to-virtual-human motion synthesis algorithms failing to achieve satisfactory naturalness and fluency. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, device, storage medium and product for generating virtual human actions, in order to solve the technical problem that the existing technology of generating virtual human actions through text or voice is not effective.

[0006] To achieve the above objectives, this application proposes a method for generating virtual human actions, the method comprising:

[0007] Emotion recognition is performed on the target facial image using an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user.

[0008] Based on the aforementioned basic emotional data, character action data is determined, which includes action data for multiple body parts of the character.

[0009] Animation synthesis is performed based on the character's motion data to generate virtual human motion data.

[0010] Optionally, the step of synthesizing animation based on the character's motion data to generate virtual human motion data includes:

[0011] Perform 3D character modeling to construct virtual character models;

[0012] A skeletal system is set up for the virtual character model, and weights are drawn to obtain a movable virtual human model;

[0013] Virtual human motion data is constructed based on the character's motion data and the movable virtual human model.

[0014] Optionally, the step of constructing virtual human motion data based on the character motion data and the movable virtual human model includes:

[0015] Based on the character's motion data and the movable virtual human model, a virtual human motion animation is constructed.

[0016] The virtual human motion animation is optimized to generate virtual human motion data. The optimization includes at least one of motion refinement, motion fusion, and motion detail adjustment.

[0017] Optionally, before obtaining basic emotion data by performing emotion recognition on the target facial image using an emotion recognition model, the method further includes:

[0018] Obtain a pre-acquired image dataset, which includes multiple data pairs, each including pixel data of a sample facial image and multiple emotion category labels corresponding to the sample facial image;

[0019] A model sample set is constructed based on the pre-acquired image dataset;

[0020] The initial emotion model is trained using the model sample set to obtain an emotion recognition model.

[0021] Optionally, constructing the model sample set based on the pre-acquired image dataset includes:

[0022] The pre-acquired image dataset is traversed, and the traversed data pairs are used as the current data pairs;

[0023] The data pairs are separated to obtain pixel data and emotion category labels;

[0024] The pixel data is restored to obtain a sample facial image;

[0025] The sample facial images are output as images in a preset format and stored.

[0026] Model samples are constructed based on the emotion category labels and stored sample facial images;

[0027] At the end of the traversal, the constructed model samples are combined into a model sample set.

[0028] Optionally, after generating virtual human motion data by synthesizing animation based on the character motion data, the process further includes:

[0029] The virtual human motion data is rendered in real time to generate virtual human motion animation.

[0030] The actions are displayed through the virtual human's motion animation.

[0031] Furthermore, to achieve the above objectives, this application also provides a virtual human motion generation device, the device comprising:

[0032] The recognition module is used to perform emotion recognition on the target facial image through an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user.

[0033] The determination module is used to determine character action data based on the basic emotion data, wherein the character action data includes action data of multiple body parts of the character.

[0034] The compositing module is used to perform animation compositing based on the character's motion data to generate virtual human motion data.

[0035] In addition, to achieve the above objectives, this application also provides a virtual human motion generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the virtual human motion generation method as described above.

[0036] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the virtual human motion generation method described above.

[0037] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the virtual human motion generation method described above.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] Since facial expressions are the direct carrier of emotion, by performing emotion recognition on facial images, capturing and analyzing facial expression features, and constructing virtual human actions based on the recognized emotions, we can ensure a more accurate understanding and simulation of human emotional states, thereby generating more natural and emotional virtual human actions. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating an embodiment of the virtual human motion generation method of this application.

[0043] Figure 2 This is a flowchart illustrating Embodiment 2 of the virtual human motion generation method of this application;

[0044] Figure 3 This is a schematic diagram of pre-acquired image data in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of an emotion recognition model according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the module structure of the virtual human motion generation device according to an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the virtual human motion generation method in this application embodiment.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0051] Based on this, embodiments of this application provide a method for generating virtual human actions, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the virtual human motion generation method of this application.

[0052] In this embodiment, the virtual human motion generation method includes steps S10 to S30:

[0053] Step S10: Perform emotion recognition on the target facial image using an emotion recognition model to obtain basic emotion data.

[0054] It should be noted that the execution subject of this embodiment can be the virtual human motion generation device, which can be a personal computer, server or other electronic device, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the virtual human motion generation device is used as an example to describe the virtual human motion generation method of the present invention.

[0055] It should be noted that the target facial image can be a real-time captured image of a person's face or a user-specified image of a person's face. For example, during a virtual idol live stream, the streamer's facial image can be captured in real-time as the target facial image; during video recording or animation recording, user-specified or developer-specified facial images (or image sequences) can be used as the target facial images. The emotion recognition model can be a pre-trained model, such as a convolutional neural network (CNN) model, or a model that has been optimized (e.g., by adding or modifying network layers).

[0056] In practical applications, emotion recognition models can be used to identify emotions in target facial images, determine the emotion category of the target in the facial image, and thus obtain basic emotion data.

[0057] Step S20: Determine the character's action data based on the basic emotion data.

[0058] It should be noted that character motion data can include motion data for multiple parts of the character, such as body posture, gestures, and facial expressions.

[0059] In practical use, the administrators of the virtual human motion generation device can pre-set various motion data for each part of the character for different emotions, forming an emotion-motion mapping table. Then, determining the character's motion data based on the basic emotion data can be done by looking up the motion data of each part of the character corresponding to the basic emotion data in the emotion-motion mapping table, thereby obtaining the character's motion data.

[0060] In practical use, the six most common basic emotions are: happiness, sadness, anger, surprise, fear, and disgust. For these six basic emotions, corresponding action data can be pre-set for each part of the character.

[0061] To make it easier to understand, the following examples are provided:

[0062] Regarding body posture, the corresponding action data for each basic emotion can be as follows: Happiness: The body is relaxed, and when standing, one may lean slightly forward, indicating excitement and anticipation; Sadness: The body may be slightly bent, and the head is lowered, showing loss and frustration; Anger: The body may be upright, with hands clenched into fists or crossed in front of the chest, showing tension and dissatisfaction; Surprise: The body may suddenly straighten, the head tilts back, and the hands are open or placed near the mouth; Fear: The body may stiffen, with hands clenched or placed in front of the chest, showing tension and unease; Disgust: The body may lean slightly backward, with brows furrowed, showing aversion to something or someone.

[0063] Regarding gestures, the corresponding action data for each basic emotion are as follows: Happiness: possibly raising both hands or clapping to express celebration and joy; Sadness: possibly wiping away tears or covering the face to express pain and sorrow; Anger: possibly pointing at someone or something to express blame and dissatisfaction; Surprise: possibly covering the mouth or eyes to express shock and disbelief; Fear: possibly clenching hands or placing them on the chest to express tension and fear; Disgust: possibly pointing at someone or something to express aversion and dissatisfaction.

[0064] Regarding facial expressions, the corresponding action data for each basic emotion are as follows: Happiness: smiling, corners of the mouth turned up, eyes curved into crescent shapes; Sadness: crying, brows furrowed, corners of the mouth downturned; Anger: brows furrowed, eyes wide open, corners of the mouth closed or turned down; Surprise: eyes wide open, mouth slightly open, possibly showing a surprised expression; Fear: eyes wide open, brows furrowed, possibly accompanied by opening the mouth or biting the lip; Disgust: brows furrowed, eyes narrowed, corners of the mouth turned down, possibly accompanied by wrinkling of the nose.

[0065] Finally, the designed body postures, gestures, and facial expressions are integrated to form a complete emotion-action mapping table.

[0066] In practical use, after constructing the emotion-action mapping table, it can be tested in actual applications to determine whether these actions can accurately convey the corresponding emotions, and the emotion-action mapping table can be adjusted and optimized based on the test results.

[0067] Of course, in practical applications, a large number of action samples labeled with basic emotions can be used to train the action generation model. Then, the basic emotion data can be input into the trained action generation model to generate character action data.

[0068] Step S30: Perform animation synthesis based on the character's motion data to generate virtual human motion data.

[0069] In practical use, after determining the character's motion data, animation synthesis can be performed to generate virtual human motion data that can be rendered as animation.

[0070] In practical implementation, to ensure the rationality of the animation and avoid overly exaggerated or unrealistic movements, step S30 in this embodiment may include:

[0071] Perform 3D character modeling to construct virtual character models;

[0072] A skeletal system is set up for the virtual character model, and weights are drawn to obtain a movable virtual human model;

[0073] Virtual human motion data is constructed based on the character's motion data and the movable virtual human model.

[0074] It should be noted that, in order to avoid the constructed movements not conforming to the actual movement habits of the human body, 3D character modeling can be performed first during animation compositing to construct a 3D virtual character model. Then, to ensure that subsequent movements conform to the actual movement habits of the human body, a skeletal system can be set for the virtual character model, and weights can be drawn for its skeletal system. This ensures that when the virtual character model with the set skeletal system moves, its movements conform to the skeletal constraints of the human body (i.e., ensuring that each part of the model can move correctly with the skeleton), thereby generating a movable virtual human model. Finally, the movements of each part of the movable virtual human model are adjusted based on the character's motion data to generate virtual human motion data.

[0075] When performing 3D modeling, existing 3D modeling software such as Blender, Maya, or 3ds Max can be used. Of course, other software with the same or similar functions or custom software can also be used. This embodiment does not limit this.

[0076] Understandably, because the constructed 3D virtual character model has a skeletal system, when synthesizing animation based on the character's motion data, it can be ensured that the virtual character's movements in the synthesized animation conform to the skeletal limitations of the actual human body, thereby avoiding overly exaggerated movements and ensuring that the virtual character's movements are more human-like and closely resemble actual human movements.

[0077] In practical use, when performing 3D modeling, the user identifier of the user currently logged in to the virtual human motion generation device can be obtained. The corresponding 3D modeling data (such as height, body type, etc.) can be found based on the user identifier, and then 3D modeling can be performed based on the found data. If the user has not set specific 3D modeling data, the default 3D modeling data can be used for 3D modeling.

[0078] Of course, since the generation of virtual human actions is continuous, the 3D virtual character model can be reused. That is, 3D modeling is only performed when the virtual human action is generated for the first time, and the already constructed virtual character model can be reused directly thereafter. Furthermore, it can be differentiated by user, meaning that a virtual character model can be built for each different user.

[0079] In specific implementations, to ensure that the movements corresponding to the final constructed virtual human motion data are smooth and coherent, the step of constructing virtual human motion data based on the character motion data and the movable virtual human model described in this embodiment may further include:

[0080] Based on the character's motion data and the movable virtual human model, a virtual human motion animation is constructed.

[0081] The virtual human motion animation is optimized to generate virtual human motion data.

[0082] It should be noted that, in order to ensure that the movements corresponding to the final constructed virtual human motion data are smooth and coherent, after constructing the animation based on the character motion data and the movable virtual human model, it can be used as virtual human motion animation instead of virtual human motion data at first. Then, the virtual human motion animation can be optimized and used as virtual human motion data.

[0083] In practical applications, when optimizing the motion animation of virtual humans, optimization may include at least one of the following: motion refinement, motion fusion, and motion detail adjustment.

[0084] Among them, motion refinement refers to refining the animation, such as adding secondary and transitional actions to make the animation richer and more realistic; motion fusion refers to using motion fusion technology to mix different actions to create more natural motion transitions; detail adjustment refers to making detailed adjustments to the animation, such as adjusting the timing, speed, and rhythm of the actions to ensure the smoothness and realism of the actions.

[0085] In a specific implementation, to ensure the actual display effect, after step S30 in this embodiment, the following may also be included:

[0086] The virtual human motion data is rendered in real time to generate virtual human motion animation.

[0087] The actions are displayed through the virtual human's motion animation.

[0088] It should be noted that when rendering in real time based on virtual human motion data, after loading the corresponding background and scene items, the virtual human motion data can be rendered in real time based on the loaded background and scene items to generate a smooth virtual human motion animation. Afterwards, the virtual human motion animation can be displayed to ensure that it can be interacted with immediately.

[0089] Among them, displaying actions through virtual human motion animation can be done by playing the virtual human motion animation on the display screen of the virtual human motion generation device, or by sending the virtual human motion animation to other devices for playback, such as pushing the virtual human motion animation to a live streaming playback terminal (such as a smartphone) for playback.

[0090] In practical applications, when rendering in real time based on virtual human motion data, existing real-time animation engines can be used, such as Unity3D or Unreal Engine.

[0091] This embodiment provides a method for generating virtual human actions. Since facial expressions are the direct carrier of emotion expression, by performing emotion recognition on facial images, capturing and analyzing facial expression features, and constructing virtual human actions based on the recognized emotions, it is possible to more accurately understand and simulate human emotional states, thereby generating more natural and emotional virtual human actions.

[0092] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S10, the virtual human motion generation method further includes steps S01 to S03:

[0093] Step S01: Obtain the pre-acquired image dataset.

[0094] It should be noted that the pre-acquired image dataset can include multiple data pairs. Each data pair can include pixel data of the sample facial image and multiple emotion category labels corresponding to the sample facial image. Based on different emotions, the emotion category labels can be divided into eight types: Neutral, Happiness, Surprise, Sadness, Anger, Disgust, Fear, and Contempt. The multiple emotion category labels in each data pair can be labeled by multiple different observers to ensure the generation of an emotion probability distribution, rather than just a single emotion label. During this process, opinions from various observers can also be collected to improve the accuracy of labeling and the depth of emotion recognition.

[0095] In practical use, the administrators of the virtual human motion generation device can pre-collect multiple sample facial images, convert them into pixel data for storage, and assign multiple emotion category labels to them, thereby constructing a pre-collected image dataset.

[0096] Of course, datasets provided by authoritative institutions can also be used as pre-collected image datasets.

[0097] For example, the FER+ dataset, an extended version of the FER2013 (Facial Expression Recognition 2013) dataset released by authoritative institutions, is used as a pre-collected image dataset. The FER+ dataset is an extended version of the original FER dataset. It re-annotates the images and sets multiple emotion category labels for each image to include more detailed emotion categories.

[0098] To facilitate understanding, we will now combine... Figure 3 This explanation is provided, but it does not limit the scope of this solution. Figure 3 This is a schematic diagram of the pre-acquired image dataset in this embodiment.

[0099] like Figure 3 As shown, a total of 12 images are included (the pre-collected image dataset stores the pixel data corresponding to the images; for ease of understanding and explanation, it is converted into images for illustration), and each image is assigned two emotion category labels (e.g., Figure 3 The categories such as Surprise / Happiness, Fear / Anger, and Fear / Sadness are for illustrative purposes only; in actual implementation, more emotion category labels may be included.

[0100] Step S02: Construct a model sample set based on the pre-acquired image dataset.

[0101] It should be noted that the pixel data of the sample facial images and the multiple emotion category labels corresponding to the sample facial images in the pre-acquired image dataset are integrated together. In order to facilitate subsequent model training and model validation, it is necessary to avoid data integration and separate the data. At this time, the data in the pre-acquired image dataset can be processed to separate the data, thereby constructing the model sample set.

[0102] In a specific implementation, in order to reasonably construct the model sample set, step S02 of this embodiment may include:

[0103] The pre-acquired image dataset is traversed, and the traversed data pairs are used as the current data pairs;

[0104] The data pairs are separated to obtain pixel data and emotion category labels;

[0105] The pixel data is restored to obtain a sample facial image;

[0106] The sample facial images are output as images in a preset format and stored.

[0107] Model samples are constructed based on the emotion category labels and stored sample facial images;

[0108] At the end of the traversal, the constructed model samples are combined into a model sample set.

[0109] It should be noted that the preset format can be set in advance by the administrator of the virtual human motion generation device, for example, setting the preset format to JPG format.

[0110] In practical use, the pre-acquired image dataset can be traversed, and the traversed data pairs can be used as the current data pairs. Then, the current data pairs can be separated (e.g., by character cutting, data truncation, array splitting, etc.) to split them into pixel data and emotion category labels. After that, the sample facial images can be converted into a preset format for output and stored. Then, model samples can be constructed based on the emotion category labels and the stored sample facial images.

[0111] In practical applications, building model samples based on emotion category labels and stored sample facial images can be achieved by mapping emotion category labels to the filenames of the stored sample facial images to generate mapping records. A model sample can then be formed based on these mapping records and the stored sample facial images. Finally, at the end of the traversal, the model samples can be combined into a model sample set. For example, multiple mapping records can be stored in a relational mapping table. After that, the relational mapping table and all the stored sample facial images can be combined to construct the model sample set.

[0112] To make it easier to understand, the following examples are provided:

[0113] The given pre-acquired image dataset is presented as a .csv file. The dataset can be visualized first, converting it into an image file for further image processing. In a Python environment, the OpenCV library can be used to restore the pixel data in the CSV file into an image and save it. After detailed data separation, all face pixel data is stored in a file named pixels.csv. Each row in this file represents an independent face image. By reading this data row by row, OpenCV can be used to restore the pixel data in each row into a 48×48 pixel image and output it as a JPG file.

[0114] Subsequently, during the saving process, a systematic naming rule can be adopted: for example, the face image corresponding to the first row of data is named 0.jpg, the second row is named 1.jpg, and so on, to ensure that the name of each image is consistent with its label [0], label [1], etc., which facilitates subsequent data matching and management. Afterwards, since the customized dataset class inherits the torch.utils.data.Dataset class in PYtorch, the efficiency and accuracy of data loading are guaranteed. And since it is necessary to synchronously obtain sample images and their corresponding emotion labels during the data loading process, based on this, a fine relational mapping table can be designed to record the file name (i.e. the file name when storing) of each stored image and its corresponding emotion expression label. Finally, the relational mapping table and all stored images are combined to form the model sample set.

[0115] This mapping table not only optimizes the data retrieval process but also greatly facilitates dataset management. In this way, it ensures that images and their sentiment tags are accurately matched at every stage of data processing. The main code for its implementation is shown below:

[0116] #Read the emotion file

[0117] file_emotion=pd.read_csv('emotion.csv', header=None)

[0118] #Read all files

[0119] files_document=os.listdir(path)

[0120] #Image storage name

[0121] path_list = []

[0122] #Stores emotion-related information associated with the image

[0123] emotion_list=[

[0124] # Traverse all files in this folder

[0125] for file_document in files_document:

[0126] #If a file is an image, extract its filename and corresponding emotion, and add them to the path_list and emotion_list lists respectively.

[0127] if os.path.splitext(file_document)[1]==".jpg":

[0128] path_list.append(file_document)

[0129] index=int(os.path.splitext(file_document)[0])

[0130] emotion_list.append(file_emotion.iat[index,0])

[0131] # Write the two lists into the image_emotion.csv file

[0132] df = pd.DataFrame()

[0133] df['path']=pd.Series(path_list)

[0134] df['emotion']=pd.Series(emotion_list)

[0135] df.to_csv(path+'\\ImageEmotion.csv',index=False,header=False)

[0136] It should be noted that the Python code above is for illustrative purposes only, and no specific limitations are imposed on the solution for ease of understanding.

[0137] Step S03: Train the initial emotion model using the model sample set to obtain an emotion recognition model.

[0138] In practical applications, the initial emotion model can be trained using a model sample set, and the initial emotion model that has been trained to convergence can be used as the emotion recognition model.

[0139] In specific training, the model sample set can be split into a model training set and a model validation set (e.g., if there are 30,000 model samples in the model sample set, the first 25,000 model samples can be selected as the model training set, and the remaining 5,000 model samples can be used as the model validation set). The initial emotion model is trained using the model training set, and the initial emotion model trained to convergence is validated using the model validation set (e.g., accuracy validation, robustness, etc. During validation, the metrics that can be used are accuracy, precision, recall, F1 score, etc., or at least one of similar metrics). When the validation is successful, it is used as the emotion recognition model.

[0140] To facilitate understanding, we will now combine... Figure 4 This explanation is provided, but it does not limit the scope of this solution. Figure 4 This is a schematic diagram of the emotion recognition model structure in this embodiment. The initial emotion model structure can be as follows: Figure 4 As shown in Model A or Model B, the input images have dimensions of 48×48×1. They are first convolved with a 3×3×64 kernel, followed by a 2×2 pooling operation. These steps produce a 24×24×64 feature set. Figure 1 Before each convolution, padding with a size of 1 is used to maintain the spatial dimension of the feature map, and the stride is also set to 1. This process effectively extracts deeper features and further reduces the spatial dimension of the feature map.

[0141] Next, a flattening operation is performed, transforming the data into a one-dimensional tensor of length 9216. This step is to adapt the input requirements of the fully connected layer. The flattened data is first processed by dropout to reduce the risk of overfitting, and then flows into a hidden layer containing 4096 neurons. The data is then processed by dropout again, continuing to flow into a hidden layer with 1024 neurons, then through a hidden layer with 256 neurons, and finally reaching the output layer with 7 neurons.

[0142] The main execution code for this model can be shown below:

[0143] self.conv = nn.Sequential(

[0144] # Input channels in_channels, output channels (i.e., the number of channels in the convolution kernel) out_channels, kernel size kernel_size, stride, padding (number of rows and columns for symmetric zero padding)

[0145] #input:(bitch_size,1,48,48),output:(bitch_size,64,48,48),(48-3+2*1) / 1+1=48

[0146] nn.Conv2d(in_channels=1, out_channels=64, kernel_size=3, stride=1, padding=1), # Convolutional layer

[0147] nn.BatchNorm2d(num_features=64), # Normalize

[0148] nn.RReLU(inplace=True), # Activation function

[0149] #output(bitch_size,64,24,24)

[0150] nn.MaxPool2d(kernel_size=2, stride=2), #Max pooling )

[0152] self.fc = nn.Sequential(

[0153] nn.Dropout(p=0.2),

[0154] nn.Linear(in_features=256*6*6,out_features=4096),

[0155] nn.RReLU(inplace=True),

[0156] nn.Dropout(p=0.5),

[0157] nn.Linear(in_features=4096, out_features=1024),

[0158] nn.RReLU(inplace=True),

[0159] nn.Linear(in_features=1024, out_features=256),

[0160] nn.RReLU(inplace=True),

[0161] nn.Linear(in_features=256,out_features=7) )

[0163] It should be noted that the Python code above is for illustrative purposes only, and no specific limitations are imposed on the solution for ease of understanding.

[0164] This embodiment provides a method for generating virtual human actions. Because an emotion recognition model is pre-trained, it ensures that when virtual human actions need to be generated later, the pre-trained model can be directly used for emotion recognition, thus ensuring the feasibility of the virtual human action generation method of this application.

[0165] This application also provides a virtual human motion generation device; please refer to [reference needed]. Figure 5 The virtual human motion generation device includes:

[0166] The recognition module 10 is used to perform emotion recognition on the target facial image through an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user.

[0167] The determining module 20 is used to determine character action data based on the basic emotion data, wherein the character action data includes action data of multiple parts of the character.

[0168] The compositing module 30 is used to perform animation compositing based on the character's motion data to generate virtual human motion data.

[0169] The virtual human motion generation device provided in this application, employing the virtual human motion generation method in the above embodiments, can solve the technical problem of poor virtual human motion generation effect through text or voice in the prior art. Compared with the prior art, the beneficial effects of the virtual human motion generation device provided in this application are the same as those of the virtual human motion generation method provided in the above embodiments, and other technical features in the virtual human motion generation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0170] This application provides a virtual human motion generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the virtual human motion generation method in the above embodiment 1.

[0171] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a virtual human motion generation device suitable for implementing embodiments of this application. The virtual human motion generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The virtual human motion generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0172] like Figure 6As shown, the virtual human motion generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the virtual human motion generation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the virtual human motion generation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show virtual human motion generation devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0173] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0174] The virtual human motion generation device provided in this application, employing the virtual human motion generation method in the above embodiments, can solve the technical problem of poor virtual human motion generation effect through text or voice in the prior art. Compared with the prior art, the beneficial effects of the virtual human motion generation device provided in this application are the same as those of the virtual human motion generation method provided in the above embodiments, and other technical features in this virtual human motion generation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0175] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0177] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the virtual human action generation method in the above embodiments.

[0178] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0179] The aforementioned computer-readable storage medium may be included in the virtual human motion generation device; or it may exist independently and not be assembled into the virtual human motion generation device.

[0180] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the virtual human motion generation device, cause the virtual human motion generation device to:

[0181] Emotion recognition is performed on a target facial image using an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user. Based on the basic emotion data, human action data is determined, which includes action data of multiple human body parts. Based on the human action data, animation is synthesized to generate virtual human action data.

[0182] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0184] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0185] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described virtual human motion generation method, which can solve the technical problem of poor virtual human motion generation effect through text or voice in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the virtual human motion generation method provided in the above embodiments, and will not be repeated here.

[0186] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the virtual human motion generation method described above.

[0187] The computer program product provided in this application can solve the technical problem that the existing technology produces poor virtual human motion generation effects through text or voice. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the virtual human motion generation method provided in the above embodiments, and will not be repeated here.

[0188] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for generating virtual human actions, characterized in that, The method includes: Emotion recognition is performed on the target facial image using an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user. Based on the aforementioned basic emotional data, character action data is determined, which includes action data for multiple body parts of the character. Animation synthesis is performed based on the character's motion data to generate virtual human motion data.

2. The virtual human motion generation method as described in claim 1, characterized in that, The process of synthesizing animation based on the character's motion data to generate virtual human motion data includes: Perform 3D character modeling to construct virtual character models; A skeletal system is set up for the virtual character model, and weights are drawn to obtain a movable virtual human model; Virtual human motion data is constructed based on the character's motion data and the movable virtual human model.

3. The virtual human motion generation method as described in claim 2, characterized in that, The construction of virtual human motion data based on the character motion data and the movable virtual human model includes: Based on the character's motion data and the movable virtual human model, a virtual human motion animation is constructed. The virtual human motion animation is optimized to generate virtual human motion data. The optimization includes at least one of motion refinement, motion fusion, and motion detail adjustment.

4. The virtual human motion generation method as described in claim 1, characterized in that, Before obtaining basic emotion data by performing emotion recognition on the target facial image using an emotion recognition model, the process also includes: Obtain a pre-acquired image dataset, which includes multiple data pairs, each including pixel data of a sample facial image and multiple emotion category labels corresponding to the sample facial image; A model sample set is constructed based on the pre-acquired image dataset; The initial emotion model is trained using the model sample set to obtain an emotion recognition model.

5. The virtual human motion generation method as described in claim 4, characterized in that, The step of constructing a model sample set based on the pre-acquired image dataset includes: The pre-acquired image dataset is traversed, and the traversed data pairs are used as the current data pairs; The data pairs are separated to obtain pixel data and emotion category labels; The pixel data is restored to obtain a sample facial image; The sample facial images are output as images in a preset format and stored. Model samples are constructed based on the emotion category labels and stored sample facial images; At the end of the traversal, the constructed model samples are combined into a model sample set.

6. The virtual human motion generation method according to any one of claims 1-5, characterized in that, After generating virtual human motion data by synthesizing animation based on the character motion data, the process further includes: The virtual human motion data is rendered in real time to generate virtual human motion animation. The actions are displayed through the virtual human's motion animation.

7. A virtual human motion generation device, characterized in that, The device includes: The recognition module is used to perform emotion recognition on the target facial image through an emotion recognition model to obtain basic emotion data. The target facial image is a real-time captured human facial image or a human facial image specified by the user. The determination module is used to determine character action data based on the basic emotion data, wherein the character action data includes action data of multiple body parts of the character. The compositing module is used to perform animation compositing based on the character's motion data to generate virtual human motion data.

8. A virtual human motion generation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the virtual human motion generation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the virtual human motion generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the virtual human motion generation method as described in any one of claims 1 to 6.

Images