Digital human motion driving method and device

Abstract

The invention discloses a digital human action driving method and device, and relates to the technical field of computer vision. The specific implementation mode comprises the following steps: receiving voice information sent by a user; the voice information is sent to a first model for processing, emotion parameters are obtained, and the first model is subjected to common sense training and emotion training; sending the emotion parameter to a second model for processing to obtain an action parameter; and performing digital human motion driving based on the motion parameters. According to the method and the device, the voice information is processed by using the first model subjected to common sense training and emotion training, and the emotion is judged by using common sense in the processing process, so that the recognition of the model on the emotion in the voice information in a use scene without context semantic support can be remarkably improved. After emotion parameters are obtained, the emotion parameters are sent to the second model to be processed, action parameters are obtained, and finally digital human action driving is carried out based on the action parameters.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of computer vision technology, and in particular to a method, device, electronic device and medium for driving digital human motion. Background Art

[0002] The development of computer vision AI in image processing has had a profound impact on today's metaverse and gaming industry. Computer vision technology can achieve more realistic and delicate image rendering, making virtual scenes in the metaverse more realistic.

[0003] At the current stage, virtual digital humans are undoubtedly one of the most valued, most invested, and fastest-implemented areas in the metaverse. However, in terms of interaction methods, current virtual digital humans are still unable to communicate and interact naturally with real people, especially in terms of their lack of emotions and personality.

[0004] Nowadays, computers can use trained large language models to perform natural language processing and judge the keywords in the language from a semantic perspective, not a simple keyword matching. Most digital human research focuses on the realism of the face, facial features, and head, and the driving force is also mainly on the face, facial features, mouth, head, and the driving force of the digital human body parts. Summary of the invention

[0005] Aiming at the problem of digital human motion driving, a digital human motion driving method, device, electronic equipment and medium are provided.

[0006] According to a first aspect, a method for driving digital human actions is provided, comprising: receiving voice information sent by a user; sending the voice information to a first model for processing to obtain emotion parameters, wherein the first model has undergone common sense training and emotion training; sending the emotion parameters to a second model for processing to obtain action parameters; and driving digital human actions based on the action parameters.

[0007] In combination with the first aspect, in some examples, the training method of the first model includes: determining common sense tasks and emotion tasks based on a data set with emotion labels and a common sense knowledge base; and obtaining the first model by fine-tuning the common sense tasks and emotion tasks on a large language model.

[0008] In combination with the first aspect, in some examples, common sense tasks and emotion tasks are determined based on a dataset with emotion labels and a common sense knowledge base, including: determining a common sense task template and an emotion task template respectively; under the constraints of the dataset with emotion labels and the common sense knowledge base, a large language model generates common sense training data and emotion training data through the common sense task template and the emotion task template respectively; determining the common sense task through the common sense training data; and determining the emotion task through the emotion training data.

[0009] In combination with the first aspect, in some examples, the first model is obtained by fine-tuning the large language model through common sense tasks and emotional tasks, including: during a round of fine-tuning training, inputting common sense training data or emotional training data into the large language model for forward calculation; calculating the loss value of the forward calculation result through a loss function, and performing back propagation to update the gradient of the large language model; calculating the score of the forward calculation result through a preset evaluation method; and determining to perform the next round of data set training or obtain the first model based on the loss value and the score.

[0010] In combination with the first aspect, in some examples, the training method of the second model includes: identifying character actions in the data set with emotion labels to form a skeletal action sequence data set; and performing fine-tuning training on the skeletal action sequence data set based on the large language model to obtain the second model.

[0011] In combination with the first aspect, in some examples, driving the digital human action based on the action parameters includes: determining multiple digital human actions through the action parameters; determining default positions of the multiple digital human actions; and connecting the multiple digital human actions according to the default positions to drive the digital human action.

[0012] In combination with the first aspect, in some examples, digital human motion driving is performed based on the motion parameters, including: determining multiple digital human motions through the motion parameters; determining the skeleton information corresponding to each of the digital human motions, the skeleton information including the spatial position and distance relationship; and fusing two adjacent digital human motions using alignment and spherical linear interpolation to perform digital human motion driving.

[0013] According to the second aspect, a digital human driving device is provided, comprising: a first unit for receiving voice information sent by a user; a second unit for sending the voice information to a first model for processing to obtain emotion parameters, wherein the first model has undergone common sense training and emotion training; a third unit for sending the emotion parameters to a second model for processing to obtain action parameters; and a fourth unit for performing digital human action driving based on the action parameters.

[0014] According to the third aspect, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method as in any embodiment of the digital human motion driving method.

[0015] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, a method as in any embodiment of the digital human action driving method is implemented.

[0016] According to the solution of the present application, the first model trained with common sense and emotion is used to process voice information. During the processing, common sense is used to judge emotions, which can significantly improve the recognition of emotions in voice information by the model in usage scenarios without contextual semantic support. After obtaining the emotion parameters, the emotion parameters are sent to the second model for processing to obtain action parameters, and finally the digital human action is driven based on the action parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 is an exemplary system architecture diagram to which some embodiments of the present application may be applied;

[0019] Figure 2 is a flow chart of an embodiment of a digital human motion driving method according to the present application;

[0020] Figure 3 is a schematic diagram of an application scenario of the digital human motion driving method according to the present application;

[0021] Figure 4 is a flow chart of another embodiment of the digital human motion driving method according to the present application;

[0022] Figure 5 is a structural schematic diagram of an embodiment of a digital human motion driving device according to the present application;

[0023] Figure 6 4 is a block diagram of an electronic device used to implement the digital human motion driving method of the embodiment of the present application. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a digital human motion driving method or a digital human motion driving device of the present application can be applied.

[0027] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as video applications, live broadcast applications, instant messaging tools, email clients, social platform software, etc.

[0029] The terminal devices 101, 102, 103 here can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0030] The server 105 may be a server that provides various services, such as a background server that provides support for the terminal devices 101, 102, and 103. The background server may analyze and process the received voice information and other data, and feed back the processing results (such as action parameters) to the terminal device.

[0031] It should be noted that the digital human motion driving method provided in the embodiment of the present application can be executed by the server 105 or the terminal devices 101, 102, 103, and accordingly, the digital human motion driving device can be set in the server 105 or the terminal devices 101, 102, 103.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Continue to refer Figure 2 , shows a process 200 of an embodiment of a digital human motion driving method according to the present application. The digital human motion driving method comprises the following steps:

[0034] Step 201: Receive a voice message sent by a user.

[0035] In this embodiment, the execution subject (for example Figure 1 The server or terminal device shown in the figure) can be applied to digital human action driving in various application scenarios, such as building digital human business cards, animation generation and other scenarios.

[0036] Step 202: Send the voice information to a first model for processing to obtain emotion parameters, wherein the first model is trained with common sense and emotion.

[0037] The language sentiment classification task is based on text sentiment classification. Compared with the text sentiment classification task, the language sentiment classification task is affected by the language environment context. However, identifying similar emotions often encounters difficulties, the speaker information is complex, the emotional response is uncertain, and the emotion recognition is difficult. Therefore, the emotion of the current sentence in the language depends more on the interaction between the context information and the real environment, rather than on the sentence text itself.

[0038] As mentioned above, general emotion recognition is difficult to accurately identify the user's emotions in certain scenarios. For example, when creating a digital business card, the user will send a short voice message to introduce himself. At this time, because the user sends a short voice message to introduce himself, in this scenario, since the voice message sent by the user contains relatively little content and there is no contextual semantics for reference, it is difficult for general emotion recognition models to accurately identify the user's emotions in the voice message sent by the user.

[0039] In this embodiment, in order to deeply explore the implicit information of the speaker (i.e., the user, hereinafter referred to as the speaker for ease of description), the first model is trained with common sense and emotion, and can perform emotion recognition based on human common sense in the emotion recognition process of some short voice information.

[0040] Step 203: Send the emotion parameters to the second model for processing to obtain action parameters.

[0041] In this embodiment, the original data set has emotion labels. In some examples, the original data set is collected from videos such as domestic and foreign TV series. Taking "Friends" as an example, it includes 97 episodes, 897 scenes and 12,606 utterances. There are 7 emotion labels in the data set: neutral, happy, peaceful, powerful, scared, crazy and sad.

[0042] During the training process of the second model, the above-mentioned execution subject first performs character recognition on the videos in the data set to form a skeletal action sequence data set, and then fine-tunes the training based on the large language model through the skeletal action sequence data set to finally obtain the second model.

[0043] Step 204: driving the digital human action based on the action parameters.

[0044] In this embodiment, in the process of driving the digital human's movements based on the movement parameters, the execution subject performs smoothing processing on the digital human in order to make the movements of the digital human more natural.

[0045] Specifically, in some examples, after the execution subject determines multiple digital human actions to drive the digital human through action parameters, the execution subject uses the amplitude of each digital human's limb action to take the middle position of the amplitude of the limb action as the default position of the multiple digital human actions, and finally connects the multiple digital human actions according to the default position to drive the digital human.

[0046] In other embodiments, after the execution subject determines multiple digital human actions through the action parameters, it determines the skeleton information corresponding to each digital human action, the skeleton information includes the spatial position and distance relationship, and finally merges two adjacent digital human actions using alignment and spherical linear interpolation to drive the digital human action. In some preferred examples, through the Slerp linear interpolation operation, through the smooth interpolation between two quaternions representing rotation, the digital human action transformation process looks smoother.

[0047] In some other embodiments, training can also be performed based on Transformer by using TEMOS. TEMOS is a 3D human motion generation framework designed based on the Transformer architecture, which can generate a variety of different human motions based on text descriptions. During training, a series of motion data will be divided into many subsets, each of which contains some transitional motions, so that the Transformer can learn to transition and output.

[0048] In summary, according to the solution of the present application, the first model that has been trained with common sense and emotion is used to process voice information. During the processing, common sense is used to judge emotions, which can significantly improve the model's recognition of emotions in voice information in usage scenarios without contextual semantic support. After obtaining the emotion parameters, the emotion parameters are sent to the second model for processing to obtain action parameters, and finally the digital human action is driven based on the action parameters.

[0049] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the digital human motion driving method according to this embodiment. Figure 3 In the application scenario, after receiving the voice information, the execution subject 301 uses the first model 302 to process the received voice information and convert the voice information into emotion parameters 303. The first model 302 in this embodiment has been trained in common sense and emotion. In the scenario without contextual semantic support, it can solve the problem of insufficient mining of implicit clue information of the speaker, thereby accurately identifying the emotions in the voice information. After obtaining the emotion parameters 303, the above-mentioned execution subject 301 uses the second model 302 to process the emotion parameters 303 to obtain the action parameters 304, and finally drives the action of the digital human through the action parameters.

[0050] Further references Figure 4 , which shows a process 400 of another embodiment of a digital human motion driving method. The process 400 is used for training a first model, and specifically includes the following steps:

[0051] Step 401 , determining common sense tasks and emotion tasks based on a dataset with emotion labels and a common sense knowledge base.

[0052] In this embodiment, the execution subject (for example Figure 1When training the first model, the server or terminal device shown in the figure first determines the common sense task template and the emotion task template respectively. Then, under the constraints of the dataset with emotion labels and the common sense knowledge base, the large language model generates common sense training data and emotion training data through the common sense task template and the emotion task template respectively. Finally, the common sense task is determined through the common sense training data, and the emotion task is determined through the emotion training data.

[0053] In some examples, the common sense knowledge base adopts knowledge ATOMIC. ATOMIC is a large-scale knowledge base that focuses on describing causal reasoning (if-then relationships) in human common sense knowledge. It not only includes factual descriptions such as "the earth revolves around the sun", but also covers the complex relationships between human behavior, social interaction, and psychological state.

[0054] Step 402, fine-tuning training is performed on the large language model through common sense tasks and emotion tasks to obtain the first model.

[0055] In this embodiment, during a round of fine-tuning training, the execution subject takes common sense training data as an example, inputs common sense training data or emotion training data into the large language model for forward calculation, calculates the loss value of the forward calculation result through the loss function, and performs reverse transmission to update the gradient of the large language model; calculates the score of the forward calculation result through a preset evaluation method, and determines whether to conduct the next round of common sense task training or determine that the common sense task training is completed and continue the emotion task training based on the loss value and score. When the loss value and score in the emotion task training process meet the preset requirements, the first model is obtained.

[0056] The following describes some specific parameter settings of process 400 in detail through some examples.

[0057] In order to deeply explore the implicit information of the speaker, the embodiment of the present application trains the model through common sense tasks and emotional tasks, which can understand the speaker's intention, the speaker's psychological state and the speaker's effect. The speaker common sense task template is as follows.

[0058] {"input": Now you are an expert at using common sense to make inferences. The following recorded conversation between "######" involves several speakers. ###U{u1,u2...u i-1}###Based on the above historical discourse, please use common sense to infer u i The potential audience response should be no more than ten words: "Target": "t i ”}.

[0059] Here U represents the historical dialogue of the current utterance, including utterance i-1. irepresents the current discourse. i It represents u i Common sense of the speaker.

[0060] The common sense recognition loss function is as follows:

[0061]

[0062] Among them, input i Hint indicating the speaker's common sense recognition. i is u i The common sense of the speaker labels in . n represents the number of utterances in the conversation.

[0063] Based on the same idea, the prompt template for emotion recognition is as follows:

[0064] {"input": Now you are an expert in sentiment and emotion analysis. The following recorded conversation between "######" involves multiple speakers. ###U{u1,u2...u i-1}###Please select from <Happy, Sad, Neutral i Emotional labels, anger, excitement, frustration > "," target ": "e i ”}.

[0065] Here U represents the historical dialogue of the current utterance, including utterance i-1. i Represents the current discourse. i Represents u i The speaker's emotion label.

[0066] The loss function for emotion recognition is as follows:

[0067]

[0068] Here input i Represents the prompt for emotion recognition in the conversation. i is the sentiment label of the current utterance. n represents the number of utterances in the conversation.

[0069] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a digital human motion driving device. Figure 2 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 2 The device can be applied to various electronic devices.

[0070] like Figure 5As shown, the digital human driving device 500 of this embodiment includes: a first unit 501, a second unit 502, a third unit 503 and a fourth unit 504. The first unit 501 is used to receive voice information sent by a user; the second unit 502 is used to send the voice information to the first model for processing to obtain emotional parameters, wherein the first model is trained with common sense and emotion; the third unit 503 is used to send the emotional parameters to the second model for processing to obtain action parameters; the fourth unit is used to perform digital human action driving based on the action parameters.

[0071] In this embodiment, the specific processing of the first unit 501, the second unit 502, the third unit 503 and the fourth unit 504 of the digital human motion driving device 500 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0072] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0073] like Figure 6 , is a block diagram of an electronic device according to a digital human action driving method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0074] like Figure 6 As shown, the electronic device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6A processor 601 is taken as an example.

[0075] The memory 602 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to execute the digital human motion driving method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to execute the digital human motion driving method provided in the present application.

[0076] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the digital human motion driving method in the embodiment of the present application (for example, the attached Figure 5 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the digital human action driving method in the above method embodiment is realized.

[0077] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the digital human motion-driven electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the digital human motion-driven device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0078] The electronic device of the digital human motion driving method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0079] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the digital human motion-driven electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a track ball, a joystick, and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0080] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0083] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0084] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0085] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be set in a processor. For example, they may be described as: a processor includes a first unit, a second unit, a third unit, and a fourth unit. The names of these units do not constitute limitations on the units themselves in some cases. For example, the first unit may also be described as a "unit for receiving voice information sent by a user."

[0087] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: receives voice information sent by the user; sends the voice information to the first model for processing to obtain emotion parameters, wherein the first model has been trained with common sense and emotion; sends the emotion parameters to the second model for processing to obtain action parameters; and drives the digital human action based on the action parameters.

[0088] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for driving digital human motion, characterized in that: The method comprises: Receive voice messages sent by users; Sending the speech information to a first model for processing to obtain an emotion parameter, wherein the first model is trained with common sense and emotion; Send the emotion parameters to the second model for processing to obtain action parameters; The digital human action is driven based on the action parameters.

2. The method according to claim 1, characterized in that The training method of the first model includes: Determine common sense tasks and emotion tasks based on emotion-labeled datasets and common sense knowledge bases; The first model is obtained by fine-tuning the large language model through common sense tasks and sentiment tasks.

3. The method according to claim 2, characterized in that Determine common sense tasks and emotion tasks based on emotion-labeled datasets and common sense knowledge bases, including: Determine common sense task templates and emotion task templates respectively; Under the constraints of the dataset with emotion labels and the common sense knowledge base, the large language model generates common sense training data and emotion training data through the common sense task template and the emotion task template respectively; determining a common sense task through the common sense training data; An emotion task is determined using the emotion training data.

4. The method according to claim 2, characterized in that: The first model is obtained by fine-tuning the common sense task and the emotion task on the large language model, including: During a round of fine-tuning training, common sense training data or emotion training data is input into the large language model for forward calculation; Calculate the loss value of the forward calculation result through the loss function, and perform reverse transmission to update the gradient of the large language model; Calculate the score of the forward calculation result by a preset evaluation method; Based on the loss value and the score, determine to perform the next round of data set training or obtain the first model.

5. The method according to claim 2, characterized in that: The training method of the second model includes: Identify the character actions in the data set with emotion labels to form a skeleton action sequence data set; The second model is obtained by fine-tuning the skeleton action sequence data set on the basis of the large language model.

6. The method according to claim 1, characterized in that Driving the digital human action based on the action parameters includes: Determine multiple digital human actions through the action parameters; Determining default positions of the plurality of digital human actions; The plurality of digital human actions are connected according to the default position to drive the digital human actions.

7. The method according to claim 1, characterized in that Driving the digital human action based on the action parameters includes: Determine multiple digital human actions through the action parameters; Determine the skeleton information corresponding to each digital human action, wherein the skeleton information includes a spatial position and a distance relationship; The two adjacent digital human movements are fused using alignment and spherical linear interpolation to drive the digital human movements.

8. A digital human motion driving device, characterized in that: include: The first unit is used to receive the voice message sent by the user; A second unit is used to send the voice information to a first model for processing to obtain an emotion parameter, wherein the first model is trained with common sense and emotion; The third unit is used to send the emotion parameters to the second model for processing to obtain the action parameters; The fourth unit is used to drive the digital human action based on the action parameters.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images