A method for generating motion for a virtual character, a method for building a motion library, a device for building a motion library for a virtual character, a computer device, and a computer program.

Abstract

The present application relates to a method for generating motion for a virtual character, a method for constructing a motion library, and an apparatus therefor, which belong to the field of computer technology. The method includes the steps of: acquiring a voice and text of a virtual character, the text indicating semantic information of the voice (201); determining a semantic tag for the text based on the text, the semantic tag indicating at least one of part-of-speech information of words in the text or emotional information expressed by the text (202); searching a predetermined motion library for motion categories matching the semantic tag and motion data belonging to the motion categories (203), the predetermined motion library including motion data for virtual characters belonging to multiple motion categories; and generating a motion sequence for the virtual character based on the motion data, the motion sequence controlling the virtual character to perform a motion that matches the voice (204). The present application not only improves the efficiency of motion generation for a virtual character but also improves the accuracy of motion generation.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application filed on May 15, 2023, with the application number 202310547509.7 and the invention title "Method for Generating Motions of Virtual Characters, Method for Constructing a Motion Library, and Device", and all its contents are incorporated herein by reference.

[0002] This application relates to the technical field of computers, and particularly to a method for generating motions of virtual characters, a method for constructing a motion library, and a device.

Background Art

[0003] With the development of computer technology, the application of virtual characters in live streaming, videos, animations, games, virtual social, human-computer interaction, etc. has become increasingly widespread. Taking the live streaming scene as an example, when a virtual character conducts live streaming or interaction as a caster, regarding the generation of motions of the virtual character in order to improve the rendering effect of the virtual character.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of this application provide a method for generating motions of virtual characters, a method for constructing a motion library, and a device, which can quickly and efficiently synthesize a motion sequence with a higher accuracy rate for virtual characters, and improve the efficiency of generating motions of virtual characters. The technical means are as follows.

Means for Solving the Problems

[0005] According to one aspect, there is provided a method for generating motions of virtual characters executed by a computer device, the method comprising: A step of obtaining the voice and text of a virtual character, wherein the text indicates the semantic information of the voice, A step of determining a semantic tag of the text based on the text, wherein the semantic tag represents at least one of the part-of-speech information of a word in the text or the sentiment information expressed by the text; A step of searching a predetermined motion library for motion categories that match the semantic tag and motion data belonging to the motion category, wherein the predetermined motion library includes motion data belonging to multiple motion categories of the virtual character. The method includes the step of generating a motion sequence for the virtual character based on the motion data, wherein the motion sequence controls the virtual character to perform motions that synchronize with the sound.

[0006] According to one embodiment, a method is provided for constructing a motion library for a virtual character executed by a computer device, and the method is A step of obtaining a sample motion sequence, reference audio, and reference text for each sample character, wherein the reference text indicates semantic information of the reference audio, and the sample motion sequence controls the sample character to perform motions that match the reference audio. A step of dividing the sample motion sequence into a plurality of sample motion segments based on the relationship between a word in the reference text and a phoneme in the reference audio, wherein each sample motion segment comprises a step relating to one word in the reference text and one phoneme in the reference audio, A step of clustering each sample motion segment of each sample character based on the motion characteristics of the aforementioned sample motion segment and obtaining multiple motion sets, wherein each motion set represents motion data belonging to the same motion category and belonging to different sample characters. The process includes the step of building a motion library based on a plurality of the aforementioned motion sets.

[0007] According to one embodiment, a motion generation device for a virtual character is provided, and the device is An acquisition module for acquiring the voice and text of a virtual character, wherein the text includes an acquisition module that indicates the semantic information of the voice, An analysis module that determines the semantic tags of the text based on the text, wherein the semantic tags represent at least one of the part-of-speech information of the words in the text or the sentiment information expressed by the text, A search module that searches for motion categories and motion data belonging to a given motion category that match the given semantic tag from a predetermined motion library, wherein the predetermined motion library includes a search module that includes motion data belonging to multiple motion categories of the virtual character, A generation module that generates a motion sequence for the virtual character based on the motion data, the generation module comprising a generation module that controls the virtual character to perform motions synchronized with the sound.

[0008] According to one embodiment, a device for constructing a motion library for a virtual character is provided, and the device is A sample acquisition module that acquires a sample motion sequence, reference audio, and reference text for each sample character, wherein the reference text indicates the semantic information of the reference audio, and the sample motion sequence controls the sample character to perform motions that match the reference audio. A segmentation module that divides the sample motion sequence into a plurality of sample motion segments based on the relationship between a word in the reference text and a phoneme in the reference audio, wherein each sample motion segment comprises a segmentation module associated with one word in the reference text and one phoneme in the reference audio, A clustering module that clusters each sample motion segment of each sample character based on the motion characteristics of the sample motion segment and obtains multiple motion sets, wherein each motion set represents motion data belonging to the same motion category and belonging to different sample characters, Includes a build module that constructs a motion library based on multiple motion sets.

[0009] According to one embodiment, a computer device is provided that includes one or more processors and one or more memories, the one or more memories storing at least one computer program, and the at least one computer program is loaded and executed by the one or more processors, thereby realizing a method for generating motion for a virtual character or a method for constructing a motion library for a virtual character according to any of the possible embodiments described above.

[0010] According to one embodiment, a computer-readable storage medium is provided, which stores at least one computer program, and which is loaded and executed by a processor to realize a method for generating motion for a virtual character or a method for constructing a motion library for a virtual character according to any of the possible embodiments described above.

[0011] According to one embodiment, a computer program product is provided, the computer program product comprising one or more computer programs, the one or more computer programs being stored on a computer-readable storage medium. One or more processors of a computer device can read the one or more computer programs from the computer-readable storage medium, and by executing the one or more computer programs, the computer device can execute a method for generating virtual character motion or a method for constructing a virtual character motion library according to any of the possible embodiments described above. [Brief explanation of the drawing]

[0012] To more clearly illustrate the technical means in the embodiments of the present application, the drawings that need to be used in the description of the embodiments are briefly described below, and as will be apparent, the drawings in the following description are only a few embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without any creative work. [Figure 1] This is a schematic diagram of the implementation environment for the motion generation method for a virtual character according to an embodiment of the present invention. [Figure 2] This is a flowchart of the motion generation method for a virtual character according to an embodiment of the present invention. [Figure 3] This is a flowchart of the motion generation method for a virtual character according to an embodiment of the present invention. [Figure 4] It is a principle diagram of a method for generating motion of a virtual character according to an embodiment of the present application. [Figure 5] It is a flowchart of a method for constructing a motion library of a virtual character according to an embodiment of the present application. [Figure 6] It is a principle diagram of a method for constructing a motion library according to an embodiment of the present application. [Figure 7] It is a principle diagram of data cleaning of a motion set according to an embodiment of the present application. [Figure 8] It is a principle diagram of data replenishment of a new additional motion segment according to an embodiment of the present application. [Figure 9] It is a schematic configuration diagram of a motion generation device for a virtual character according to an embodiment of the present application. [Figure 10] It is a schematic configuration diagram of a device for constructing a motion library of a virtual character according to an embodiment of the present application. [Figure 11] It is a schematic configuration diagram of a computer device according to an embodiment of the present application.

Embodiments for Carrying Out the Invention

[0013] To make the object, technical means, and advantages of the present application clearer, the embodiments of the present application will be described in more detail below with reference to the drawings.

[0014] Terms such as "first" and "second" in the present application are used to distinguish the same item or similar items whose functions and operations are substantially the same. Additionally, there is no logical or chronological dependency between "first", "second", and "n", nor are they limited by numbers and execution orders.

[0015] The term "at least one" in the present application means one or more, and "a plurality" means two or more. For example, a plurality of motion segments means two or more motion segments.

[0016] In this application, the term "including at least one of A or B" refers to cases where A is included only, B is included only, and both A and B are included.

[0017] User-related information (including, but not limited to, user device information, personal information, and behavioral information), data (including, but not limited to, data used for analysis, stored data, and displayed data) and signals relating to this application, when applied to a specific product or technology in the manner of the embodiments of this application, must all be obtained with the user's permission or consent, or fully authorized by each party, and the collection, use, and processing of related information, data, and signals must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the motion data of virtual characters relating to this application must all be obtained with full authorization.

[0018] Artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or devices controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that aims to understand the essence of intelligence and create new intelligent devices that can respond in a manner similar to human intelligence. Artificial intelligence involves studying the design principles and implementation methods of various intelligent devices so that they can have the functions of sensing, reasoning, and deciding.

[0019] Artificial intelligence technology is a comprehensive discipline, encompassing a wide range of related fields, including not only hardware technologies but also software technologies. The fundamental technologies of artificial intelligence generally include sensors, specialized AI chips, cloud computing, distributed memory, big data processing technologies, operating / interaction systems, and mechatronics. Software technologies in artificial intelligence primarily encompass several major areas, such as computer vision technologies, speech processing technologies, natural language processing technologies, and machine learning / deep learning, autonomous driving, and smart mobility.

[0020] Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, and among these, speech will be one of the most promising forms of human-computer interaction. Key technologies in speech technology include automatic speech recognition (ASR), speech synthesis, and voiceprint recognition.

[0021] Machine learning (ML) is a multidisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or replicate human learning behavior to acquire new knowledge and skills, and how they reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence, a fundamental method for giving computers intelligence, and its applications cover all areas of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0022] Natural Language Processing (NLP) is a crucial field in computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field is closely related to linguistics, as it deals with natural language, i.e., the language that humans use in everyday life. Natural Language Processing techniques typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0023] With the ongoing research and advancement of artificial intelligence (AI) technology, it has come to be studied and applied in many fields, such as general smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, unmanned aerial vehicles, robotics, smart healthcare, smart customer service, car internet, and smart mobility. As technology develops, AI technology is expected to be applied to even more fields and play an increasingly important role.

[0024] The technical means relating to the embodiments of this application relate to artificial intelligence voice technology, NLP, and machine learning, and more specifically, to the application of the above various technologies or combinations thereof to the generation of motion for virtual characters, which will be explained in the following embodiments.

[0025] The following terms relating to the embodiments of this application will be explained.

[0026] A virtual character is an object that can operate in a virtual world, and is a virtual, anthropomorphic digital character in a virtual world, such as a virtual person, anime character, or virtual role. A virtual character may be a three-dimensional model, and the three-dimensional model may be a three-dimensional character constructed based on three-dimensional human skeleton technology. Preferably, a virtual character may be realized as a 2.5-dimensional or 2-dimensional model, and the embodiments of this application are not limited thereto. A 3D model of a virtual character may be created using MMD (Miku Miku Dance, 3D computer graphics software) or the Unity engine, and of course, a 2D model of a virtual character may be created using Live2D (2D computer graphics software). The dimensions of the virtual character are not specifically limited here.

[0027] The metaverse, also known as the metauniverse, metaphysical universe, hypersensory space, or virtual space, is a 3D virtual world network focused on social links. The metaverse is about persistent and decentralized online 3D virtual environments.

[0028] A digital human is a virtual character created by 3D modeling the human body using information science methods, achieving the effect of simulating and mimicking the human body. In other words, a digital human is a digital character created using digital technology that closely resembles a human image. Digital humans are widely applied in various scenarios such as video production, live streaming, industry broadcasting, social entertainment, and voice prompts. For example, a digital human can serve as a virtual caster or virtual avatar. Digital humans are also sometimes called virtual humans or virtual digital humans.

[0029] A virtual caster refers to a caster who uses a virtual character, such as a Virtual YouTuber (VTuber) or Virtual Uploader (VUP), to post content on video sites. Typically, virtual casters operate on video sites and social platforms with original virtual persona settings and characters, enabling various forms of human-computer interaction such as broadcasting, performance, live streaming, and conversation.

[0030] The "person behind the scenes" refers to the person who performs in the background or controls the virtual caster during a live stream. For example, sensors attached to the person's head and body are used to capture their body motion and facial expressions using an optical motion capture system. This motion data is then synchronized with the virtual caster. In this way, a real-time motion capture mechanism enables real-time interaction between the virtual caster and viewers watching the live stream.

[0031] Motion capture (MoCap), also known as motion capture, involves placing sensors on key parts of a moving object or a real person, capturing the sensor positions using a motion capture system, and then processing this data on a computer to obtain motion data in three-dimensional spatial coordinates. After the motion data is recognized by the computer, it can be applied to fields such as animation production, gait analysis, biomechanics, and man-machine engineering. Common motion capture devices include motion capture suits and are often applied to generating motion for 3D virtual characters. A real person wears the motion capture suit and performs motions, and the 3D skeletal data of the human body captured by the motion capture system is transferred to a 3D model of a virtual character. This 3D skeletal data of the virtual character is then used to control the 3D model of the virtual character so that it performs the same motions as the real person.

[0032] Optical motion capture is a type of equipment used in engineering fields related to information and systems science.

[0033] Inertial motion capture uses inertial sensors to measure the movement of major skeletal parts of the human body in real time, measure the position of human joints based on the principles of inverse kinematics, and input the data to the corresponding (virtual character) skeleton.

[0034] Tokenization refers to the process of dividing a given segment of text into a data structure consisting of words (tokens), each containing one or more characters.

[0035] A word (token) is obtained by performing word segmentation on a given text segment, dividing the text into a single word list. Each element in the word list is a token obtained through word segmentation, and each token contains one or more characters. For example, the text "I am having a lot of fun" is segmented to obtain a single token list {"I", "very", "fun"}.

[0036] A phoneme is the smallest unit of speech, divided based on the natural attributes of the sound. It is analyzed based on the pronunciation motion within a syllable, with one motion constituting one phoneme. For example, each letter in a word can be divided into one or more phonemes according to its pronunciation motion.

[0037] Phoneme alignment involves taking a single segment of speech and the text corresponding to the meaning of that speech, dividing the phonemes of each character in the text, and aligning them to each speech frame on the speech timeline. That is, for each character in the text, one or more phonemes are determined according to the pronunciation motion of that character, and then one or more speech frames are found in the speech that emit each phoneme. In this way, all the speech frames covered by all the phonemes necessary to speak this character constitute a single speech segment, and the timestamp interval of this speech segment is found on the speech timeline, thereby reflecting in which timestamp interval in the speech the speaker speaks this character.

[0038] Interpolation is a motion prediction and motion compensation method that can extend the number of motion frames in a motion segment to ensure continuity when the number of frames is insufficient. For example, one new motion frame can be inserted between each of the original two motion frames in a motion segment, and the new motion frame can be used to fill in the intermediate state of motion change in the two motion frames.

[0039] Text sentiment analysis is a process that takes a single segment of text and performs analysis, processing, induction, and inference on that text. It is also called opinion mining or trend analysis because it typically outputs the sentiment tag with the highest degree of matching to the text. Depending on the granularity of the processed text, sentiment analysis is broadly categorized into three research levels: word level, sentence level, and paragraph level. Methods for text sentiment analysis are broadly categorized into four types: keyword recognition, lexical association, statistical methods, and conceptual-level techniques.

[0040] The technical concept of the embodiment of this application will be explained below.

[0041] With the rapid development of technologies such as 3D modeling, virtual reality (VR), augmented reality (AR), and the metaverse, the application of virtual characters to live streaming, video, animation, games, virtual social media, and human-computer interaction is expanding more and more.

[0042] Taking a live streaming scene as an example, the virtual character acts as a broadcaster or presenter, and the method relates to generating motion for the virtual character in order to improve its rendering effect. Similarly, in a video creation scene, for example, when creating a video posted by a virtual presenter, a digital human video, etc., the method also relates to generating motion for the virtual character.

[0043] Typically, when generating the body motion of a virtual character, a motion capture method is used. This involves a real person (or actor) wearing a motion capture suit equipped with full-body sensors, performing motion based on script content and audio, and the motion capture suit capturing the motion data of the real person's performance (i.e., 3D skeletal data of the human body). This data is reported to a computer connected to the motion capture suit, which then transfers the 3D skeletal data of the human body to a 3D model of the virtual character, obtaining the 3D skeletal data of the virtual character. Subsequently, the 3D skeletal data of the virtual character forms a motion sequence in a series of consecutive time points. A professional animator then adds some adjustments or corrections to the motion sequence of the virtual character to modify the motion, ultimately obtaining a series of scripted virtual character motion expressions. The motion generation method based on motion capture described above requires artificial intervention throughout the entire process, and each captured 3D human skeleton data is customized based on a specific script, making it unusable and lacking versatility. In other words, if voice or text not present in the script appears, motion generation cannot be achieved, and this new voice or text must be performed as a new script, resulting in low motion generation efficiency.

[0044] Furthermore, when generating the body motion of a virtual character, video motion capture is performed based on a large amount of publicly available 2D video material (e.g., lecture videos, talk show videos, etc.) to obtain 2D video data, which is then converted into 3D skeletal data. A training dataset is then constructed using the 3D skeletal data and the audio and text marked on it, and a motion generation model is trained. This allows the motion generation model to generate the body motion of the virtual character in a voice-driven manner. However, because the data source is single and human body motion is complex, the effect of the motion generation model is not ideal, and the resulting synthesized virtual character has problems such as flat body motion and inaccurate performance, resulting in poor accuracy in motion generation.

[0045] In view of this, the embodiment of the present invention provides a method for constructing a motion library for a virtual character. This method divides the sample motion sequences into sample motion segments based on a large number of sample objects, their reference texts, and reference audio. The sample motion segments are then matched to their respective motion categories. Furthermore, motion data cleaning and filtering are performed on the motion sets of each motion category, ultimately enabling the construction of a complete motion library for a virtual character, which can cover many motion categories. Next, based on the constructed motion library, a speech-triggered body motion generation algorithm framework can be provided. When generating virtual character motion in real time, the user only needs to provide one segment of audio and a portion of the text indicating its meaning. The device can then quickly generate 3D data of body motion triggered by the audio and text, and output a motion sequence for the virtual character. The entire motion generation process requires no artificial intervention. The device can quickly and accurately generate motion sequences that match the audio and text, resulting in high motion generation efficiency and high motion generation accuracy.

[0046] If motion segments are searched and motion sequences are synthesized based solely on similarity to the input speech, without considering the semantic information of the text modal, the final body motion will easily change based only on the rhythm of the speech, failing to reflect actual semantic body motion and merely repeating dialogue motion effects without expressing semantic accuracy and richness. This clearly results in poor motion generation and a poor simulation of the virtual character.

[0047] However, in the above technical means, by considering dual-mode information of voice and text, driving the generation of body motion of the virtual character, combining text and voice, considering the relationship between the two, and matching the text meaning with a rich set of semantic motions from a predetermined motion library as guidance, the expression of the virtual character's body motion in the synthesized motion sequence can be made more accurate, richer, and more lifelike. It can be applied to scenes where various virtual characters need to perform motion, such as virtual live streaming and digital human video, achieving accuracy at the level of motion capture, but with motion generation efficiency far superior to motion capture methods.

[0048] The implementation environment for the embodiments of this application will be described below.

[0049] Figure 1 is a schematic diagram of the implementation environment for a virtual character motion generation method according to an embodiment of the present application. As shown in Figure 1, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected directly or indirectly via a wireless network or a wired network, and are not limited to the present application.

[0050] Terminal 101 has an application installed that supports virtual characters, and through this application, terminal 101 can implement functions such as generating body motions for virtual characters. Of course, the application may also have other functions, such as network social functions, video sharing functions, video posting functions, or chat functions. The application may be a native application within the operating system of terminal 101 or an application provided by a third party. For example, the application may include, but is not limited to, live streaming applications, short video applications, audio-video applications, game applications, social applications, 3D animation applications, or other applications, and the embodiments of this disclosure are not limited thereto.

[0051] Preferably, the terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0052] Server 102 provides background services to applications supporting virtual characters on terminal 101, creates and maintains a motion library for virtual characters, and caches 3D skeletal models of multiple types of virtual characters. Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Preferably, server 102 is responsible for the primary calculation of motion generation and terminal 101 is responsible for the secondary calculation of motion generation, or server 102 is responsible for the secondary calculation of motion generation and terminal 101 is responsible for the primary calculation of motion generation, or cooperative calculation is performed between server 102 and terminal 101 using a distributed computing architecture.

[0053] Preferably, server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0054] Terminal 101 generally refers to one of several terminals, and the embodiments of this disclosure will be described using only terminal 101 as an example. As those skilled in the art will know, the number of terminals may be more or less.

[0055] In one exemplary scene, during a real-time body motion generation process, the user uploads a single segment of audio to an application on terminal 101 to trigger a motion generation command. Terminal 101 responds to the motion generation command by sending a motion generation request to server 102, which carries the audio. Server 102 responds to the motion generation request by performing automatic speech recognition (ASR) on the audio and obtaining text that indicates the meaning of the audio. Next, using the audio and text, it executes the virtual character motion generation method according to the embodiment of the present invention, searches for appropriate motion data from a predetermined motion library, and then synthesizes a motion sequence that matches the audio. In this way, terminal 101 achieves motion generation of a virtual character driven by sound, while server 102 uses dual-mode information of audio and text to synthesize a virtual character body motion that can represent the semantic level of the audio (or text).

[0056] In another exemplary scene, during a real-time body motion generation process, the user uploads a single segment of text to an application on terminal 101 to trigger a motion generation command. Terminal 101 responds to the motion generation command by sending a motion generation request to server 102, which carries the text. Server 102 responds to the motion generation request by finding a sound source library for a virtual character, generating audio from the sound source library to broadcast the text (i.e., incorporating audio into the text), and then using the audio and text, executes the virtual character motion generation method according to the embodiment of the present invention, searching for appropriate motion data from a predetermined motion library, and further synthesizing a motion sequence that matches the text. In this way, terminal 101 achieves text-driven virtual character motion generation, while server 102 uses dual-mode information of audio and text to synthesize a virtual character body motion that can represent the semantic level of the audio (or text).

[0057] In yet another exemplary scene, in a real-time body motion generation process, the user uploads a single segment of audio and its corresponding text (i.e., text representing the semantic information of the audio) to the application on terminal 101 to trigger a motion generation command. Terminal 101 responds to the motion generation command by sending a motion generation request to server 102, which carries the audio and text. Server 102 responds to the motion generation request by using the audio and text to execute the virtual character motion generation method according to the embodiment of the present invention, searches for appropriate motion data from a predetermined motion library, and further synthesizes a motion sequence that matches the audio and text. In this way, terminal 101 realizes virtual character motion generation through the joint driving of audio and text, while server 102 synthesizes a virtual character body motion that can represent the semantic level of the audio (or text) using double-mode information of audio and text.

[0058] In each of the above types of scenes, the terminal 101 provides either single-modal or double-mode information as a drive signal. Through a conversion means between text and speech in audio technology, the server 102 synthesizes the virtual character's body motion using the double-mode information of speech and text. In this way, the final motion sequence not only moves according to the rhythm of the speech, but can also express rich semantic information at the word level, and consequently expresses the emotional state of the virtual character during broadcast. This results in high motion generation efficiency, high accuracy, and the generated body motions that match the rhythm of the speech, possess rich semantic information, significantly improve the simulation accuracy of the virtual character, and greatly optimize the rendering effect.

[0059] The motion generation method for a virtual character according to the embodiment of the present invention can be applied to any scene where it is necessary to generate body motion for a virtual character. For example, in a digital human live streaming scene, the person inside does not need to wear a motion capture suit to perform; by simply providing at least one of the text or voice during live streaming interaction, the digital human can be controlled to perform body motions in sync with the voice and its subtitles (which may not be present) in the live stream, driven by dual-mode information of voice and text, thereby improving the authenticity and entertainment value of the digital human live stream. Also, for example, in a scene where a digital human video is created, the user can control the generation of body motions of the digital human to match the voice or text simply by creating the audio or text for the video. Furthermore, by combining the body motion (i.e., video screen) and audio (i.e., video audio) into a single-segment digital human video, video posting and distribution can be performed, improving the efficiency of digital human video generation and enhancing the convenience and flexibility of its creation. Furthermore, the embodiment of this invention can be applied to various scenes where it is necessary to generate the body motion of a virtual character, such as digital human customer service, animation production, video visual effects, and digital human casters, and the embodiment of this invention does not specifically limit the application scenes.

[0060] The following describes the flow of the motion generation method for a virtual character according to an embodiment of the present invention.

[0061] Figure 2 is a flowchart of a motion generation method for a virtual character according to an embodiment of the present invention. As shown in Figure 2, the embodiment is executed by a computer device, and the computer device is described as a server as an example, and the server may be server 102 in the above implementation environment, and the embodiment includes the following steps 201 to 204.

[0062] In step 201, the server retrieves the voice and text of the virtual character, with the text indicating the semantic information of the voice.

[0063] A virtual character is an object that can operate in a virtual world, and is a virtual, anthropomorphic digital character in a virtual world. For example, a virtual character includes, but is not limited to, game characters, virtual casters, virtual avatars, video characters, anime characters, digital humans, and virtual humans, and the embodiments of this application do not specifically limit what constitutes a virtual character.

[0064] In embodiments of the present invention, if it is necessary to control the virtual character to broadcast sound, the server generates a motion sequence for the virtual character, as it is necessary to control the virtual character to perform motions that correspond to the sound.

[0065] The audio includes at least one audio frame, and the text is text that represents the semantic information of the audio, and the text includes at least one word, each word containing at least one character. The audio and text are related, that is, the text is semantic information obtained by performing ASR recognition on the audio, or the audio is an audio signal emitted when broadcasting the text, and the audio signal may be a synthesized signal output by a machine or a human voice signal collected by a microphone, and the type of audio signal is not specifically limited here.

[0066] In some embodiments, the server searches a local database for a pair of related audio and text; or the server retrieves a single segment of audio from the local database, performs ASR recognition on the audio to obtain text that represents the semantic information of the audio; or the server retrieves a single segment of text from the local database, performs speech synthesis on the text to obtain audio that incorporates the audio into the text.

[0067] In some other embodiments, the server downloads a related pair of audio and text from a cloud database, or the server downloads a single segment of audio from the cloud database, performs ASR recognition on the audio to obtain text indicating the semantic information of the audio, or the server downloads a single segment of text from the cloud database, performs speech synthesis on the text to obtain audio to incorporate the audio into the text.

[0068] In some embodiments, the server receives a related pair of audio and text uploaded from a terminal, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request and obtains the audio and text. Alternatively, the server receives audio uploaded from a terminal, performs ASR recognition on the audio and obtains text indicating the semantic information of the audio, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request, obtains the audio, performs ASR recognition on the audio and obtains text indicating the semantic information of the audio. Alternatively, the server receives text uploaded from a terminal, performs speech synthesis on the text and obtains audio that adds narration to the text, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request, obtains the text, performs speech synthesis on the text and obtains audio that incorporates the audio into the text.

[0069] In the above process, since text and audio can be converted to each other, the user may provide only audio, only text, or both audio and text simultaneously. In addition to what the user specifies, the audio may be read from a local database or downloaded from a cloud database. The embodiments of the present application do not specifically limit the sources of the audio and text.

[0070] The server, after acquiring audio and text, generates a motion sequence by performing the method according to the embodiment of the present invention, matches the motion sequence with the semantic information of the text, and then controls the virtual character to perform the body motions shown in the motion sequence in a process to broadcast the audio, thereby matching the semantic information of the body motions performed by the virtual character with the broadcast audio.

[0071] In step 202, the server determines the semantic tags of the text based on the text, and these semantic tags represent either the part-of-speech information of the words in the text or the sentiment information expressed by the text.

[0072] In some embodiments, the server analyzes the text obtained in step 201 to obtain at least one semantic tag of the text, which may include at least one of part-of-speech tags or sentiment tags, wherein the part-of-speech tag represents part-of-speech information of a word in the text, where part-of-speech information refers to information for describing the part of speech of the word, such as subject, verb, state, etc., and the sentiment tag represents sentiment information expressed by the text, where sentiment information refers to information for describing the emotion expressed by the text, such as joy, disappointment, anger, etc. Both part-of-speech information and sentiment information describe the text, but from different angles, and embodiments of the present application do not specifically limit the content of semantic tags. The number of semantic tags may be one or more, and embodiments of the present application do not specifically limit the number of semantic tags.

[0073] In some embodiments, the server determines, based on the text, at least one word contained in the text, determines the part-of-speech tag to which each word belongs, and uses the part-of-speech tags of all words in the text as the semantic tags of the text. The method for extracting part-of-speech tags will be described in detail in the following embodiments and will not be described here.

[0074] In another embodiment, the server determines at least one sentiment tag for the text based on the text, and uses this at least one sentiment tag as the semantic tag for the text. The method for extracting sentiment tags will be described in detail in the next embodiment and will not be described here.

[0075] In some further embodiments, the server determines the part-of-speech tag for each word based on the text, determines the sentiment tag for each word based on the text, and then uses the part-of-speech tag and sentiment tag together as the semantic tag for the text.

[0076] In one example, the text "This is my first live stream!" is subjected to word segmentation to obtain a word list {"I", "first", "live stream!"}. In the parts-of-speech table, it is searched to find that the part-of-speech tag to which the word "I" belongs is "subject", the part-of-speech tag to which the word "first" belongs is "state", and the part-of-speech tag to which the word "live stream!" belongs is "verb". Furthermore, based on the text, it is determined that the emotion tag of the text is "joy", and finally, the four semantic tags "subject", "state", "verb", and "joy" are output.

[0077] In the process described above, by analyzing a given text, feature information at the semantic level of the text can be extracted, and this feature information can be represented in a simple manner such as semantic tags. In the motion generation process, these semantic tags at the semantic level are convenient to use as guide signals, and furthermore, they match the semantics of the speech well, which is advantageous for synthesizing natural body motions of virtual characters.

[0078] In step 203, the server searches a predetermined motion library for motion categories that match the semantic tag and motion data belonging to those motion categories, and the predetermined motion library includes motion data belonging to multiple motion categories for the virtual character.

[0079] In some embodiments, for each semantic tag obtained in step 202, the semantic tag is used as an index to search for a motion category that matches the semantic tag from among multiple candidate categories in a predetermined motion library. The predetermined motion library is a motion database created and maintained on the server side, which stores motion sets for each motion category, with each motion set containing motion data clustered within that motion category. The method for creating the predetermined motion library will be described in detail in subsequent embodiments and will not be described here.

[0080] Furthermore, in a single possible implementation scenario, it is not always possible to find a motion category that matches every semantic tag. If a semantic tag does not match any of the candidate categories, one predetermined motion category can be designated as the motion category that matches the semantic tag, in order to avoid a gap of a certain amount of time in the motion sequence. The predetermined motion category may be a default motion category set in advance by the engineer, for example, a standing motion category without a semantic, or a sitting motion category. Here, the predetermined motion category is not specifically limited, and the engineer may set different predetermined motion categories for different virtual characters.

[0081] In the above process, using the text's semantic tags as an index, the system can search a predetermined motion library for the motion category that best matches the audio and semantic level. This motion category not only moves according to the rhythm of the audio, but can also be highly aligned with the semantic information of the audio, reflecting the emotional tendencies and underlying meanings of the virtual character during voice broadcast. Thus, the motion data selected from this motion category can be used to synthesize a more accurate motion sequence for the virtual character.

[0082] After searching for a motion category that matches the voice, motion data belonging to that motion category is retrieved from a predetermined motion library, and this motion data can be controlled so that the virtual character performs a specific motion.

[0083] In step 204, the server generates a motion sequence for the virtual character based on the motion data, and the motion sequence controls the virtual character to perform motions that match the sound.

[0084] In some embodiments, after setting a motion category for each semantic tag, motion data belonging to that motion category can be searched from a predetermined motion library for each semantic tag. For example, the motion data may include 3D skeleton data for multiple frames in consecutive time (i.e., the 3D skeleton data for each frame may be called a motion frame), and the 3D skeleton data for each frame may include pose data for each skeleton keypoint in the motion screen displayed in that frame. In this way, the virtual character can be controlled to perform a specific motion simply by transitioning the 3D skeleton data for each frame to the 3D skeleton model of the virtual character. Next, the motion data matched to each semantic tag can be spliced ​​according to the timestamp order in the audio of the word corresponding to each semantic tag to form a motion sequence for the virtual character. This motion sequence represents the changes in body motion over consecutive time periods in which the virtual character broadcasts audio, and controls the virtual character to perform body motions that match the audio when it broadcasts audio.

[0085] In some embodiments, a phoneme alignment tool can be used to find a timestamp interval corresponding to each semantic tag in the audio timeline. This timestamp interval refers to the time period when the virtual character broadcasts a word belonging to that semantic tag. Next, motion data matching the semantic tag is searched from the motion set of the motion category in a predetermined motion library. Furthermore, this motion data is used to pad this timestamp interval in the motion sequence, and the motion data within the timestamp interval connecting each end constitutes a motion sequence of the virtual character at consecutive times. The phoneme alignment method and the motion data search method will both be described in detail in the following embodiments and will not be described here.

[0086] In the process described above, each motion frame in the final synthesized motion sequence is aligned with the timestamp of each audio frame in the audio. This ensures that the motion frames reflect body motions that match the audio frames at a semantic level, significantly improving the degree of audio-visual compatibility and accuracy, avoiding mechanical and unnatural visual effects, enhancing the simulation and human-like quality of the virtual character, and optimizing the rendering effect of the virtual character.

[0087] The embodiments of this application do not limit the devices and timing for controlling a virtual character to broadcast the sound and perform motions that correspond to the sound. In some embodiments, a server controls the virtual character to broadcast the sound and perform motions that correspond to the sound based on the motion sequence, and in some other embodiments, the server transmits the generated motion sequence to a related terminal, and the terminal controls the virtual character to broadcast the sound and perform motions that correspond to the sound based on the motion sequence. Alternatively, the server may control the virtual character to broadcast the sound and perform motions that correspond to the sound immediately after generating the motion sequence, or it may first store the motion sequence in association with sound or text, and then, upon receiving a broadcast command, control the virtual character to broadcast the sound and perform motions that correspond to the sound based on the motion sequence.

[0088] All of the above preferred technical means can be combined in any way to form preferred embodiments of the present disclosure, which will not be described individually here.

[0089] The method according to the embodiment of the present invention uses voice and text as dual-mode drive signals, extracts semantic tags at the semantic level based on the text, making it easier to search for motion categories that match the semantic tags from a predetermined motion library. These motion categories can be highly aligned with the semantic information of the voice, reflecting the emotional tendencies and potential semantics in the broadcast voice of the virtual character. Furthermore, motion data belonging to the motion category is searched, and based on the motion data, a motion sequence with higher accuracy is quickly and efficiently synthesized for the virtual character, improving not only the efficiency of motion generation for the virtual character but also the accuracy of motion generation.

[0090] Furthermore, motion sequences can control virtual characters to perform body movements that synchronize with speech at a semantic level. This goes beyond simply following the rhythm of the speech; it significantly improves the degree of matching and accuracy between speech and video, avoids creating mechanical and unnatural visual effects, enhances the simulation and human-like qualities of the virtual character, and optimizes the rendering effect of the virtual character.

[0091] In the above embodiment, the flow of the motion generation means for a virtual character is briefly described, providing a body motion generation framework triggered by sound and text. When a virtual character broadcasts text, it emits sound and performs body motion, so there is a potential mapping relationship between the three: sound, text, and body motion, and they can be aligned on the time axis of the sound. In the embodiment of this application, such a mapping relationship is mined, and after obtaining the sound and its text, a motion category that matches the sound at the semantic level is searched from a predetermined motion library using the semantic tags of the text, and then a motion sequence for the virtual character is synthesized based on the motion data belonging to that motion category. The above motion generation means can be applied to body motion generation scenes for any virtual character such as game characters, virtual casters, video characters, and anime characters.

[0092] In this embodiment, specific examples of each step in the motion generation means for a virtual character will be described in detail. Figure 3 is a flowchart of the motion generation method for a virtual character according to an embodiment of the present invention. As shown in Figure 3, the embodiment is executed by a computer device, and the computer device is described as a server as an example, and the server may be server 102 of the above implementation environment, and the embodiment includes the following steps 301 to 309.

[0093] In step 301, the server retrieves the voice and text of the virtual character, the text indicating the semantic information of the voice.

[0094] The audio includes at least one audio frame, and the text is text that represents the semantic information of the audio, and the text includes at least one word, each word containing at least one character. The audio and text are related, that is, the text is semantic information obtained by performing ASR recognition on the audio, or the audio is an audio signal emitted when broadcasting the text, and the audio signal may be a synthesized signal output by a machine or a human voice signal collected by a microphone, and the type of audio signal is not specifically limited here.

[0095] In some embodiments, the server searches a local database for a pair of related audio and text; or the server retrieves a single segment of audio from the local database, performs ASR recognition on the audio to obtain text that represents the semantic information of the audio; or the server retrieves a single segment of text from the local database, performs speech synthesis on the text to obtain audio that incorporates the audio into the text.

[0096] In some other embodiments, the server downloads a related pair of audio and text from a cloud database, or the server downloads a single segment of audio from the cloud database, performs ASR recognition on the audio to obtain text indicating the semantic information of the audio, or the server downloads a single segment of text from the cloud database, performs speech synthesis on the text to obtain audio to incorporate the audio into the text.

[0097] In some embodiments, the server receives a related pair of audio and text uploaded from a terminal, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request and obtains the audio and text. Alternatively, the server receives audio uploaded from a terminal, performs ASR recognition on the audio and obtains text indicating the semantic information of the audio, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request, obtains the audio, performs ASR recognition on the audio and obtains text indicating the semantic information of the audio. Alternatively, the server receives text uploaded from a terminal, performs speech synthesis on the text and obtains audio that adds narration to the text, for example, the terminal sends a motion generation request to the server, the server receives and analyzes the motion generation request, obtains the text, performs speech synthesis on the text and obtains audio that incorporates the audio into the text.

[0098] In the above process, since text and audio can be converted to each other, the user may provide only audio, only text, or both audio and text simultaneously. In addition to what the user specifies, the audio may be read from a local database or downloaded from a cloud database. The embodiments of the present application do not specifically limit the sources of the audio and text.

[0099] In one exemplary scene, as shown in Figure 4, Figure 4 is a principle diagram of a motion generation method for a virtual character according to an embodiment of the present invention, where the user inputs voice and text on the terminal, the terminal uploads the input voice and text to the server, the server obtains voice 41 and text 42 "This is my first live stream!", voice 41 is an audio file in which the virtual character broadcasts text 42, and voice 41 may be an audio file in any format such as a WAV file, MP3 file, or MP4 file.

[0100] The server, after acquiring audio and text, generates a motion sequence by performing the method according to the embodiment of the present invention, matches the motion sequence with the semantic information of the text, and then controls the virtual character to perform the body motions shown in the motion sequence in a process to broadcast the audio, thereby matching the semantic information of the body motions performed by the virtual character with the broadcast audio.

[0101] In step 302, the server determines the sentiment tag of the text based on the text.

[0102] The emotion tag represents emotional information expressed by the text, such as joy, disappointment, or anger, and the embodiments of the present application do not specifically limit the content of the emotion tag.

[0103] In some embodiments, multiple candidate sentiment tags are pre-stored on the server, multiple sentiment keywords are assigned to each candidate sentiment tag, the mapping relationship between sentiment keywords and sentiment tags is stored, and a sentiment analysis method based on keyword matching is provided. If any of the sentiment keywords are included in the text, the sentiment tag to which the sentiment keyword is mapped is searched based on the mapping relationship, and the searched sentiment tag can be made one of the sentiment tags for the text. Of course, if the text contains multiple sentiment keywords, all of the sentiment tags to which each sentiment keyword is mapped will be made into sentiment tags for the text. Note that if multiple sentiment keywords are mapped to the same sentiment tag, it is necessary to remove duplicate sentiment tags for the text.

[0104] The sentiment analysis method based on keyword matching described above is computationally intensive, has low computational complexity, performs sentiment analysis quickly, and is highly efficient.

[0105] In some other embodiments, the server pre-stores multiple candidate sentiment tags, assigns one sentiment feature to each candidate sentiment tag, then extracts text features from the entire text, calculates the feature similarity between the text features and the sentiment features of each candidate sentiment tag, and sets the sentiment tag with the highest feature similarity as the sentiment tag for the text.

[0106] Furthermore, considering that sometimes text needs to be broadcast plainly and directly without including emotional tendencies, engineers may pre-set a feature similarity threshold. If the feature similarity of all candidate sentiment tags is less than this threshold, the sentiment tag with the highest feature similarity will not be selected. In this case, the sentiment tag will be empty, or the default sentiment tag "no sentiment" will be used as the sentiment tag for the text. In this way, the accuracy of sentiment tag recognition can be improved, and it can be ensured that inappropriate sentiment tags are not added to sentiment-less text.

[0107] Furthermore, considering the complexity of emotional composition when broadcasting text, multiple emotions may coexist. Therefore, if a feature similarity threshold is set in advance, emotion tags with a feature similarity greater than the feature similarity threshold may be used as the emotion tags for the text. In this way, the accuracy of emotion tag recognition can be further improved, resulting in better expressive capabilities for texts where multiple emotions intersect.

[0108] The number of sentiment tags determined by the sentiment analysis method based on the above feature similarity may be zero, one, or one or more; the number of sentiment tags is not specifically limited here. The sentiment tendency of the entire text is evaluated by the similarity of the feature space, and in this way, the accuracy of sentiment analysis is higher compared to keyword matching methods. Some texts themselves may not contain any sentiment keywords, but they express a clear sentiment tendency at the semantic level of the entire text, and such situations can be detected by comparing feature similarity.

[0109] In some other embodiments, a sentiment analysis model is pre-trained on a server, the text is input to the sentiment analysis model, the model calculates the matching probability between the text and each candidate sentiment tag, and then the sentiment analysis model outputs one or more sentiment tags that match the text based on the matching probability between the text and each candidate sentiment tag. In such cases, it may be necessary to add one "no sentiment" sentiment tag to the candidate sentiment tags to cover the recognition accuracy in the case of no sentiment. Similarly, the engineer may pre-set a probability threshold, so that the single sentiment tag with the highest matching probability can be selected and output. The probability threshold is a number between 0 and 1, or greater than or equal to 0. Alternatively, all sentiment tags whose matching probability is greater than the probability threshold may be output, or the top N (N≧1) sentiment tags sorted in descending order of matching probability may be output, and the embodiments of this application are not specifically limited thereto. Preferably, the sentiment analysis model may be a classification model, decision tree, deep neural network, convolutional neural network, multilayer perceptron, etc., and the embodiments of this application are not specifically limited thereto.

[0110] The sentiment analysis method based on the above sentiment analysis model can improve the accuracy of sentiment analysis by learning the potential mapping relationship between text and sentiment tags using machine learning methods, thereby evaluating the matching probability between text and each candidate sentiment tag, and the embodiment of this application does not specifically limit the sentiment analysis method.

[0111] Step 302 is an optional step, and if sentiment tags are not considered for semantic tags, it is not necessary to perform sentiment analysis on the text. The embodiments of this application do not specifically limit whether or not text sentiment analysis is necessary.

[0112] In one example scene, as still shown in Figure 4, performing sentiment analysis on text 42 "This is my first live stream!" using any of the sentiment analysis methods described above and obtaining the sentiment tag "joy" for text 42 indicates that the virtual character needs to be in a happy mood when broadcasting text 42.

[0113] In step 303, the server determines, based on the text, at least one word contained in the text.

[0114] In some embodiments, the server performs word segmentation on the text to obtain a word list of the text, the word list records at least one word contained in the text, and each word contains at least one character.

[0115] The word segmentation process can be implemented using a word segmentation tool, and different word segmentation tools can be used depending on the language of the text. For example, a Chinese word segmentation tool can be used to segment a Chinese text to obtain a word list of the Chinese text, and an English word segmentation tool can be used to segment an English text to obtain a word list of the English text. The embodiments of this application do not specifically limit the language of the text or the type of word segmentation tool.

[0116] In one example scene, as still shown in Figure 4, word segmentation is performed on the text 42 "This is my first live stream!" to obtain the word list {"I", "first", "live stream!"}. Text 42 contains three words: the first word "I" contains one character, the second word "first" contains four characters, and the third word "live stream!" contains eight characters.

[0117] In step 304, the server searches the parts of speech table for the part of speech tag to which each word belongs.

[0118] The part-of-speech tag represents the part-of-speech information of a word in the text, such as subject, verb, state, etc., and the embodiments of this application do not specifically limit the content of the part-of-speech tag.

[0119] In some embodiments, the server pre-stores a part-of-speech table containing candidate part-of-speech tags. Next, for each word obtained by word-segmenting the text, the server searches the part-of-speech table, calculates the vector similarity between the word vector of the word and the tag vector of each part-of-speech tag, and selects the part-of-speech tag with the highest vector similarity to be the part-of-speech tag to which the word belongs.

[0120] In one example scene, as still shown in Figure 4, word segmentation is performed on the text 42 "This is my first live stream!" to obtain the word list {"I", "first", "live stream!"}. Next, in the parts of speech table, it is searched that the part of speech tag to which the first word "I" belongs is "subject", the part of speech tag to which the second word "first" belongs is "state", and the part of speech tag to which the third word "live stream!" belongs is "verb".

[0121] In steps 303-304 above, one possible embodiment is provided for extracting part-of-speech tags for each word in the text. This method of searching a part-of-speech table is computationally intensive, has low computational complexity, is fast, and is highly efficient. Of course, one could also train a part-of-speech analysis model, input the text into the model, and have the model output a series of words and the part-of-speech tags to which they belong. In this way, the accuracy of part-of-speech analysis is high, and the embodiments of this application do not specifically limit the part-of-speech analysis method.

[0122] In the part-of-speech analysis process described above, since we typically use different body movements for words of different parts of speech when speaking, the difference in parts of speech influences the type or range of body movements. By considering the part-of-speech tag of each word, we can better reflect the hidden information at the semantic level of the text.

[0123] Step 304 is an optional step, and if part-of-speech tags are not considered for semantic tags, it is not necessary to perform part-of-speech analysis on the text (however, word segmentation is necessary, and only after word segmentation is alignment of words, phonemes, and motions useful). The embodiments of this application do not specifically limit whether or not text part-of-speech analysis is necessary.

[0124] In step 305, the server determines the sentiment tag and the part-of-speech tag to which the at least one word belongs as the semantic tag of the text.

[0125] The word meaning tag represents either the part-of-speech information of the word in the text or the sentiment information expressed by the text.

[0126] In some embodiments, the sentiment tags obtained in step 302 and the part-of-speech tags obtained in step 304 are determined as semantic tags of the text, and the number of semantic tags may be one or more, and the embodiments of the present application do not specifically limit the number of semantic tags.

[0127] In one example scene, as still shown in Figure 4, word segmentation is performed on the text 42 "This is my first live stream!" to obtain the word list {"I", "first", "live stream!"}. In the parts of speech table, it is found that the part of speech tag to which the first word "I" belongs is "subject", the part of speech tag to which the second word "first" belongs is "state", and the part of speech tag to which the third word "live stream!" belongs is "verb". Furthermore, sentiment analysis is performed on the text 42 to obtain the sentiment tag "joy", to which the text 42 belongs, and finally, the four semantic tags "subject", "state", "verb", and "joy" are output. The process of analyzing the text 42 and extracting semantic tags described above is called the "speech-to-text analysis" process.

[0128] Steps 302-305 present one possible embodiment in which a server determines the semantic tags of a given text, using the example that semantic tags consider both part-of-speech tags and sentiment tags. By analyzing a given text, feature information at the semantic level of the text can be extracted, and this feature information can be represented in a simple form such as semantic tags. In the motion generation process, semantic tags at the semantic level are useful as guide signals, and furthermore, they are advantageous for synthesizing natural virtual character body motions that closely match the semantics.

[0129] Furthermore, the word definition tag may include at least one part-of-speech tag or sentiment tag. If part-of-speech tags are not considered for the word definition tag, step 304 does not need to be performed. If sentiment tags are not considered for the word definition tag, step 302 does not need to be performed. The embodiments of this application do not specifically limit the content of the word definition tag.

[0130] In step 306, for each word in the text, the server determines the speech segment to which the phoneme belongs from the speech, based on the phoneme associated with the word.

[0131] In some embodiments, for each word contained in the text in step 303, a phoneme associated with the word can be determined, where the phoneme associated with the word refers to the phoneme that needs to be pronounced in order to broadcast the word. There may be one or more phonemes associated with each word, and embodiments of the present application do not specifically limit the number of phonemes. Next, at least one audio frame corresponding to the phoneme is found in the audio, and this at least one audio frame constitutes the audio segment to which the phoneme belongs. In this way, by finding one audio segment from the audio for each word using the phoneme alignment method, the word can be aligned to an audio segment on the audio time axis.

[0132] In one exemplary scene, as still shown in Figure 4, after obtaining each word in text 42 by word segmentation in step 303, phoneme alignment can be performed, that is, N (N≧1) phonemes that broadcast the word are determined, at least one audio frame (e.g., frame 2 to frame 37) that emits these N phonemes is found from speech 41, and this at least one audio frame is designated as the audio segment for the word alignment. The above process can be considered as a process of determining one aligned audio segment from speech for each word in text 42.

[0133] Furthermore, step 306 can be executed after the word segmentation in step 303 is completed, and can be executed in parallel with the extraction of emotion tags in step 302 and the extraction of part-of-speech tags in step 304, or in order. The embodiment of this application does not limit the execution order of steps 302, 304, and 306.

[0134] In step 307, the server searches a predetermined motion library for motion categories that match the semantic tag of the word and motion data belonging to those motion categories, based on the semantic tag to which the word belongs.

[0135] The predetermined motion library includes motion data belonging to multiple motion categories for the virtual character.

[0136] In some embodiments, each semantic tag obtained in step 305 is associated with a single word in the text, and regarding the part-of-speech tags in semantic tags, the part-of-speech tags themselves are obtained by searching the part-of-speech table on a word-by-word basis, so there is a natural relationship between the part-of-speech tags and words, and each word must belong to one part-of-speech tag, but different words may have the same part-of-speech tag, and regarding the sentiment tags in semantic tags, sentiment analysis is performed by analyzing the text of the entire paragraph, so in this way, the sentiment tendency can be better judged by comprehensively considering the context of the entire text, but regarding the sentiment tags In contrast, it is necessary to find the word that best matches the text. For example, if sentiment tags are determined using a keyword matching sentiment analysis method, the matched sentiment keyword (which is always a single word in the text) is directly considered the word that best matches the sentiment tag. If sentiment tags are determined using a feature similarity or sentiment analysis model-based sentiment analysis method, and the sentiment tag and the words obtained from each word segment are known, the vector similarity between the word vector of the sentiment tag and the word vector of each word is calculated inversely, and the word with the highest vector similarity is considered the word that best matches the sentiment tag.

[0137] With the above method, regardless of whether the part-of-speech tag or emotion tag is included in the definition tag, the single word that best matches can be found for each definition tag. Furthermore, the same word may have one or more definition tags. For example, as still shown in Figure 4, in the text 42 "This is my first live stream!", the word "live stream!" has two definition tags: one is the part-of-speech tag "verb" and the other is the emotion tag "joy". The embodiments of this application do not specifically limit the number and type of definition tags each word may have.

[0138] In this way, after determining one or more semantic tags to which each word in the text belongs, the motion data belonging to that motion category can be retrieved by using each semantic tag as an index to search for and retrieve motion categories that match the semantic tags from multiple candidate categories in a predetermined motion library.

[0139] The following describes one possible implementation of searching for motion categories based on semantic tags, using steps A1 to A4 as an example. In this implementation, we evaluate whether the semantic tags are similar to the candidate categories from the feature space.

[0140] In step A1, the server extracts the semantic features of each word tag.

[0141] In some embodiments, for each semantic tag to which each word in the text belongs, the server extracts the semantic features of the semantic tag, for example, by directly using the word vector of the semantic tag as the semantic feature, or by pre-training a feature extraction model, inputting the semantic tag into the feature extraction model, processing the semantic tag with the feature extraction model, and outputting the semantic features of the semantic tag. The feature extraction model may be any NLP model. Furthermore, to improve feature extraction efficiency, the semantic features of all candidate part-of-speech tags and all sentiment tags can be pre-extracted, and each part-of-speech tag or sentiment tag can be associated with its own semantic features and stored. In this way, for each semantic tag, the stored semantic features associated with the tag ID (Identification) can be directly and quickly retrieved based on the tag ID of the semantic tag. This is equivalent to calculating the semantic features of each semantic tag offline, requiring only a small search overhead in the online motion generation stage, eliminating the need to calculate semantic features in real time, and improving feature extraction efficiency.

[0142] In some embodiments, a Key-Value data structure is used to store a tag ID and its semantic features, where the tag ID is the Key (key name) and the semantic feature is the Value (key value). During the online search phase, the tag ID is used as an index to search whether a match can be found in any of the Key-Value data structures. If a match can be found in a Key-Value data structure, the semantic feature stored in the Value is retrieved, and this semantic feature is the semantic feature of the tag indicated by the tag ID.

[0143] In step A2, the server searches for category features of multiple candidate categories in a given motion library.

[0144] In some embodiments, a predetermined motion library is created and maintained on a server, and this predetermined motion library contains motion data belonging to multiple motion categories of the virtual character. The process for constructing the motion library is described in detail in the following embodiments and is omitted here. The predetermined motion library stores a large amount of motion data, and in order to facilitate searching, this motion data is divided into multiple motion categories by clustering according to the semantic level, and each motion category has one motion set that stores the motion data clustered to the corresponding motion category. In some embodiments, the motion data may be executed as 3D skeletal data of multiple frames at consecutive time points in the virtual character when executing the motion of this motion category.

[0145] Furthermore, all motion categories in a given motion library constitute multiple candidate categories for the current semantic tag. In this case, the server can compute category features for each candidate category. For example, the word vector of the candidate category can be used as the category feature of the candidate category. Alternatively, the feature extraction model used in step A1 can be multiplexed, the candidate categories can be input to the feature extraction model, the feature extraction model can process the candidate categories, and the category features of the candidate categories can be output. Here, we will only describe the multiplexing of the feature extraction model in step A1 as an example. This saves server-side training overhead, eliminates the need to retrain one feature extraction model, and allows semantic tags and motion categories to be projected into the same feature space. Of course, the server can also train one semantic feature extraction model for semantic tags and one category feature extraction model for motion categories. This makes the extraction process for semantic and category features more accurate and improves the expressive power of semantic and category features, respectively. The embodiments of this application are not specifically limited to this.

[0146] Furthermore, to improve feature extraction efficiency, a pre-trained feature extraction model can be used to extract category features for all motion categories (i.e., all candidate categories) in a given motion library. Then, each motion category can be associated with its own category features and stored. In this way, during the online motion generation stage, for each candidate category, the category features stored in association with the category ID can be directly and quickly retrieved based on the category ID of the candidate category. This is equivalent to calculating the category features of each candidate category offline, thus requiring less search overhead during online searches, eliminating the need to calculate category features in real time, and improving feature extraction efficiency.

[0147] In some embodiments, a key-value data structure is used to store a category ID and its category features, where the category ID is the key and the category features are the values. During the online search phase, the category ID is used as an index to search whether any key-value data structures can be matched. If a key-value data structure can be matched, the category features stored in the Value are retrieved, and these category features are those of the candidate category indicated by the category ID.

[0148] In the embodiment of this application, in order to distinguish between motion categories that match semantic tags and motion categories used as candidates, the terms "candidate category" and "motion category" are distinguished. That is, while "candidate category" and "motion category" refer to semantic tags, for the given motion library itself, all categories are motion categories supported by the given motion library, and there is no concept of a "candidate category."

[0149] In step A3, the server determines the motion category from among several candidate categories, and the category features and semantic features of the motion category satisfy the similarity condition.

[0150] The similarity condition indicates whether the word tag is similar to the candidate category.

[0151] In some embodiments, for each semantic tag to which each word in the text belongs, the server obtains the semantic features of the semantic tag from step A1, and obtains the category features of all candidate categories in a given motion library from step A2. Next, it calculates the feature similarity between the semantic feature and the category features of each candidate category, and selects from the multiple candidate categories whose feature similarity satisfies the similarity condition as the motion category to match the semantic tag, that is, the category features and semantic features of the determined motion category satisfy the similarity condition. The feature similarity may be cosine similarity, the reciprocal of the Euclidean distance, etc., and the embodiments of this application are not specifically limited thereto.

[0152] In some embodiments, the similarity condition is the highest feature similarity. The candidate category with the highest feature similarity from all candidate categories is searched for as the motion category that matches the word-meaning tag. In this way, it is possible to guarantee that each word-meaning tag will always have at least one motion category that is most similar at the word-meaning level. This ensures that some word-meaning tags will not fail to match a motion category, and the motion category screening process is simple and computationally efficient.

[0153] In some other embodiments, the similarity condition is that the feature similarity is greater than a predetermined similarity threshold, which is a number greater than 0 that is predefined by the engineer. If there is only one candidate category that satisfies the similarity condition, that single candidate category is selected as the motion category that matches the word-meaning tag. If there are more than one candidate category that satisfies the similarity condition, the candidate category with the highest feature similarity is selected as the motion category that matches the word-meaning tag. If there are zero candidate categories that satisfy the similarity condition, i.e., if none of the candidate categories satisfy the similarity condition, the process proceeds to step A4. In this way, by setting a predetermined similarity threshold, it is possible to take into account situations where the mood during a broadcast is calm and does not contain any specific clear meaning. In this case, the virtual character does not need to perform a physical motion that has a certain meaning (doing so may be exaggerated). In this case, the overall value of each feature similarity is low, and if a predetermined similarity threshold is not set, the feature similarity with the highest relative value can be directly selected. If a predetermined similarity threshold is set, a policy is provided in which all candidate categories do not match. In this case, the process proceeds to step A4, and the motion category that matches the meaning tag is directly set to a predetermined motion category that does not have a special meaning, such as a standing motion category or a sitting motion category.

[0154] In steps A1 to A3 described above, an embodiment is provided in which a semantic tag is evaluated from the feature space to determine whether it is similar to a candidate category. In this way, for any semantic tag, a motion category that satisfies the similarity condition with the semantic tag is found in the feature space, and by controlling the similarity condition, it is possible to flexibly control whether or not to pad a semantic tag when it is not very similar to all candidate categories using a predetermined motion category. This improves the recognition efficiency of motion categories and the controllability of motion categories.

[0155] In step A4, if none of the category features or semantic features of the multiple candidate categories satisfy the similarity condition, the server sets the motion category that matches the semantic tag as the predetermined motion category.

[0156] In some embodiments, the similarity condition does not necessarily mean that a matching motion category can be found for every semantic tag. If neither the semantic features of the semantic tag nor the category features of any of the candidate categories satisfy the similarity condition, it means that the semantic tag will not match any of the candidate categories. To avoid a gap of a certain amount of time in the motion sequence, one predetermined motion category can be designated as the motion category that matches the semantic tag. The predetermined motion category may be a default motion category set in advance by the engineer, for example, a standing motion category without a semantic, or a sitting motion category. Here, the predetermined motion category is not specifically limited, and the engineer may set different predetermined motion categories for different virtual characters.

[0157] Steps A1 to A4 above provide one possible embodiment in which a motion category is selected for each semantic tag, using the semantic tag as the unit. Using the text's semantic tags as an index, the motion category that best matches the speech and semantic level can be found from a given motion library. This motion category not only moves according to the rhythm of the speech, but can also be highly aligned with the semantic information of the speech, reflecting the emotional tendencies and potential semantics of the virtual character during voice broadcast. In this way, the motion data selected from the motion category can be synthesized for the virtual character with a higher degree of accuracy.

[0158] In some other embodiments, in addition to steps A1 to A4, a motion classification model may be trained, each semantic tag may be input to the motion classification model, the motion classification model may predict the matching probability between the semantic tag and each candidate category, and the motion category with the highest matching probability may be output. In this way, by simply adding the above-mentioned predetermined motion categories to the candidate categories, it is possible to cover scenes where it is not necessary to perform body motions that include semantic meaning during broadcasting, and the accuracy of motion category recognition can be further improved.

[0159] Each semantic tag is associated with a single word, but each word may have multiple semantic tags. Therefore, to establish a one-to-one correspondence between words and motion categories, if a word has multiple semantic tags, there may be multiple matching motion categories. In this case, after finding motion categories that match each semantic tag, the motion category that matches all of the word's semantic tags simultaneously is preferentially selected and set as the final selected motion category for the word. If no motion category matches all of the word's semantic tags simultaneously, the motion category with the highest feature similarity is preferentially selected, or the predetermined motion category is set directly. For example, if a word has two semantic tags a and b, and semantic tag a matches motion categories 1 and 2, and semantic tag b matches motion categories 1 and 3, motion category 1 is directly selected as the final motion category for this word. However, if semantic tag b matches motion categories 3 and 4, the motion category with the highest feature similarity from motion categories 1 to 4 is selected, or the predetermined motion category is set as the final motion category for this word.

[0160] In one example scene, as still shown in Figure 4, each semantic tag obtained in the speech-to-text analysis stage is used as an index, and a motion category matching that semantic tag is selected from K (K≧2) motion categories in a predetermined motion library 43. For example, for the three words "I", "first time", and "live stream!" in text 42, the first word "I" has only one semantic tag "subject", but since the semantic tag "subject" has not found a matching motion category in the predetermined motion library 43, it is matched to the predetermined motion category "stand up", and the second word "first The word "teno" has only one semantic tag, "state," but the semantic tag "state" finds a matching motion category, "shrugging shoulders while acting cute," from the given motion library 43. The third word, "It's a live stream!" has two semantic tags, "verb" (part of speech tag) and "joy" (emotion tag). The semantic tags "verb" and "joy" jointly lock onto one motion category, "raising hands with joy." That is, the motion category "raising hands with joy" matches both semantic tags "verb" and "joy" simultaneously, and is therefore selected as the motion category that best matches the third word, "It's a live stream!".

[0161] The specified motion library 43 is also called a dynamic definition specified motion library containing a large amount of motion data, and this large amount of motion data may be data of 3D motion segments of virtual characters that have been collected, published, and comply with the law, for example, each 3D motion segment contains 3D skeletal data of multiple frames at consecutive times. The process of searching for motion categories based on the above definition tags is also called key pose search (key motion search) for each definition tag.

[0162] Furthermore, because the data level of the given motion library 43 may be large, each motion category can be further divided into multiple subcategories. For example, the motion category "raise hand" can be further divided into multiple subcategories such as "raise one hand" and "raise both hands." In one example, as shown in Figure 4, motion category 1 includes 10 subcategories, motion category 2 includes 3 subcategories, motion category 3 includes 6 subcategories, and so on. By analogy, motion category K includes 2 subcategories. Here, we do not specifically limit whether or not each motion category is divided into subcategories.

[0163] Furthermore, if each motion category includes subcategories for each semantic tag, feature similarity can be calculated using the same method as in steps A1 to A3. This allows for the identification of a subcategory that matches the semantic tag from all subcategories of the determined motion category, thereby further improving the degree of semantic-level matching between the motion data and semantic tags used in step 308.

[0164] In step 308, the server generates a motion segment that matches the audio segment based on the motion data corresponding to the word.

[0165] In step 307 above, a unique motion category can be found for each word, and can be divided into the following two cases: 1) If the word has one semantic tag and that semantic tag has a motion category that satisfies the similarity conditions, this motion category is selected; if the semantic tag does not have a motion category that satisfies the similarity conditions, a predetermined motion category is selected; 2) If the word has multiple semantic tags and each semantic tag has selected a motion category (including a predetermined motion category) according to step 1), if a motion category matches all of the word's semantic tags simultaneously, that simultaneously matching motion category is selected; if there are multiple simultaneously matching motion categories, the motion category that matches simultaneously and has the highest feature similarity is selected; if no motion category matches all of the word's semantic tags simultaneously, the motion category with the most matching semantic tags is selected, or the motion category with the highest feature similarity is selected, or a predetermined motion category is selected. The embodiments of this application are not specifically limited thereto.

[0166] Based on the above, each word has a one-to-one corresponding motion category (including a predetermined motion category), and based on the correspondence of the audio time axis, each word can find one audio segment in step 306, one motion category in step 307, and one motion segment can be synthesized for the word based on the motion data belonging to that motion category in a predetermined motion library, thereby ensuring the alignment of the timestamps of the motion segment and the audio segment, and ensuring a high degree of semantic alignment.

[0167] The following describes the motion segment synthesis method possible by steps B1 and B2. In this synthesis method, a one-to-one correspondence between audio frames and key motion frames can be achieved, and the timestamps of both are aligned.

[0168] In step B1, the server determines from the motion data at least one key motion frame that has the highest degree of semantic matching with the word.

[0169] In some embodiments, since there is a one-to-one correspondence between words and motion categories, for each word, the server searches a predetermined motion library for motion data belonging to the motion category corresponding to that word, and further selects motion data to obtain at least one key motion frame with the highest degree of semantic matching with that word.

[0170] In some embodiments, each motion category in a given motion library stores one motion set, the motion set stores motion data belonging to the motion category, for example, the motion set includes multiple motion segments, each motion segment includes multiple motion frames, each motion frame shows the pose of each skeletal keypoint at a certain point in the process of a virtual character performing a certain motion in the motion category, and each motion segment has its annotated reference audio and reference text. During the library construction phase, timestamp alignment is performed on three elements: words in the reference text, phonemes in the reference audio, and motion frames in the motion segments. Therefore, when comparing the degree of semantic matching between a word and a key motion frame, the first step is to search the motion set for any reference text in the motion segment containing the word. If a reference text is found, at least one key motion frame that matches (i.e., timestamp alignment) with the word is directly extracted from the motion segment corresponding to the found reference text. If no reference text is found, the vector similarity between the word vector of the current word and the word vector of each word in each reference text is further calculated. The reference text to which the closest approximant word (usually a synonym and / or related word) with the highest vector similarity belongs is found, and at least one key motion frame that matches (i.e., timestamp alignment) with the found reference text is extracted from the motion segment corresponding to the found reference text.

[0171] In the process described above, one possible embodiment is provided for selecting key motion frames from the motion data of a motion set. Thus, the method of first detecting duplicate words and then detecting similar words ensures that similar words only need to be detected if no duplicate words are found, thereby reducing the computational overhead of the server. In some other embodiments, when detecting similar words, instead of evaluating similar words based on vector similarity, synonyms and / or related words can be directly obtained from a word table for the word in question. Furthermore, synonyms and / or related words can be used as an index to search whether they hit a certain reference text. Thus, synonyms and / or related words only need to be searched if no duplicate words are found, similarly achieving the effect of saving computational overhead.

[0172] In some other embodiments, we consider one more scenario in which multiple subcategories are further subdivided under each motion category in a given motion library. In the preferred method of step 307, a subcategory matching the word can be found from among the multiple subcategories of the motion category. Thus, in the step of searching for key motion frames, only motion data belonging to the selected subcategory needs to be considered, and motion data belonging to the unselected subcategory does not need to be considered. This corresponds to narrowing the search range for key motion frames, further improving the search efficiency of key motion frames. Typically, key motion frames found within a narrow range will match both the motion category (i.e., the major category) and the subcategory to which they belong (i.e., the minor category), thus improving the search accuracy of key motion frames and enabling better coordination and linking at the word and semantic level.

[0173] In some other embodiments, if only one standard motion segment is saved for each motion category from a given motion library, then at least one key motion frame closest to the central median motion frame can be sampled from the central median motion frame of the motion segment. In this way, the key motion frame can be aligned to the center of the pre-stored standard motion segment, and typically, the standard and important motion / pose is located in the center.

[0174] In step B2, the server synthesizes at least one key motion frame based on the audio segment into a motion segment that matches the audio segment.

[0175] In some embodiments, for each word in the text, the server can search for at least one key motion frame based on step B1, determine the number of frames in the key motion frame, and in step 306 determine the number of audio frames in the audio segment aligned with the timestamp of the word. Furthermore, it can compare the number of audio frames with the number of frames in the key motion frame and, preferably, scale the key motion frame by a certain percentage of double speed based on the number of audio frames, ensuring that the length of the final synthesized motion segment and the audio segment in step 306 are the same (i.e., timestamp alignment). In this case, there is no need to trim or modify the key motion frame, and only the playback speed needs to be adjusted, so that many details of the key motion frame are retained overall, and the complete pose changes of the key motion matching the word can be displayed as much as possible.

[0176] In some other embodiments, when the difference between the number of audio frames and the number of key motion frames is large, simply adjusting the playback speed may result in an abrupt transition, for example, the motion of a virtual character may suddenly slow down or suddenly speed up, thus affecting the fluency and naturalness of the motion. Therefore, embodiments of the present application further provide a method for interpolating or trimming key motion frames to improve the above situation and optimize the fluency and naturalness of the motion. Below, we will consider two cases: Case 1, where the number of key motion frames does not exceed the number of audio frames in the audio segment, and Case 2, where the number of key motion frames exceeds the number of audio frames in the audio segment.

[0177] In Case 1, the number of key motion frames does not exceed the number of audio frames in the audio segment.

[0178] In some embodiments, if the number of frames in the key motion frame does not exceed the number of audio frames in the audio segment, the server interpolates the at least one key motion frame to obtain a motion segment of the same length as the audio segment.

[0179] In some embodiments, interpolation can be performed on at least one key motion frame to ensure that the lengths of the motion segment and the audio segment are equal, for example, by inserting one or more intermediate motion frames between any pair or more pairs of adjacent key motion frames, each intermediate motion frame being intermediate motion data calculated based on the pair of adjacent key motion frames to be inserted.

[0180] In some embodiments, when calculating intermediate motion data using a linear interpolation method, if the calculation is actually performed using the linear interpolation method, for example, when inserting i (i≧1) intermediate motion frames between key motion frame 1 and key motion frame 2, taking the same skeletal keypoint of the left shoulder as an example, if the skeletal keypoint is in pose θ1 in key motion frame 1 and in pose θ2 in key motion frame 2, then by calculating the i intermediate poses obtained by converting the skeletal keypoint from pose θ1 to pose θ2, the i intermediate poses of the skeletal keypoint in the i intermediate motion frames can be obtained. By analogy, by calculating i intermediate poses for all skeletal keypoints of the whole body, the insertion of i intermediate motion frames can be realized. In the linear interpolation method, the skeletal keypoint is considered to change uniformly according to a fixed step size in the i intermediate motion frames, and the skeletal keypoint is considered to move at a constant velocity. Therefore, by calculating the fixed step size when inserting i intermediate motion frames based on the poses in the initial and final states (i.e., pose θ1 and pose θ2), the calculation of i intermediate poses can be easily realized. The linear interpolation methods described above consume few computational resources, have low computational overhead, synthesize motion segments quickly, and have low latency.

[0181] In some other embodiments, a motion adjustment model is pre-trained, and if the number of key motion frames is less than the number of audio frames in the audio segment, the motion adjustment model performs nonlinear interpolation on the key motion frames, that is, the motion adjustment model learns a nonlinear interpolation mode for key motion frames, and such a nonlinear interpolation mode may be fitted according to a certain motion curve, and it may also learn a pause change rule for such amplitude changes by further fitting the motion amplitude according to the audio rhythm, specifically which nonlinear interpolation mode is learned is determined by the input training samples, and after the training of the motion adjustment model is complete, at least one key motion frame can be input to the motion adjustment model, and the number of audio frames can be controlled as a hyperparameter, and furthermore, the motion adjustment model outputs at least one intermediate motion frame awaiting insertion, and between each intermediate motion frame inserted into two adjacent key motion frames, the pause changes are performed at an uneven rate based on the nonlinear interpolation mode learned by the motion adjustment model, rather than changing uniformly according to a fixed step size, and in embodiments of the present application, there is no particular limitation on whether or not a linear interpolation method is used. The nonlinear interpolation method based on the above motion adjustment model can improve the mechanical feel achieved by the linear interpolation method and optimize the fluency of the motion segments.

[0182] In the process described above, if the number of key motion frames is less than the number of audio frames, the missing motion frames can be filled in by interpolating the key motion frames. Intermediate motion states are then filled in between adjacent key motion frames, resulting in the virtual character moving more continuously within the motion segment.

[0183] In Case 2, the number of key motion frames exceeds the number of audio frames in the audio segment.

[0184] In some embodiments, if the number of frames in the key motion frame exceeds the number of audio frames, a motion segment of the same length as the audio segment is created, and each frame of the motion segment is padded to a predetermined motion frame in a predetermined motion category. The predetermined motion category may be a default motion category set in advance by the engineer, for example, a meaningless standing motion category or a sitting motion category. Here, the predetermined motion category is not specifically limited, and the engineer may set different predetermined motion categories for different virtual characters. A predetermined motion frame is a predetermined relatively stationary motion frame in a predetermined motion category. For example, if the predetermined motion category is a standing motion category, the predetermined motion frame is a standing motion frame; if the predetermined motion category is a sitting motion category, the predetermined motion frame is a sitting motion frame. When a predetermined motion category is maintained, the virtual character usually maintains the same motion for multiple frames.

[0185] In the above process, if the number of key motion frames exceeds the number of audio frames, the key motion frames in this portion are discarded, and the motion segment is padded with predetermined motion frames. In this way, there is no need to play the key motion frames at high speed, which would impair the viewer's experience, and no problems are created in any motion segment.

[0186] In some other embodiments, if the number of frames in the key motion frame exceeds the number of audio frames, the key motion frame may be trimmed, for example, by discarding some of the key motion frames at the beginning and end so that the number of frames in the trimmed key motion frame does not exceed the number of audio frames. In this way, padding of the audio segment of a long word using predetermined motion frames is avoided, resulting in a better motion generation effect, but there is a possibility of compromising the integrity of the key motion frame, in which case it needs to be corrected by the motion smoothing operation in step 309. The trimming logic for the beginning and end key motion frames may be set by the technician, for example, by trimming according to a set number of frames or according to a set ratio, and the embodiments of the present application are not specifically limited thereto.

[0187] Steps B1 and B2 provide a possible motion segment synthesis method, which enables a one-to-one correspondence between audio frames and key motion frames, and aligns the timestamps of both. Even if the number of key motion frames and the number of audio frames do not match, smooth synthesis of motion segments can be ensured and the efficiency of motion segment synthesis can be improved by performing interpolation, trimming, or padding on predetermined motion frames.

[0188] In step 309, the server generates a motion sequence that matches the speech based on each motion segment that matches the speech segment of each word, and the motion sequence controls the virtual character to perform motions that match the speech.

[0189] In some embodiments, for each word in the text, it is possible to find a single corresponding audio segment in step 306 and synthesize a single corresponding motion segment in step 308. Thus, the audio segment in step 306 and the motion segment in step 308 achieve a one-to-one correspondence between the three, with the word acting as a bridge, and achieve timestamp alignment. In this way, a motion sequence can be obtained simply by sequentially splicing each motion segment according to the timestamp order of each audio segment, ensuring that each motion segment in the motion sequence is highly aligned at the semantic level with a single audio segment in the text.

[0190] In some other embodiments, motion smoothing can be performed on the spliced ​​motion sequence to increase the naturalness and fluency when different motion segments are joined together, which will be described in detail below in steps C1 to C2.

[0191] In step C1, the server splices each motion segment that matches each audio segment based on the timestamp order of each audio segment, and obtains a spliced ​​motion sequence.

[0192] In some embodiments, to achieve a one-to-one correspondence between audio segments and motion segments using words as bridges, for each motion segment, the timestamp interval of the corresponding audio segment is found on the audio time axis, and then each motion segment is spliced ​​according to the order of the timestamp intervals to obtain a spliced ​​motion sequence. Preferably, the spliced ​​motion sequence is output directly to simplify the motion synthesis process, or the motion smoothing operation in step C2 is performed to increase the naturalness and fluency when different motion segments are joined.

[0193] In step C2, the server performs motion smoothing on each motion frame in the splicing motion sequence to obtain the motion sequence.

[0194] In some embodiments, the splicing motion sequence obtained in step C1 may consist of some key motion frames, some interpolated intermediate motion frames, and some padded predetermined motion frames. Therefore, the motion data of each frame in the splicing motion sequence is referred to as a single motion frame, and the motion frame may be a key motion frame, an intermediate motion frame, or a predetermined motion frame, and the embodiments of this application do not specifically limit this. Next, motion smoothing is performed on each motion frame in the splicing motion sequence to obtain the final motion sequence.

[0195] In some embodiments, a window smoothing method is used to perform global processing on each connected motion frame to obtain a globally smoothed motion sequence. The window smoothing method determines the pose of each motion frame of the same skeletal keypoint, using the skeletal keypoint as the unit. In this way, a series of pose changes in the motion sequence of the skeletal keypoint can be obtained, thereby fitting a single pose change polyline. Furthermore, the pose change polyline is smoothed using a moving window average smoothing algorithm to obtain a single pose change curve. Then, the updated pose for each motion frame of the skeletal keypoint is obtained by sampling the pose in each motion frame of the skeletal keypoint from the pose change curve according to the timestamp. Thus, when there is a large difference in the motion categories matched to two adjacent motion segments, the window smoothing method can produce a motion sequence with a smoother, more continuous, and natural connection between the two adjacent motion segments, resulting in better visual effects and improving the accuracy of motion synthesis.

[0196] In some other embodiments, smoothing may be performed on the pose-changing polyline using a different smoothing algorithm in addition to the window smoothing method, or the same motion smoothing effect can be achieved by directly fitting the pose-changing curve to the pose-changing polyline using machine learning.

[0197] In steps C1-C2, motion smoothing is applied to the spliced ​​motion sequence formed by mechanical splicing. This results in a smoother, more continuous, and natural connection between two adjacent motion segments, producing a motion sequence with better visual effects and improving the accuracy of motion synthesis. Of course, it is also possible to output the spliced ​​motion sequence directly without motion smoothing, thus simplifying the motion synthesis process and improving motion synthesis efficiency.

[0198] In one example scene, as still shown in Figure 4, for the text 42 "This is my first live stream!", the first word "I" matches the predetermined motion category "standing up", the second word "first time" matches the motion category "shrugging shoulders in a cutesy manner", and the third word "This is a live stream!" matches the motion category "raising hands with joy". In this case, three motion segments, namely the "standing up" motion segment, the "shrugging shoulders in a cutesy manner" motion segment, and the "raising hands with joy" motion segment, are combined. For the method of combining the motion segments, please refer to step 308, and the explanation will be omitted here. Next, the three motion segments, "standing up", "shrugging shoulders in a cutesy manner", and "raising hands with joy", are spliced ​​to obtain a spliced ​​motion sequence, and motion smoothing is performed on the spliced ​​motion sequence to obtain the final output motion sequence. As shown in Figure 4, a smoothed pose change curve (i.e., motion curve) is further output, indicating that the pose change curve of the skeletal keypoints in the output motion sequence is smooth and fluid, thus eliminating the mechanical feel of the body motion. Preferably, the embodiments of this application are applied to the synthesis of the body motion of a virtual character, but it is necessary to combine the facial expressions of the virtual character to generate the final screen, and by combining the screen and sound, the final virtual character video (e.g., digital human video) can be generated.

[0199] Steps 306-309 provide one possible embodiment for generating a motion sequence for the virtual character based on motion data. If each motion category has a large amount of motion data, the most representative key motion frames with the highest degree of semantic matching can be selected, a series of motion segments can be synthesized, and these can be spliced ​​into a single motion sequence. This motion sequence represents the changes in the virtual character's body motion over consecutive time periods when the virtual character broadcasts sound, and controls the virtual character to perform body motions that match the sound when it broadcasts sound.

[0200] In the process described above, each motion frame in the final synthesized motion sequence is aligned with the timestamp of each audio frame in the audio. This ensures that the motion frames reflect body motions that match the audio frames at a semantic level, significantly improving the fit and accuracy of audio and video, avoiding mechanical and unnatural visual effects, enhancing the simulation and human-like quality of the virtual character, and optimizing the rendering effect of the virtual character.

[0201] All of the above preferred technical means can be combined in any way to form preferred embodiments of the present disclosure, which will not be described individually here.

[0202] The method according to the embodiment of the present invention uses voice and text as dual-mode drive signals, extracts semantic tags at the semantic level based on the text, making it easier to search for motion categories that match the semantic tags from a predetermined motion library. These motion categories can be highly aligned with the semantic information of the voice, reflecting the emotional tendencies and potential semantics in the broadcast voice of the virtual character. Furthermore, motion data belonging to the motion category is searched, and based on the motion data, a motion sequence with higher accuracy is quickly and efficiently synthesized for the virtual character, improving not only the efficiency of motion generation for the virtual character but also the accuracy of motion generation.

[0203] Furthermore, motion sequences can control virtual characters to perform body movements that synchronize with speech at a semantic level. This goes beyond simply following the rhythm of the speech; it significantly improves the degree of matching and accuracy between speech and video, avoids creating mechanical and unnatural visual effects, enhances the simulation and human-like qualities of the virtual character, and optimizes the rendering effect of the virtual character.

[0204] In the motion generation method described above, the potential mapping relationship between audio text and body motion is mined, and an automated process is realized that triggers the generation of virtual character body motion in dual modes of text and audio. This eliminates the need for artificial intervention, the need to combine the performance of a real person with a motion capture system, and the need for animators to make animation corrections. Given text and audio, the device can quickly and automatically generate motion sequences of a virtual character's body motion, replacing complex motion capture and correction processes. It has extremely high versatility and can be applied to virtual character body motion generation tasks in various scenes such as games, live streaming, animation, and movies. It has high practicality, significantly reduces equipment, labor, and time costs, is easy and quick to apply, is dependency-free, and generates motion sequences of high quality and accuracy.

[0205] In each of the above embodiments, the motion generation means for virtual characters will be described in detail, and motion sequences that closely match the semantic level can be rapidly and automatically synthesized using dual-mode voice and text drive signals without the need for artificial intervention. The above motion generation means depends on a pre-built predetermined motion library, and in the embodiments of this application, the library construction process for said predetermined motion library will be described in detail.

[0206] Figure 5 is a flowchart of a method for constructing a motion library for a virtual character according to an embodiment of the present invention. As shown in Figure 5, the embodiment is executed by a computer device, and the computer device is described as a server as an example, and the server may be server 102 in the above implementation environment, and the embodiment includes the following steps 501 to 504.

[0207] In step 501, the server retrieves a sample motion sequence, reference audio, and reference text for each sample character. The reference text indicates the semantic information of the reference audio, and the sample motion sequence controls the sample character to perform motions that match the reference audio.

[0208] The sample characters are publicly available, collectible, and legally compliant virtual or real characters. For example, the sample characters may be virtual characters such as anime characters, virtual newscasters, or digital humans, or real characters such as actors, speakers, or newscasters. The embodiments of this application are not specifically limited to these.

[0209] The collection and use of sample motion sequences, reference audio, and reference text for sample characters are all in compliance with the law, and each sample motion sequence has a one-to-one corresponding reference audio (i.e., embedded audio) and reference text (i.e., subtitles or speech-recognized text).

[0210] In some embodiments, the server acquires sample motion sequences of multiple sample characters, excludes low-quality samples that are not marked with reference audio or reference text, further excludes low-quality samples that do not contain body motion (e.g., a viewpoint where only the head of the virtual character is visible), further excludes low-quality samples that are too short or too long in duration, for example, retaining only sample motion sequences with a duration of 1 to 10 seconds. If a sample motion sequence has both reference audio and reference text, all three are stored in correspondence. If a sample motion sequence has only reference audio, ASR is performed on the reference audio to obtain the corresponding reference text, and all three are stored in correspondence. If a sample motion sequence has only reference text, audio is incorporated into the reference text (i.e., speech synthesis is performed based on the text), the corresponding reference audio is obtained, and all three are stored in correspondence. The number of sample characters and sample motion sequences are not specifically limited.

[0211] In step 502, the server divides the sample motion sequence into multiple sample motion segments based on the relationship between words in the reference text and phonemes in the reference audio, with each sample motion segment relating to one word in the reference text and one phoneme in the reference audio.

[0212] In some embodiments, the server processes each sample motion sequence as a unit, retrieves the reference text and reference audio stored in relation to the sample motion sequence, and, as can be seen from the previous embodiments, establishes the relationship between words in the reference text and phonemes in the reference audio using a phoneme alignment method. In this case, the sample motion sequence can be divided into multiple sample motion segments based on this relationship.

[0213] In some embodiments, one possible method of dividing the sample motion segment is described by the following steps D1-D2.

[0214] In step D1, for each word in the reference text, the server determines the sample audio segment associated with the phoneme from the sample audio, based on the phoneme associated with the word.

[0215] Step D1 described above is the same as step 306 in the previous embodiment, and therefore its explanation is omitted here.

[0216] In step D2, the server divides the sample motion sequence into multiple sample motion segments based on the timestamp interval of each sample audio segment, and aligns each sample motion segment with the timestamp interval of one sample audio segment.

[0217] In some embodiments, for each sample audio segment, the start timestamp of the first audio frame and the end timestamp of the last audio frame in the sample audio segment can be found on the audio time axis. These start and end timestamps constitute a single timestamp interval, and since the reference audio, reference text itself, and sample motion sequence are aligned according to the timestamps, the sample motion sequence can be divided into multiple sample motion segments by directly dividing it according to the timestamp interval of each sample audio segment, while ensuring alignment between the timestamp interval of each sample motion segment and the timestamp interval of the sample audio segment.

[0218] Step 503 involves the server clustering each sample motion segment of each sample character based on the motion characteristics of the sample motion segment, thereby obtaining multiple motion sets, each motion set representing motion data belonging to the same motion category and belonging to different sample characters.

[0219] In some embodiments, the server performs step 502 for each sample motion sequence to divide it into multiple sample motion segments, thus obtaining a set of sample motion segments from different sample characters or different sample motion sequences, then extracts motion features for each sample motion segment, preferably trains a motion feature extraction model, inputs the sample motion segments into the motion feature extraction model, processes the sample motion features by the motion feature extraction model, and outputs the motion features of the sample motion segment.

[0220] Furthermore, after extracting the motion features of each sample motion segment, each sample motion segment is clustered based on a clustering algorithm to form multiple motion sets. Each motion set represents one motion category, and each motion set contains motion data belonging to the corresponding motion category (i.e., each sample motion segment clustered to this motion category). The clustering algorithms include, but are not limited to, KNN (K-Nearest Neighbor) clustering algorithms, K-means clustering algorithms, and hierarchical clustering algorithms.

[0221] In one exemplary scenario, the clustering process of sample motion segments is explained using the K-means clustering algorithm as an example. The K-means clustering algorithm is an iterative clustering analysis algorithm, and its steps are as follows: All sample motion segments are divided into K motion categories. First, K sample motion segments are randomly selected to be the initial cluster centers for each of the K motion categories. Then, the distance between each remaining sample motion segment and the K initial cluster centers (in other words, the distance between motion features) is calculated. Each remaining sample motion segment is assigned to the cluster center closest to it. Thus, the cluster centers and the remaining sample motion segments assigned to them represent a motion set. Each time a sample motion segment is assigned to a motion set, the cluster centers for that motion set are recalculated based on all existing sample motion segments. The above process is repeated until certain termination conditions are met, which include, but are not limited to, that the sample motion segments have not been reassigned to different motion sets (or the minimum number of sample motion segments have been reassigned to different motion sets), that the cluster centers have not changed (or the minimum number of cluster centers have changed), or that the sum of squared errors of K-means clustering is locally minimized. The embodiments of the present application do not specifically limit the termination conditions of the K-means clustering algorithm.

[0222] JPEG0007898030000001.jpg179170

[0223] In some other embodiments, the motion sets of each motion category can be subdivided into multiple subcategories in a similar manner. The clustering process for subcategories is the same as that for motion categories, and will not be explained here. The clustering method allows for the division of large amounts of motion data into multiple motion categories, ensuring that the motion data within each motion category has a certain degree of similarity and that the motion data between different motion categories has a certain degree of difference. Thus, each motion category can represent one motion term, meaning that motion data belonging to different motion categories are considered to be different from each other at the semantic level.

[0224] In step 504, the server builds a motion library based on multiple motion sets.

[0225] In some embodiments, the server directly constructs a motion library based on the K motion sets formed by clustering in step 504. The motion library contains the K motion sets, i.e., motion data belonging to K motion categories, including virtual characters.

[0226] In some embodiments, the categorical features of the motion category to which each of the K motion sets belongs are further calculated and stored. This simplifies the motion library construction process and improves the efficiency of motion library construction.

[0227] In some other embodiments, further data cleaning can be performed on the K motion sets formed by clustering in step 504, filtering and excluding anomalous samples that are far from the cluster center within each motion set. This improves the similarity of each sample motion segment within the same motion category and decreases the similarity of each sample motion segment within different motion categories. The data cleaning flow for a single motion set will be described below using steps E1 to E4 as an example.

[0228] In step E1, the server obtains the category features of the motion category represented by each motion set, and these category features are the average motion features of each sample motion segment within the motion set.

[0229] In some embodiments, for each motion set formed by clustering in step 504, the average motion feature is calculated as a categorical feature of the motion category represented by the motion set, based on the motion features of each sample motion segment in the motion set, and this categorical feature represents the cluster center of the motion set.

[0230] In step E2, the server determines the contribution score of the motion features of each sample motion segment in the motion set to the category features, and the contribution score represents the degree of matching between the sample motion segment and the motion category.

[0231] Each sample motion segment belongs to a single motion category, but the degree of matching between different sample motion segments and motion categories may vary. This degree of matching evaluates whether the motion performed by the sample motion segment is standard or not. For example, the category features of a motion category are the average motion features of multiple sample motion segments in a motion set. The motion features of some sample motion segments in the motion set are similar to the average motion features, indicating that the motion performed by those sample motion segments and belonging to that motion category is standard. The motion features of some sample motion segments are not similar to the average motion features, indicating that those sample motion segments perform motions belonging to that motion category, but the motions performed are not standard. Therefore, the contribution score represents the standardity of the sample motion segment to that motion category.

[0232] In some embodiments, for each sample motion segment in the motion set, the contribution score of the motion features of the sample motion segment to the category features in step E1 is calculated, preferably by directly calculating the feature similarity between the motion features and the category features, and then exponentially normalizing the feature similarity of each sample motion segment in the entire motion set to obtain the contribution score of each sample motion segment (referring to the exponentially normalized feature similarity). By using the exponentially normalized feature similarity as the metric for the contribution score in this way, the computational complexity of the contribution score can be reduced and the computational efficiency of the contribution score can be improved.

[0233] In some other embodiments, a metric is provided in which the contribution score is the within-class variance (also called the N-1 variance) after excluding the individual itself. Such an within-class variance represents the contribution of the excluded individual to the entire cluster, i.e., the contribution score of the excluded sample motion segment to the entire motion set. This contribution score has better expressive power, the metric angle is more accurate, a higher contribution score indicates that the motion of the sample motion segment is more typical, and a lower contribution score indicates that the motion of the sample motion segment is less typical. Steps E21-E22 below describe in detail how to obtain the within-class variance (i.e., one possible contribution score) of a single sample motion segment.

[0234] In step E21, the server obtains motion scores for each of the remaining motion segments in the motion set, with each motion score representing the similarity between the remaining motion segments and the category features.

[0235] The remaining motion segments refer to the sample motion segments in the motion set other than the sample motion segment in question.

[0236] In some embodiments, for each sample motion segment in the motion set, the feature similarity between the motion features of the sample motion segment and the category features in step E1 is calculated. Furthermore, the feature similarity of each sample motion segment in the entire motion set is exponentially normalized to obtain the motion score (referring to the exponentially normalized feature similarity) for each sample motion segment. Next, the current sample motion segment is excluded, and the motion scores of each remaining motion segment other than the sample motion segment are determined.

[0237] In step E22, the server determines the intraclass variance after excluding the sample motion segment based on the motion score of each remaining motion segment, and determines this intraclass variance as the contribution score of the sample motion segment.

[0238] In some embodiments, the server calculates the average of the motion scores of all remaining motion segments obtained in step E21, sets this average as a single average motion score, determines the within-class variance after excluding the sample motion segment based on this average motion score and the motion scores of each remaining motion segment, and determines this within-class variance as the contribution score of the sample motion segment.

[0239] In one example, assuming that the motion set contains N sample motion segments, and that the Nth sample motion segment is excluded, the remaining motion segments refer to the 1st to N-1th sample motion segments, and the within-class variance (also called N-1 variance) is obtained using the following formula.

number

[0240] JPEG0007898030000003.jpg46170

[0241] In the above process, the within-class variance (also called N-1 variance) after excluding the individual itself is provided as a metric indicator for the contribution score. Thus, in practice, after excluding a specified individual, the within-class variance of the remaining individuals is calculated. A larger within-class variance indicates that the excluded individual had a smaller impact on the departure from the cluster, while a larger variance indicates that the remaining individuals had a larger impact on the departure from the cluster. Therefore, the within-class variance can adequately evaluate the contribution of the excluded individual to the entire cluster, that is, it represents the contribution score of the excluded sample motion segment to the entire motion set. The representational power of this contribution score is better, and the metric angle is more accurate. A larger contribution score indicates that the motion of the sample motion segment is more standard, while a smaller contribution score indicates that the motion of the sample motion segment is less standard. Therefore, it is necessary to exclude non-standard sample motion segments (i.e., sample motion segments with low contribution scores), thus facilitating data cleaning within each motion category.

[0242] In step E3, the server removes sample motion segments from the motion set whose contribution score meets the exclusion criteria.

[0243] In some embodiments, the server sorts each sample motion segment in the motion set in descending order of contribution score and discards the last-placed sample motion segment in that sort. In this way, each data cleaning discards only the sample motion segments with the least impact on the cluster's detachment, avoiding the inadvertent exclusion of high-quality sample motion segments.

[0244] In some other embodiments, the server may sort each sample motion segment of the motion set in descending order of contribution score, and in this sort, the last j sample motion segments may be excluded. In this way, each data cleaning discards j sample motion segments that have little impact on the detachment from the cluster, and thus, by flexibly controlling the value of j, the data cleaning speed of the motion set can be finely tuned, where j is an integer greater than or equal to 1.

[0245] In one example, as shown in Figure 7, which is a diagram illustrating the principle of data cleaning of a motion set according to an embodiment of the present invention, for a motion set of a certain motion category, the first sample motion segment is excluded, the intraclass variance of the remaining N-1 motion segments is calculated, and the contribution score of the first sample motion segment is obtained as 0.2. The above operation is repeated for each sample motion segment to calculate the contribution score of each sample motion segment, and then each sample motion segment is sorted in descending order of contribution score, and then the last sample motion segment in the sort is excluded, for example, the last sample motion segment with a contribution score of 0.02 is excluded.

[0246] In step E4, the server updates the category features and contribution scores based on the excluded motion set, repeats the exclusion operation multiple times, and stops the iteration when the iteration stop conditions are met.

[0247] In some embodiments, in step E3, one (or more) sample motion segments with low contribution scores are excluded. This requires a change in the number of samples in the motion set, and the cluster-centered category features must be recalculated. This updates the category features based on the same method as in step E1. As a result, the category features change, and the contribution score of each sample motion segment must also be recalculated. This updates the contribution score based on the same method as in step E2. Furthermore, based on the updated contribution score, sample motion segments whose contribution scores meet the exclusion criteria are continuously excluded, using the same method as in step E3. Steps E1 to E3 are repeated until the iteration stop condition is met to obtain a pure, high-quality motion set. The iteration termination conditions include, but are not limited to, the following: the number of iterations reaches a threshold and the threshold is an integer greater than 0; the sample capacity of the motion set is reduced to a predetermined capacity and the predetermined capacity is an integer greater than or equal to 1; or the contribution score at the end of the sort is greater than the contribution threshold and the contribution threshold is a numerical value greater than or equal to 0. The embodiments of this application do not specifically limit the iteration termination conditions.

[0248] In steps E1-E4 described above, because the motion sets directly formed by clustering are coarse, there may be some motions with large differences within a class. To avoid affecting the accuracy of motion category clustering, such motions need to be excluded. This provides a method for performing data cleaning, data filtering, or data purification for each motion set. The motion library built based on the final cleaned motion sets will have better motion generation efficiency and higher availability. The entire iteration, sorting, and selection process can be achieved through self-monitoring without the need for artificial intervention, so the library construction stage can also be automated, resulting in lower library construction costs and higher library construction efficiency.

[0249] In steps 501 to 504 above, the process of building a motion library that provides support for the motion generation means of a virtual character was described in detail. However, considering that the motion library may not be immutable in some embodiments, it may be necessary to increase or add some motion data. Below, steps F1 to F4 will be used as an example to describe the process of storing newly added motion sequences in the library.

[0250] In step F1, the server retrieves the newly added reference audio and newly added reference text associated with any newly added motion sequence other than the motion library.

[0251] The newly added reference text indicates the semantic information of the newly added reference audio, and the newly added motion sequence controls the corresponding sample character to perform motion that matches the newly added reference audio.

[0252] Step F1 is the same as step 501, so we will omit the explanation here.

[0253] In step F2, the server divides the newly added motion sequence into multiple newly added motion segments based on the relationship between the words in the newly added reference text and the phonemes in the newly added reference audio.

[0254] Each newly added motion segment is associated with one word in the newly added reference text and one phoneme in the newly added reference audio.

[0255] Step F2 is the same as step 502, so we will omit the explanation here.

[0256] In step F3, for each newly added motion segment, the server determines the target motion set to which the newly added motion segment belongs from among multiple motion sets in the motion library, based on the motion characteristics of the newly added motion segment.

[0257] In some embodiments, for each newly added motion segment, the motion characteristics of the newly added motion segment are calculated based on the same method as in step 503, the distance between the motion characteristics of the newly added motion segment and the category characteristics of each motion set is calculated, and the newly added motion segment is assigned to the target motion set with the closest distance.

[0258] In step F4, the server adds the newly added motion segment to the target motion set, updates the category features and contribution score, and removes sample motion segments from the target motion set whose contribution score meets the exclusion criteria.

[0259] In some embodiments, after assigning the newly added motion segment to the target motion set, the number of samples in the target motion set changes, so the cluster-centered category features must be recalculated. Therefore, the category features are recalculated based on the same method as in step E1, and accordingly, the category features change, so the contribution score of each sample motion segment (including the newly added motion segment) must also be recalculated. Therefore, the contribution score is recalculated based on the same method as in step E2, and furthermore, based on the newly calculated contribution score, sample motion segments whose contribution score meets the exclusion criteria are continuously excluded.

[0260] In one example, as shown in Figure 8, Figure 8 is a diagram illustrating the principle of data supplementation for newly added motion segments according to an embodiment of the present invention. Assume that two newly added motion segments have been added to a target motion set containing the newly added motion segments, and that the contribution scores of these two newly added motion segments are calculated to be 0.7 and 0.04, respectively, then, for each sample motion segment in the entire target motion set, including these two newly added motion segments, the data is re-sorted (reverse order) according to the contribution score, and the last sample motion segment after re-sorting is excluded. For example, the last newly added motion segment with a contribution score of 0.04 is excluded.

[0261] All of the above preferred technical means can be combined in any way to form preferred embodiments of the present disclosure, which will not be described individually here.

[0262] The method according to the embodiment of the present invention divides a sample motion sequence into a series of sample motion segments according to the guidance of reference text and reference audio, then divides the sample motion segments into multiple motion categories by clustering, and stores the clustered motion data in each motion category, each motion category having one motion set. In this way, a motion library having multiple motion categories can be constructed, distinguishing motion data belonging to different motion categories at the semantic level, making it easier to send to the subsequent motion generation process, and improving the efficiency and accuracy of motion generation by detecting the motion category that best matches using semantic tags as an index.

[0263] The above motion library construction method provides an automatic learning-based semantic generation, automatic classification, and automatic selection mechanism, automatically excluding low-quality samples and allowing new samples to be added to any motion category at any time. By simply cleaning the motion data in each motion category again using contribution scores, the high quality of the motion library is ensured, and the uniformity of the motion data in each motion category is improved.

[0264] Figure 9 is a schematic diagram of a motion generation device for a virtual character according to an embodiment of the present application, and as shown in Figure 9, the device is An acquisition module for acquiring the voice and text of a virtual character, wherein the text includes an acquisition module 901 that indicates the semantic information of the voice, An analysis module that determines the semantic tags of the text based on the text, wherein the semantic tags represent at least one of the part-of-speech information of the words in the text or the sentiment information expressed by the text, A search module that searches for motion categories and motion data belonging to a given motion category that match a given semantic tag from a predetermined motion library, wherein the predetermined motion library includes a search module 903 that contains motion data belonging to multiple motion categories of the virtual character, A generation module that generates a motion sequence for the virtual character based on the motion data, the motion sequence includes a generation module 904 that controls the virtual character to perform motions that match the sound.

[0265] The apparatus according to the embodiment of the present invention uses voice and text as dual-mode drive signals, extracts semantic tags at the semantic level based on the text, making it easier to search for motion categories that match the semantic tags from a predetermined motion library. These motion categories can be highly aligned with the semantic information of the voice, reflecting the emotional tendencies and potential semantics in the broadcast voice of the virtual character. Furthermore, it searches for motion data belonging to the motion category and, based on the motion data, quickly and efficiently synthesizes motion sequences with higher accuracy for the virtual character, improving not only the efficiency of motion generation for the virtual character but also the accuracy of motion generation.

[0266] Furthermore, motion sequences can control virtual characters to perform body movements that synchronize with speech at a semantic level. This goes beyond simply following the rhythm of the speech; it significantly improves the degree of matching and accuracy between speech and video, avoids creating mechanical and unnatural visual effects, enhances the simulation and human-like qualities of the virtual character, and optimizes the rendering effect of the virtual character.

[0267] In some embodiments, the analysis module 902 determines the sentiment tags of the text based on the text, determines at least one word contained in the text based on the text, searches the part-of-speech tag to which each word belongs from the part-of-speech table, and determines the sentiment tags and the part-of-speech tag to which the at least one word belongs as the semantic tags of the text.

[0268] In some embodiments, the search module searches for a motion category that matches the semantic tag of each word in the text, based on the semantic tag to which the word belongs, from a predetermined motion library, and retrieves motion data belonging to that motion category from the predetermined motion library.

[0269] In some embodiments, based on the apparatus configuration in Figure 9, the generation module 904 is configured as follows: A decision unit that determines, for each word contained in the text, the phoneme to which the phoneme belongs based on the phoneme associated with the word, A segment generation unit that generates a motion segment that matches the audio segment based on the motion data and audio segment corresponding to the word, Includes a sequence generation unit that generates a motion sequence that matches the audio based on each motion segment that matches the audio segment of each word.

[0270] In some embodiments, based on the apparatus configuration shown in Figure 9, the segment generation unit is configured as follows: A determination subunit that determines the at least one key motion frame with the highest degree of semantic matching with the word from the motion data, The system includes a synthesis subunit that synthesizes the at least one key motion frame into a motion segment that matches the audio segment, based on the audio segment.

[0271] In some embodiments, the synthesis subunit interpolates at least one key motion frame to obtain a motion segment of the same length as the audio segment if the number of frames in the key motion frame does not exceed the number of audio frames in the audio segment, and if the number of frames in the key motion frame exceeds the number of audio frames, it creates a motion segment of the same length as the audio segment and pads each frame of the motion segment with a predetermined motion frame in a predetermined motion category.

[0272] In some embodiments, the sequence generation unit splices each motion segment that matches each audio segment based on the timestamp order of each audio segment to obtain a spliced ​​motion sequence, and then performs motion smoothing on each motion frame in the spliced ​​motion sequence to obtain a motion sequence.

[0273] In some embodiments, the search module 903 extracts the semantic features of each semantic tag, searches for the category features of multiple candidate categories in the predetermined motion library, determines the motion category from the multiple candidate categories, and confirms that the category features and semantic features of the motion category satisfy the similarity condition.

[0274] In some embodiments, the search module 903 further sets a motion category that matches the word meaning tag as a predetermined motion category if none of the category features and semantic features of the multiple candidate categories satisfy the similarity condition.

[0275] All of the above preferred technical means can be combined in any way to form preferred embodiments of the present disclosure, which will not be described individually here.

[0276] Furthermore, the virtual character motion generation device according to the above embodiment is described as merely an example of dividing each of the above-mentioned functional modules when generating the body motion of a virtual character. In actual applications, the above functions can be completed by assigning them to different functional modules as needed; that is, the internal structure of the computer device can be divided into different functional modules to complete all or some of the above functions. In addition, the virtual character motion generation device according to the above embodiment belongs to the same concept as the embodiment of the virtual character motion generation method, and its specific implementation process can be found in the embodiment of the virtual character motion generation method, and will not be described again here.

[0277] FIG. 10 is a schematic configuration diagram of a virtual character motion library construction device according to an embodiment of the present application. As shown in FIG. 10, the device includes: The sample acquisition module 1001 acquires the sample motion sequence, reference audio, and reference text of each sample character. The reference text indicates the semantic information of the reference audio, and the sample motion sequence controls the sample character to execute a motion corresponding to the reference audio.

[0278] The segment division module 1002 divides the sample motion sequence into a plurality of sample motion segments based on the association relationship between the words in the reference text and the phonemes in the reference audio. Each sample motion segment is associated with one word in the reference text and one phoneme in the reference audio.

[0279] The clustering module 1003 clusters each sample motion segment of each sample character based on the motion characteristics of the sample motion segments, and obtains a plurality of motion sets. Each motion set represents the motion data of different sample characters clustered into the same motion category.

[0280] The construction module 1004 constructs a motion library based on the plurality of motion sets.

[0281] The apparatus according to the embodiment of the present invention divides a sample motion sequence into a series of sample motion segments according to the guidance of reference text and reference audio, and then divides the sample motion segments into multiple motion categories by clustering. Each motion category has one motion set and stores all the motion data clustered in that motion category. In this way, a motion library having multiple motion categories can be constructed, distinguishing motion data belonging to different motion categories at the semantic level, making it easier to send to the subsequent motion generation process, and improving the efficiency and accuracy of motion generation by detecting the motion category that best matches using semantic tags as an index.

[0282] In some embodiments, the segmentation module 1002 determines, for each word in the reference text, the sample audio segment associated with the phoneme from the sample audio based on the phoneme associated with the word, divides the sample motion sequence into multiple sample motion segments based on the timestamp interval of each sample audio segment, and aligns each sample motion segment with the timestamp interval of one sample audio segment.

[0283] In some embodiments, based on the apparatus configuration of Figure 10, the apparatus is For each motion set, a feature acquisition module acquires the category features of the motion category represented by the motion set, wherein the category features are the average motion features of each sample motion segment within the motion set, A step of determining the contribution score of the motion features of each sample motion segment in the motion set to the category features, wherein the contribution score is determined by a determination module that represents the degree of matching between the sample motion segment and the motion category, An exclusion module that excludes sample motion segments from the motion set whose contribution score meets the exclusion criteria, The system further includes an iterative module that updates the category features and contribution scores based on the excluded motion set, performs the exclusion operation multiple times, and stops the iteration when the iteration stop condition is met.

[0284] In some embodiments, the decision module takes the step of obtaining motion scores for each of the remaining motion segments in the motion set, excluding the sample motion segment, where the motion scores represent the similarity between the remaining motion segments and the category features; determining the within-class variance after excluding the sample motion segment based on the motion scores of each remaining motion segment; and determining the within-class variance as the contribution score for the sample motion segment.

[0285] In some embodiments, the exclusion module sorts each sample motion segment in the motion set in descending order of contribution score and excludes the last sample motion segment in that sort.

[0286] In some embodiments, the sample acquisition module 1001 further acquires newly added reference audio and newly added reference text associated with any newly added motion sequence other than the motion library.

[0287] The segment division module 1002 further divides the newly added motion sequence into a plurality of newly added motion segments based on the relationship between words in the newly added reference text and phonemes in the newly added reference audio.

[0288] The clustering module 1003 further determines, for each newly added motion segment, the target motion set to which the newly added motion segment belongs from among multiple motion sets in the motion library, based on the motion characteristics of the newly added motion segment.

[0289] The construction module 1004 further adds the newly added motion segment to the target motion set, updates the category features and contribution score, and removes sample motion segments from the target motion set whose contribution score satisfies the exclusion criteria.

[0290] All of the above preferred technical means can be combined in any way to form preferred embodiments of the present disclosure, which will not be described individually here.

[0291] In addition, the virtual character motion library construction device according to the above embodiment was described using only the division of each functional module as an example when constructing the motion library. However, in actual applications, the above functions can be completed by assigning them to different functional modules as needed. In other words, all or part of the functions described above can be completed by dividing the internal structure of the computer device into different functional modules. Furthermore, the virtual character motion library construction device according to the above embodiment belongs to the same concept as the embodiment of the method for constructing a virtual character motion library, and its specific implementation process can be found in the embodiment of the method for constructing a virtual character motion library, and will not be described again here.

[0292] Figure 11 is a schematic diagram of a computer device according to an embodiment of the present application. As shown in Figure 11, the computer device 1100 may vary significantly in configuration or performance. The computer device 1100 includes one or more processors (Central Processing Units, CPUs) 1101 and one or more memories 1102. At least one computer program is stored in the memory 1102. When this at least one computer program is loaded and executed by the one or more processors 1101, a method for generating virtual character motion or a method for constructing a virtual character motion library according to each embodiment is realized. Preferably, the computer device 1100 further includes components such as a wired or wireless network interface, a keyboard, and an input / output interface for input / output. The computer device 1100 further includes other components for realizing device functions, which are not described here.

[0293] In exemplary embodiments, a computer-readable storage medium is further provided, for example, a memory containing at least one computer program, which, when executed by a processor in a computer device, can perform the motion generation method for a virtual character or the motion library construction method for a virtual character in each of the embodiments. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disks, and optical data storage devices.

[0294] In an exemplary embodiment, a computer program product is further provided, comprising one or more computer programs, which are stored on a computer-readable storage medium. One or more processors of a computer device can read the one or more computer programs from the computer-readable storage medium, and by executing the one or more computer programs, the computer device can execute the virtual character motion generation method or the virtual character motion library construction method according to the above embodiment.

[0295] As those skilled in the art will understand, all or part of the steps for realizing the above embodiment may be completed by hardware or by a program instructing the relevant hardware, preferably the program being stored in a computer-readable storage medium, preferably the storage medium being read-only memory, a magnetic disk or an optical disk, etc.

[0296] The foregoing are merely preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application should all be included within the scope of protection of the present application.

Claims

1. A method for generating motion for a virtual character executed by a computer device, A step of obtaining the voice and text of a virtual character, wherein the text indicates the semantic information of the voice, A step of determining a semantic tag of the text based on the text, wherein the semantic tag represents at least one of the part-of-speech information of a word in the text or the sentiment information expressed by the text; A step of searching a predetermined motion library for motion categories that match the semantic tag and motion data belonging to the motion category, wherein the predetermined motion library includes motion data belonging to multiple motion categories of the virtual character. A method for generating motion for a virtual character, comprising the steps of generating a motion sequence for the virtual character based on the motion data, wherein the motion sequence controls the virtual character to perform motions that synchronize with the sound.

2. The step of determining the semantic tags of the text based on the text is: The steps include determining the sentiment tags of the text based on the text, A step of determining at least one word contained in the text based on the text, The steps include searching for the part-of-speech tag to which each of the aforementioned words belongs from the part-of-speech table, The method according to claim 1, comprising the step of determining the emotion tag and the part-of-speech tag to which the at least one word belongs as semantic tags of the text.

3. The step of searching a predetermined motion library for motion categories that match the semantic tags and motion data belonging to those motion categories is: For each word contained in the aforementioned text, The steps include: searching for a motion category that matches the semantic tag of the word from a predetermined motion library based on the semantic tag to which the word belongs; The method according to claim 1, comprising the step of searching for motion data belonging to the motion category from the predetermined motion library.

4. The step of generating a motion sequence for the virtual character based on the motion data is: For each word contained in the text, the steps include: determining the speech segment to which the phoneme belongs from the speech based on the phoneme associated with the word; and generating a motion segment that matches the speech segment based on the motion data and the speech segment corresponding to the word. The method according to claim 3, comprising the step of generating a motion sequence that matches the speech based on each motion segment that matches the speech segment of each word.

5. The step of generating a motion segment that matches the voice segment based on the motion data and voice segment corresponding to the word is: The steps include determining at least one key motion frame from the motion data that has the highest degree of semantic matching with the word, The method according to claim 4, comprising the step of synthesizing the at least one key motion frame to a motion segment that matches the audio segment, based on the audio segment.

6. The step of synthesizing the at least one key motion frame to the motion segment that matches the audio segment, based on the audio segment, If the number of frames in the key motion frame does not exceed the number of audio frames in the audio segment, the step of interpolating at least one key motion frame to obtain a motion segment of the same length as the audio segment, The method according to claim 5, wherein if the number of frames of the key motion frame exceeds the number of audio frames, a motion segment of the same length as the audio segment is created, and each frame of the motion segment is padded to a predetermined motion frame in a predetermined motion category.

7. The step of generating the motion sequence that matches the speech based on each motion segment that matches the speech segment of each word is: The steps include: splicing each motion segment that matches each audio segment based on the timestamp order of each audio segment to obtain a spliced ​​motion sequence; The method according to claim 4, comprising the step of performing motion smoothing on each motion frame in the splicing motion sequence to obtain the motion sequence.

8. The step of searching a predetermined motion library for motion categories that match the semantic tags and motion data belonging to those motion categories is: The steps include extracting the semantic features of the aforementioned semantic tags, The steps include: searching for category characteristics of multiple candidate categories in the predetermined motion library; The method according to claim 1, comprising the step of determining the motion category from a plurality of candidate categories, wherein the category features and semantic features of the motion category satisfy a similarity condition.

9. The aforementioned method, The method according to claim 8, further comprising the step of setting a motion category that matches the semantic tag as a predetermined motion category if none of the category features and semantic features of the multiple candidate categories satisfy the similarity condition.

10. A method for building a motion library for a virtual character executed by a computer device, A step of obtaining a sample motion sequence, reference audio, and reference text for each sample character, wherein the reference text indicates semantic information of the reference audio, and the sample motion sequence controls the sample character to perform motions that match the reference audio. A step of dividing the sample motion sequence into a plurality of sample motion segments based on the relationship between a word in the reference text and a phoneme in the reference audio, wherein each sample motion segment comprises a step relating to one word in the reference text and one phoneme in the reference audio, A step of clustering each sample motion segment of each sample character based on the motion characteristics of the aforementioned sample motion segment and obtaining multiple motion sets, wherein each motion set represents motion data belonging to the same motion category and belonging to different sample characters, A method for building a motion library for a virtual character, comprising the steps of building a motion library based on a plurality of the aforementioned motion sets.

11. The step of dividing the sample motion sequence into multiple sample motion segments based on the relationship between words in the reference text and phonemes in the reference audio is as follows: For each word in the aforementioned reference text, the step of determining a sample audio segment associated with the phoneme from the sample audio based on the phoneme associated with the word, The method according to claim 10, comprising the steps of dividing the sample motion sequence into a plurality of sample motion segments based on the timestamp interval of each sample audio segment, wherein each sample motion segment is aligned with the timestamp interval of one sample audio segment.

12. The aforementioned method, For each motion set, a step of obtaining the category features of the motion category represented by the motion set, wherein the category features are the average motion features of each sample motion segment in the motion set; A step of determining the contribution score of the motion features of each sample motion segment in the motion set to the category features, wherein the contribution score represents the degree of matching between the sample motion segment and the motion category, The steps include: excluding sample motion segments from the motion set whose contribution score satisfies the exclusion criteria; The method according to claim 10, further comprising the steps of updating the category features and the contribution score based on the motion set after exclusion, performing the exclusion operation multiple times, and stopping the iteration when the iteration stop condition is met.

13. The step of determining the contribution score of the motion features of each sample motion segment in the motion set to the category features is: A step of obtaining motion scores for each of the remaining motion segments other than the sample motion segment for any of the sample motion segments in the motion set, wherein the motion scores represent the similarity between the remaining motion segments and the category features. The method according to claim 12, comprising the steps of determining the within-class variance after excluding the sample motion segment based on the motion scores of each remaining motion segment, and determining the within-class variance as the contribution score of the sample motion segment.

14. The step of obtaining a motion set by excluding sample motion segments from the aforementioned motion set whose contribution score satisfies the exclusion criteria is as follows: The method according to claim 12, comprising the steps of sorting each sample motion segment of the motion set in descending order of contribution score, and excluding the last sample motion segment in the sort.

15. The aforementioned method, For any newly added motion sequence other than a predetermined motion library, the steps include obtaining newly added reference audio and newly added reference text related to the newly added motion sequence, The steps include dividing the newly added motion sequence into multiple newly added motion segments based on the relationship between words in the newly added reference text and phonemes in the newly added reference audio, For each newly added motion segment, the steps include determining the target motion set to which the newly added motion segment belongs from a plurality of motion sets in the predetermined motion library, based on the motion characteristics of the newly added motion segment, The method according to claim 12, further comprising the steps of adding the newly added motion segment to the target motion set, updating the category features and the contribution score, and removing from the target motion set any sample motion segments whose contribution score satisfies the exclusion criteria.

16. An acquisition module for acquiring the voice and text of a virtual character, wherein the text includes an acquisition module that indicates the semantic information of the voice, An analysis module that determines the semantic tags of the text based on the text, wherein the semantic tags represent at least one of the part-of-speech information of the words in the text or the sentiment information expressed by the text, A search module that searches for motion categories and motion data belonging to a given motion category that match the given semantic tag from a predetermined motion library, wherein the predetermined motion library includes a search module that includes motion data belonging to multiple motion categories of the virtual character, A motion generation device for a virtual character, comprising: a generation module that generates a motion sequence of the virtual character based on the motion data, wherein the motion sequence controls the virtual character to perform motions synchronized with the sound.

17. A sample acquisition module that acquires a sample motion sequence, reference audio, and reference text for each sample character, wherein the reference text indicates the semantic information of the reference audio, and the sample motion sequence controls the sample character to perform motions that match the reference audio. A segmentation module that divides the sample motion sequence into a plurality of sample motion segments based on the relationship between a word in the reference text and a phoneme in the reference audio, wherein each sample motion segment comprises a segmentation module associated with one word in the reference text and one phoneme in the reference audio, A clustering module that clusters each sample motion segment of each sample character based on the motion characteristics of the aforementioned sample motion segments and obtains multiple motion sets, wherein each motion set represents motion data belonging to the same motion category and belonging to different sample characters, A device for building a motion library for a virtual character, comprising a construction module that constructs a motion library based on a plurality of the aforementioned motion sets.

18. A computer device comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to realize the motion generation method for a virtual character described in any one of claims 1 to 9, or the motion library construction method for a virtual character described in any one of claims 10 to 15.

19. A computer program that causes a computer to implement a method for generating motion for a virtual character according to any one of claims 1 to 9, or a method for constructing a motion library for a virtual character according to any one of claims 10 to 15.

Images