Movement generation method and apparatus for virtual character, and construction method and apparatus for movement library of virtual avatar
By building a virtual image action library, using dual-modal information of audio and text, the problems of low efficiency and poor accuracy of virtual image action generation are solved, efficient and accurate action generation is achieved, and the simulation and rendering of virtual image are improved. Effect.
Patent Information
- Application Number
- PCT/CN2024/093505
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-21
AI Technical Summary
The prior art is inefficient and has poor accuracy when generating body movements of virtual images. The motion capture method requires manual intervention, and the effect of the action generation model is not ideal and cannot effectively reflect the semantic information of audio and text.
By building a virtual image action library, using dual-modal information of audio and text, dividing action clips, clustering action data, and generating action sequences, realizing action generation of virtual image without manual intervention, improving the efficiency and accuracy of action generation.
It realizes efficient and accurate virtual image action generation, can effectively reflect the semantic information of audio and text, and improves the simulation and rendering effect of virtual image.
Smart Images

Figure CN2024093505_21112024_PF_FP_ABST
Abstract
Description
Method for generating virtual image motion, method and device for constructing motion library
[0001] This application claims priority to the Chinese patent application filed on May 15, 2023, with application number 202310547509.7, and invention name “Method for generating motion of virtual image, method and device for constructing motion library”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a method for generating a virtual image's actions, and a method and device for constructing an action library. Background Art
[0003] With the development of computer technology, avatars are increasingly being used in live streaming, film and television, animation, gaming, virtual social networking, and human-computer interaction. For example, in live streaming scenarios, avatars act as anchors, delivering announcements or conducting conversations. To enhance the rendering quality of avatars, motion generation is required.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a method for generating an action for an avatar, a method for constructing an action library, and a device for generating an action library, which can quickly and efficiently synthesize an action sequence with higher accuracy for an avatar, thereby improving the efficiency of generating an action for the avatar. The technical solution is as follows:
[0006] In one aspect, a method for generating an action of a virtual image is provided, which is applied to a computer device, and the method comprises:
[0007] obtaining audio and text of the avatar, wherein the text indicates semantic information of the audio;
[0008] Determining, based on the text, a semantic tag for the text, where the semantic tag represents at least one of part-of-speech information of a word in the text or sentiment information expressed by the text;
[0009] Retrieving an action category matching the semantic tag and action data belonging to the action category from a preset action library, wherein the preset action library includes action data of the avatar belonging to multiple action categories;
[0010] Based on the action data, an action sequence of the virtual image is generated, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
[0011] In one aspect, a method for constructing an action library of a virtual character is provided, which is applied to a computer device, and the method comprises:
[0012] Acquire a sample action sequence, reference audio, and reference text for each sample image, wherein the reference text indicates semantic information of the reference audio, and the sample action sequence is used to control the sample image to perform an action coordinated with the reference audio;
[0013] Based on the association relationship between words in the reference text and phonemes in the reference audio, dividing the sample action sequence into a plurality of sample action segments, each sample action segment being associated with a word in the reference text and a phoneme in the reference audio;
[0014] Clustering each sample action segment of each sample image based on the action features of the sample action segments to obtain a plurality of action sets, each action set indicating action data belonging to the same action category and belonging to different sample images;
[0015] An action library is constructed based on the multiple action sets.
[0016] In one aspect, a device for generating a motion of a virtual image is provided, the device comprising:
[0017] an acquisition module, configured to acquire audio and text of the avatar, wherein the text indicates semantic information of the audio;
[0018] An analysis module, configured to determine a semantic tag of the text based on the text, wherein the semantic tag represents at least one of part-of-speech information of a word in the text or sentiment information expressed by the text;
[0019] a retrieval module, configured to retrieve an action category matching the semantic tag and action data belonging to the action category from a preset action library, wherein the preset action library includes action data of the avatar belonging to multiple action categories;
[0020] A generation module is used to generate an action sequence for the virtual image based on the action data, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
[0021] In one aspect, a device for constructing an action library of a virtual image is provided, the device comprising:
[0022] a sample acquisition module, configured to acquire a sample action sequence, reference audio, and reference text for each sample avatar, wherein the reference text indicates semantic information of the reference audio, and the sample action sequence is used to control the sample avatar to perform an action coordinated with the reference audio;
[0023] a segment division module, configured to divide the sample action sequence into a plurality of sample action segments based on the association between words in the reference text and phonemes in the reference audio, each sample action segment being associated with a word in the reference text and a phoneme in the reference audio;
[0024] a clustering module for clustering each sample action segment of each sample image based on the action features of the sample action segments to obtain a plurality of action sets, each action set indicating action data belonging to the same action category and belonging to different sample images;
[0025] A construction module is used to construct an action library based on the multiple action sets.
[0026] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement a method for generating a virtual image's action or a method for constructing a virtual image's action library as described in any possible implementation method described above.
[0027] On the one hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement a method for generating an action of a virtual image or a method for constructing an action library of a virtual image as described in any possible implementation method described above.
[0028] In one aspect, a computer program product is provided, comprising one or more computer programs stored in a computer-readable storage medium. One or more processors of a computer device can read the one or more computer programs from the computer-readable storage medium and execute the one or more computer programs, enabling the computer device to perform any of the aforementioned possible implementations of the method for generating an avatar's actions or the method for constructing an avatar's action library. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0030] FIG1 is a schematic diagram of an implementation environment of a method for generating motions of a virtual image provided by an embodiment of the present application;
[0031] FIG2 is a flow chart of a method for generating an action of a virtual image provided by an embodiment of the present application;
[0032] FIG3 is a flow chart of a method for generating an action of a virtual image provided by an embodiment of the present application;
[0033] FIG4 is a schematic diagram of a method for generating a virtual image's motion according to an embodiment of the present application;
[0034] FIG5 is a flowchart of a method for constructing an action library of a virtual image provided by an embodiment of the present application;
[0035] FIG6 is a schematic diagram of a method for creating an action library according to an embodiment of the present application;
[0036] FIG7 is a diagram showing a data cleaning principle of an action set provided in an embodiment of the present application;
[0037] FIG8 is a schematic diagram showing a data supplement principle for a newly added action segment provided in an embodiment of the present application;
[0038] FIG9 is a schematic structural diagram of a device for generating motions of a virtual image provided by an embodiment of the present application;
[0039] FIG10 is a schematic structural diagram of a device for constructing an action library of a virtual image provided by an embodiment of the present application;
[0040] FIG11 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0042] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.
[0043] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of action clips means two or more action clips.
[0044] In this application, the term "including at least one of A or B" refers to the following situations: including only A, including only B, and including both A and B.
[0045] The user-related information (including but not limited to the user's device information, personal information, behavioral information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, when applied to specific products or technologies in the manner of the embodiments of this application, are all permitted, agreed, authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant information, data and signals must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the action data of the virtual image involved in this application are all obtained with full authorization.
[0046] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.
[0048] Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with voice becoming one of the most promising methods of human-computer interaction. Key technologies in speech technology include automatic speech recognition (ASR), speech synthesis, and voiceprint recognition.
[0049] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.
[0050] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.
[0051] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0052] The solutions provided in the embodiments of this application involve artificial intelligence voice technology, NLP and machine learning, and specifically involve the application of the above various technologies or their combinations in the generation of virtual image movements, which will be explained in detail in the following embodiments.
[0053] The following describes the terms used in the embodiments of the present application.
[0054] Avatar: An object that can move in a virtual world. A vatar is a virtual, anthropomorphic digital image in the virtual world, such as a virtual person, anime character, or virtual character. The vatar can be a three-dimensional model, which can be a three-dimensional character built based on three-dimensional human skeleton technology. Optionally, the vatar can also be implemented using a 2.5D or 2D model, which is not limited in this embodiment of the present application. The 3D model of the vatar can be created using MMD (Miku Miku Dance, a three-dimensional computer graphics software) or the Unity engine. Of course, the 2D model of the vatar can also be created using Live2D (a two-dimensional computer graphics software). The dimensions of the vatar are not specifically limited here.
[0055] Metaverse: Also known as the metaverse, metaphysical universe, supersensory space, and virtual space, it is a network of 3D virtual worlds focused on social connections. The metaverse involves a persistent and decentralized online three-dimensional virtual environment.
[0056] Digital Human: A virtual image created by 3D modeling of the human body using information science methods, achieving a realistic simulation of the human body. Alternatively, a digital human is a digitized humanoid created using digital technology that closely resembles a human. Digital Humans are widely used in video creation, live broadcasts, industry reporting, social entertainment, and voice prompts. For example, digital Humans can serve as virtual anchors and avatars. Digital Humans are also referred to as virtual humans or virtual digital humans.
[0057] Virtual anchors (VATs) are hosts who use avatars to contribute content to video sites, such as virtual YouTubers (VTubers) and virtual uploaders (VUPs). Typically, VATs use their own personalities and images to engage in various forms of human-computer interaction, including broadcasting, performances, live broadcasts, and conversations.
[0058] The person behind the scenes refers to the person who performs or controls the virtual anchor behind the scenes during a live broadcast. For example, with the help of sensors installed on the head and limbs of the person behind the scenes, the body movements and facial expressions of the person behind the scenes are captured through an optical motion capture system, and the motion data is synchronized to the virtual anchor. In this way, with the help of the real-time motion capture mechanism, real-time interaction between the virtual anchor and the audience watching the live broadcast can be achieved.
[0059] Motion Capture (MoCap): Also known as motion capture. Refers to setting sensors on key parts of moving objects or real people, with the motion capture system capturing the sensor positions, and then obtaining motion data in three-dimensional space coordinates after computer processing. Once the motion data is recognized by the computer, it can be applied in animation production, gait analysis, biomechanics, ergonomics and other fields. Common motion capture equipment includes motion capture suits, which are mostly used in the generation of movements for 3D virtual images. Real people wear motion capture suits to perform movements, thereby migrating the 3D skeleton data of the human body captured by the motion capture system to the 3D model of the virtual image, obtaining the 3D skeleton data of the virtual image. This 3D skeleton data of the virtual image will be used to control the 3D model of the virtual image to perform the same movements as a real person.
[0060] Optical motion capture: An instrument used in the fields of engineering and technology related to information and systems science.
[0061] Inertial motion capture: Using inertial sensors, the movement of the main skeletal parts of the human body can be measured in real time, and then the position of the human joints can be calculated based on the principle of inverse kinematics, and the data can be applied to the corresponding (virtual image) bones.
[0062] Tokenization: refers to breaking down a given text into a data structure based on words (Tokens), where each word contains one or more characters.
[0063] Tokens: Tokenize a given text, breaking it down into a list of tokens. Each element in the token list is a token, and each token contains one or more characters. For example, tokenizing the text "I am very happy" yields a token list of {"I", "very", "happy"}.
[0064] Phoneme: The smallest unit of speech, defined by the natural properties of speech. Each phoneme is analyzed based on the pronunciation action within a syllable. For example, each character in a word may be divided into one or more phonemes based on the pronunciation action.
[0065] Phoneme alignment: Given an audio clip and a semantically corresponding text, the phonemes of each character in the text are split and aligned onto each audio frame on the audio timeline. Specifically, for each character in the text, one or more phonemes are determined based on the pronunciation of the character. Then, one or more audio frames that produce each phoneme are found in the audio. The audio segments comprise all the audio frames covered by all the phonemes required to produce the character. The timestamp interval of this audio segment on the audio timeline indicates the timestamp interval in the audio during which the speaker pronounced the character.
[0066] Frame interpolation is a motion estimation and compensation method that can extend the number of action frames in an action clip to make the action more coherent when the number of frames is insufficient. For example, a new action frame is inserted between every two action frames in the action clip, using the new action frame to fill in the intermediate state of the action change between the two previous action frames.
[0067] Text sentiment analysis: Given a piece of text, the process of analyzing, processing, summarizing, and reasoning about it typically outputs the sentiment label that best matches the text. This is also known as opinion mining or sentiment analysis. Depending on the granularity of the text processed, sentiment analysis can be broadly categorized into three levels: word level, sentence level, and paragraph level. Approaches to text sentiment analysis can be broadly grouped into four categories: keyword identification, lexical association, statistical methods, and concept-level techniques.
[0068] The following describes the technical concepts of the embodiments of the present application.
[0069] With the rapid development of technologies such as three-dimensional (3D) modeling, virtual reality (VR), augmented reality (AR), and the metaverse, virtual images are being used more and more widely in live broadcasting, film and television, animation, games, virtual social networking, human-computer interaction, and other aspects.
[0070] For example, in live streaming scenarios, where a virtual character acts as the host, delivering announcements or conversations, motion generation is required to enhance the rendering quality of the virtual character. Similarly, in video creation scenarios, such as creating videos for virtual hosts and digital human videos, motion generation is also required.
[0071] Typically, motion capture is used to generate the body movements of an avatar. A real person (or actor) wears a motion capture suit equipped with full-body sensors and performs movements according to the script content and audio. The motion capture suit captures the real person's motion data (i.e., 3D human skeleton data) and reports it to a computer connected to the motion capture suit. The computer then transfers the 3D human skeleton data to the avatar's 3D model, generating the avatar's 3D skeleton data. Since the avatar's 3D skeleton data at consecutive moments forms a sequence of movements, professional animators then perform some dithering or correction on the avatar's movement sequence to restore the avatar's movements, ultimately generating a series of avatar movements that match the script. This motion capture-based motion generation method requires manual intervention, and the captured 3D human skeleton data is customized to a specific script, non-reusable, and not universal. In other words, if a segment of audio or text not included in the script is encountered, motion generation becomes impossible, requiring the actor to perform using the new audio or text as a new script. This results in low motion generation efficiency.
[0072] Furthermore, when generating the avatar's body movements, it is also possible to capture video motions using a large amount of publicly available 2D video material (such as speech videos, talk show videos, etc.). The 2D video data is then converted into 3D skeletal data. A training dataset is constructed using the 3D skeletal data and its annotated audio and text to train a motion generation model. This allows the motion generation model to generate the avatar's body movements driven by audio. However, due to the relatively single data source and the high complexity of human movements, the effect of the motion generation model is not ideal. The resulting avatar has problems such as bland body movements and inaccurate performances, resulting in poor motion generation accuracy.
[0073] In view of this, the embodiment of the present application proposes a method for constructing an action library of a virtual image, which can divide the sample action sequence into sample action segments based on the sample action sequences of a large number of sample objects collected and their reference texts and reference audios, and then match the sample action segments to the action categories to which they belong, and then clean and filter the action data of the action sets of each action category, and finally establish a relatively complete action library of the virtual image, which can cover more action categories. Then, based on the established action library, an audio-triggered body action generation algorithm framework can be provided. When generating the virtual image action in real time, the user only needs to provide a piece of audio and its interpretation text, and the machine can quickly realize the generation of audio and text-triggered body action 3D data and output the action sequence of the virtual image. The entire action generation process does not require human intervention. The machine can quickly and accurately generate action sequences that match audio and text, and its action generation efficiency is high and the action generation accuracy is high.
[0074] If the semantic information of the text modality is not taken into account, and action clips are only searched for based on their similarity to the input audio to synthesize action sequences, then the final body movements will only simply change according to the audio rhythm, and will not be able to reflect the body movements at the real semantic level. Moreover, it can only simply repeat the dialogue action effects, and will not be able to show semantic accuracy and richness. This obviously results in poor action generation effects and poor virtual image simulation.
[0075] However, in the above technical solutions, since the dual-modal information of audio and text is taken into consideration to drive the generation of the virtual image's body movements, text and audio are combined, and the correlation between the two is considered, rich semantic actions are matched from the preset action library under the guidance of text semantics. This can make the virtual image's body movements in the synthesized action sequence more accurate, rich and vivid, and can be applied to various scenarios where virtual images need to perform actions, such as virtual live broadcasts, digital human videos, etc., to achieve the accuracy level of motion capture, but the efficiency of motion generation is far better than the motion capture method.
[0076] The following describes the implementation environment of the embodiments of the present application.
[0077] FIG1 is a schematic diagram of an implementation environment for a method for generating avatar motions according to an embodiment of the present application. Referring to FIG1 , the implementation environment includes a terminal 101 and a server 102. Terminal 101 and server 102 are connected directly or indirectly via a wireless or wired network, which is not a limitation of this application.
[0078] Terminal 101 has an application installed that supports virtual images. Terminal 101 can use this application to implement functions such as generating body movements of the virtual image. Of course, this application can also have other functions, such as online social functions, video sharing functions, video submission functions, or chat functions. Among them, this application is a native application in the operating system of terminal 101, or an application provided by a third party. For example, this application includes but is not limited to: live broadcast applications, short video applications, audio and video applications, game applications, social applications, 3D animation applications, or other applications, which are not limited in the embodiments of the present disclosure.
[0079] Optionally, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
[0080] Server 102 provides background services for applications supporting avatars on terminal 101. Server 102 creates and maintains a library of avatar actions and caches 3D skeletal models for various avatars. Server 102 comprises at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 102 performs the primary action generation and computation, while terminal 101 performs the secondary action generation and computation; alternatively, server 102 performs the secondary action generation and computation, while terminal 101 performs the primary action generation and computation; alternatively, server 102 and terminal 101 utilize a distributed computing architecture for collaborative action generation and computation.
[0081] Optionally, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0082] The terminal 101 may generally refer to one of a plurality of terminals, and the embodiment of the present disclosure is illustrated only by taking the terminal 101 as an example. Those skilled in the art will appreciate that the number of the terminals may be more or less.
[0083] In an exemplary scenario, during the real-time body movement generation process, the user uploads an audio clip in the application of the terminal 101, triggering an action generation instruction. The terminal 101 responds to the action generation instruction and sends an action generation request to the server 102, and the action generation request carries the audio. In response to the action generation request, the server 102 performs automatic speech recognition (ASR) on the audio to obtain a text indicating the semantics of the audio. Then, the audio and the text are used to execute the action generation method of the virtual image involved in the embodiment of the present application, and the appropriate action data is retrieved from the preset action library to synthesize an action sequence that matches the audio. In this way, the terminal 101 side realizes the generation of virtual image actions driven by audio, but the server 102 side uses the bimodal information of audio and text to synthesize the virtual image body movements that can represent the semantic level of audio (or text).
[0084] In another exemplary scenario, during the real-time body movement generation process, the user uploads a text in the application of terminal 101, triggering an action generation instruction. Terminal 101 responds to the action generation instruction and sends an action generation request to server 102, which carries the text. Server 102 responds to the action generation request, finds the sound source library of the virtual image, and generates an audio of the text read out from the sound source library (i.e., dubbing the text) for the text. Then, the audio and the text are used to execute the action generation method of the virtual image involved in the embodiment of the present application, retrieve appropriate action data from the preset action library, and then synthesize an action sequence that matches the text. In this way, the terminal 101 side realizes the text-driven virtual image action generation, but the server 102 side uses the bimodal information of audio and text to synthesize the virtual image body movement that can represent the semantic level of audio (or text).
[0085] In another exemplary scenario, during the real-time body movement generation process, the user uploads a piece of audio and its corresponding text (i.e., text representing the semantic information of the audio) in the application of terminal 101, triggering an action generation instruction. Terminal 101 responds to the action generation instruction and sends an action generation request to server 102. The action generation request carries the audio and the text. Server 102 responds to the action generation request and uses the audio and the text to execute the action generation method of the virtual image involved in the embodiment of the present application, retrieves appropriate action data from the preset action library, and then synthesizes an action sequence that matches the audio and the text. In this way, the terminal 101 side realizes the virtual image action generation driven by both audio and text, but the server 102 side uses the bimodal information of audio and text to synthesize the virtual image body movement that can represent the semantic level of audio (or text).
[0086] In all of the above scenarios, regardless of whether the terminal 101 side provides unimodal information or bimodal information as the driving signal, the server 102 side will use the conversion method between text and audio in voice technology to synthesize the virtual image's body movements using bimodal information of audio and text, so that the final action sequence can not only follow the rhythm of the audio rhythm, but also express rich semantic information at the semantic level, and even reflect the emotional state of the virtual image when broadcasting. Therefore, not only is the action generation efficiency high, but the action generation accuracy is also high. The generated body movements are well coordinated with the audio rhythm and carry rich semantic information, which greatly improves the simulation of the virtual image and greatly optimizes the rendering effect.
[0087] The avatar motion generation method provided by the embodiments of this application can be applied to any scenario requiring the generation of avatar body movements. For example, in a live broadcast scenario, the avatar does not need a motion capture suit to perform. Simply by providing at least one of the text or audio during the live interaction, the avatar can be controlled to perform body movements that complement the audio and subtitles (or possibly even without subtitles) during the live broadcast, enhancing the authenticity and fun of the live broadcast. For another example, in the scenario of creating a avatar video, the user only needs to create the audio or text for the video to control the generation of the avatar body movements that complement the audio or text. The body movements (i.e., the video images) and audio (i.e., the video dubbing) can then be combined into a avatar video, allowing for video submission and publishing. This improves the efficiency of avatar video generation and enhances the convenience and flexibility of its creation. For another example, the method can also be applied to various scenarios requiring the generation of avatar body movements, such as avatar customer service, animation production, film and television special effects, and avatar hosting. The embodiments of this application do not specifically limit the application scenarios.
[0088] The following describes the process of the method for generating the action of a virtual character according to an embodiment of the present application.
[0089] FIG2 is a flow chart of a method for generating an action of a virtual character provided by an embodiment of the present application. Referring to FIG2 , this embodiment is executed by a computer device, and is described using the computer device as a server as an example. The server may be server 102 in the above implementation environment. This embodiment includes the following steps.
[0090] 201. The server obtains audio and text of the virtual image, where the text indicates semantic information of the audio.
[0091] Among them, a virtual image refers to an object that can move in the virtual world. A virtual image is a virtual, personified digital image in the virtual world. For example, virtual images include but are not limited to: game characters, virtual anchors, virtual avatars, film and television characters, cartoon characters, digital humans, virtual humans, etc. The embodiments of this application do not specifically limit virtual images.
[0092] In the embodiment of the present application, when it is necessary to control the virtual image to broadcast audio, it is also necessary to control the virtual image to perform actions that match the audio, so the server will generate an action sequence for the virtual image.
[0093] The audio contains at least one audio frame, and the text indicates the semantic information of the audio. The text contains at least one word, and each word contains at least one character. The audio and text are associated with each other. The text is the semantic information identified by ASR on the audio, or the audio is the voice signal broadcasting the text. The voice signal can be a synthesized signal output by a machine or a human voice signal captured by a microphone. The type of voice signal is not specifically limited here.
[0094] In some embodiments, the server queries a pair of audio and text with an associated relationship from a local database, or the server takes out a piece of audio from the local database, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio, or the server takes out a piece of text from the local database, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0095] In other embodiments, the server downloads a pair of audio and text with an associated relationship from a cloud database, or the server downloads a piece of audio from a cloud database, performs ASR recognition on the audio, and obtains text indicating semantic information of the audio, or the server downloads a piece of text from a cloud database, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0096] In some other embodiments, the server receives a pair of audio and text with an associated relationship uploaded by the terminal. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain audio and text. Alternatively, the server receives the audio uploaded by the terminal, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain audio, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio. Alternatively, the server receives the text uploaded by the terminal, performs sound synthesis on the text, and obtains audio dubbing for the text. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain text, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0097] In the above process, since text and audio can be converted into each other, the user can provide only audio, only text, or both audio and text. In addition to user specification, it can also be read from a local database or downloaded from a cloud database. The embodiment of the present application does not specifically limit the source of the audio and the text.
[0098] After the server obtains the audio and text, it executes the method provided in the embodiment of the present application to generate an action sequence, and the action sequence matches the semantic information of the text. Then, in the process of controlling the virtual image to broadcast the audio, the virtual object is controlled to perform the physical movements indicated by the action sequence, so that the semantic information of the physical movements performed by the virtual object matches the broadcast audio.
[0099] 202. The server determines a semantic tag of the text based on the text, where the semantic tag represents part-of-speech information of a word in the text or sentiment information expressed by the text.
[0100] In some embodiments, the server analyzes the text obtained in step 201 to obtain at least one semantic tag of the text, wherein the semantic tag may include at least one of a part-of-speech tag or a sentiment tag, the part-of-speech tag represents the part-of-speech information of a word in the text, the part-of-speech information of a word refers to information used to describe the part of speech of a word, such as subject, verb, state, etc., the sentiment tag represents the sentiment information expressed by the text, and sentiment information refers to information used to describe the sentiment expressed by the text, such as happiness, loss, anger, etc. Both part-of-speech information and sentiment information describe the text, but from different angles. The embodiment of the present application does not specifically limit the content of the semantic tag. The number of the semantic tags can be one or more, and the embodiment of the present application does not specifically limit the number of semantic tags.
[0101] In some embodiments, the server determines at least one word contained in the text based on the text, determines a part-of-speech tag for each word, and uses the part-of-speech tags of all words in the text as the semantic tag of the text. The method for extracting part-of-speech tags will be described in detail in the next embodiment and will not be repeated here.
[0102] In other embodiments, the server determines at least one emotion tag of the text based on the text, and uses the at least one emotion tag as the semantic tag of the text. The method for extracting emotion tags will be described in detail in the next embodiment and will not be repeated here.
[0103] In some further embodiments, the server determines both the part-of-speech tag of each word based on the text and each sentiment tag of the text based on the text, and then uses each part-of-speech tag and each sentiment tag together as semantic tags for the text.
[0104] In an example, the text "My first live broadcast!" is segmented to obtain a word list {"I", "first time", "live broadcast!"}, among which the part-of-speech tag of the word "I" is "subject", the part-of-speech tag of the word "first time" is "state", and the part-of-speech tag of the word "live broadcast!" is "verb". In addition, based on the text, the emotional tag "happy" is determined, and finally four semantic tags will be output: "subject", "state", "verb", and "happy".
[0105] In the above process, by analyzing the given text, the feature information of the text at the semantic level can be extracted, and these feature information can be represented in a concise way in the form of semantic labels, which facilitates the use of semantic labels at the semantic level as guiding signals in the action generation process, and is conducive to the synthesis of virtual image body movements that are highly matched with the semantics of the audio and are smooth and natural.
[0106] 203. The server retrieves an action category matching the semantic tag and action data belonging to the action category from a preset action library, where the preset action library includes action data of the avatar belonging to multiple action categories.
[0107] In some embodiments, for each semantic tag obtained in step 202, the semantic tag is used as an index to retrieve an action category matching the semantic tag from multiple candidate categories in the preset action library, wherein the preset action library is an action database created and maintained on the server side, which is used to store an action set for each action category in units of action categories, and each action set contains action data clustered to this action category. The method for creating the preset action library will be described in detail in subsequent embodiments and will not be repeated here.
[0108] However, it should be noted that in one possible implementation, not every semantic tag can find a matching action category. If a semantic tag does not match any candidate categories, a preset action category can be used as the action category that matches the semantic tag to avoid gaps in the action sequence. The preset action category can be a default action category pre-configured by the technician, such as a standing action category without semantics, or a sitting action category. The preset action category is not specifically limited here, and the technician can configure different preset action categories for different virtual images.
[0109] In the above process, the semantic label of the text can be used as an index to retrieve the action category that best matches the audio at the semantic level in the preset action library. This action category does not simply move with the rhythm of the audio, but is highly adaptable to the semantic information of the audio, and can reflect the emotional tendency and potential semantics of the virtual image in the broadcast audio. In this way, the action data selected from this action category can be used to synthesize a more accurate action sequence for the virtual image.
[0110] After the action category matching the audio is retrieved, action data belonging to the action category is retrieved from the preset action library. The action data can control the virtual image to present a certain specific action.
[0111] 204. The server generates an action sequence for the virtual image based on the action data, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
[0112] In some embodiments, after each semantic tag is configured with an action category, the action data belonging to the action category can be retrieved from the preset action library for each semantic tag. For example, the action data can include multiple frames of 3D skeleton data at continuous moments (i.e., each frame of 3D skeleton data can be called an action frame), and each frame of 3D skeleton data at least includes the posture data of each skeleton key point in the action picture presented in this frame. In this way, it is only necessary to migrate each frame of 3D skeleton data to the 3D skeleton model of the virtual image to control the virtual image to present a certain specific action. Then, the action data matched to each semantic tag can be spliced according to the timestamp order of the words corresponding to each semantic tag in the audio to form an action sequence of the virtual image. This action sequence represents the changes in the body movements of the virtual image at continuous moments of the broadcast audio, and is used to control the virtual image to perform body movements that match the audio when broadcasting the audio.
[0113] In some embodiments, the phoneme alignment tool can find a time stamp interval corresponding to each semantic tag on the audio timeline. This time stamp interval refers to the time period when the avatar is reading the words belonging to this semantic tag. Then, from the action set of the action category in the preset action library, the action data that matches the semantic tag is retrieved, and then the action data is used to fill this time stamp interval in the action sequence. The action data in each connected time stamp interval will constitute the action sequence of the avatar at consecutive moments. The phoneme alignment method and the method of querying action data will be explained in detail in the next embodiment and will not be repeated here.
[0114] In the above process, each action frame in the final synthesized action sequence is aligned with the timestamp of an audio frame in the audio, so that the action frame reflects the body movements that match the audio frame at the semantic level, greatly improving the adaptability and accuracy of sound and picture, and avoiding mechanical and rigid visual effects. It can improve the simulation and anthropomorphism of the virtual image and optimize the rendering effect of the virtual image.
[0115] It should be noted that the embodiments of the present application do not limit the equipment and timing for controlling the virtual image to broadcast the audio and perform actions that match the audio. In some embodiments, the server controls the virtual image to broadcast the audio and perform actions that match the audio based on the action sequence. In other embodiments, the server sends the generated action sequence to the associated terminal, and the terminal controls the virtual image to broadcast the audio and perform actions that match the audio based on the action sequence. In addition, after the server generates the action sequence, it can immediately control the virtual image to broadcast the audio and perform actions that match the audio based on the action sequence, or first store the action sequence in association with the audio or text, and then control the virtual image to broadcast the audio and perform actions that match the audio based on the action sequence when a broadcast instruction is received.
[0116] All the above optional technical solutions can be combined in any way to form optional embodiments of the present disclosure, and will not be described in detail here.
[0117] The method provided in the embodiment of the present application uses audio and text as dual-modal driving signals, extracts semantic tags at the semantic level based on the text, and facilitates the retrieval of action categories matching the semantic tags in a preset action library. This action category can be highly adapted to the semantic information of the audio, reflecting the emotional tendency and potential semantics of the virtual image in the broadcast audio, and then retrieves the action data belonging to the action category. Based on the action data, a more accurate action sequence is quickly and efficiently synthesized for the virtual image, which not only improves the action generation efficiency of the virtual image, but also improves the action generation accuracy.
[0118] Furthermore, the action sequence can control the virtual image to make body movements that are coordinated with the audio on the semantic level, rather than simply following the rhythm of the audio. This greatly improves the adaptability and accuracy of sound and picture, and does not produce mechanical and rigid visual effects. It can enhance the simulation and anthropomorphism of the virtual image and optimize the rendering effect of the virtual image.
[0119] In the above embodiments, the process of the action generation scheme for the virtual image is briefly introduced, and a physical action generation framework triggered by audio and text is proposed. Since the virtual image will emit audio and perform physical actions when reading the text, there is a potential mapping relationship between the audio, text, and physical actions, and they can be aligned on the audio time axis. In the embodiments of the present application, this mapping relationship is mined. After obtaining the audio and its text, the semantic tags of the text are used to retrieve the action categories that match the audio at the semantic level from the preset action library, and then the action sequence of the virtual image is synthesized based on the action data belonging to the action category. The above action generation scheme can be applied to any physical action generation scenario of the virtual image, such as game characters, virtual anchors, film and television characters, cartoon characters, etc.
[0120] In this embodiment, the specific implementation of each step in the avatar action generation scheme is described in detail. Figure 3 is a flow chart of a method for generating avatar actions provided in this embodiment. Referring to Figure 3, this embodiment is executed by a computer device, using a server as an example. The server may be server 102 in the aforementioned implementation environment. This embodiment includes the following steps.
[0121] 301. The server obtains audio and text of the virtual image, where the text indicates semantic information of the audio.
[0122] The audio contains at least one audio frame, and the text indicates the semantic information of the audio. The text contains at least one word, and each word contains at least one character. The audio and text are associated with each other. The text is the semantic information identified by ASR on the audio, or the audio is the voice signal broadcasting the text. The voice signal can be a synthesized signal output by a machine or a human voice signal captured by a microphone. The type of voice signal is not specifically limited here.
[0123] In some embodiments, the server queries a pair of audio and text with an associated relationship from a local database, or the server takes out a piece of audio from the local database, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio, or the server takes out a piece of text from the local database, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0124] In other embodiments, the server downloads a pair of audio and text with an associated relationship from a cloud database, or the server downloads a piece of audio from a cloud database, performs ASR recognition on the audio, and obtains text indicating semantic information of the audio, or the server downloads a piece of text from a cloud database, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0125] In some other embodiments, the server receives a pair of audio and text with an associated relationship uploaded by the terminal. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain audio and text. Alternatively, the server receives the audio uploaded by the terminal, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain audio, performs ASR recognition on the audio, and obtains text indicating the semantic information of the audio. Alternatively, the server receives the text uploaded by the terminal, performs sound synthesis on the text, and obtains audio dubbing for the text. For example, the terminal sends an action generation request to the server, and the server receives and parses the action generation request to obtain text, performs sound synthesis on the text, and obtains audio dubbing for the text.
[0126] In the above process, since text and audio can be converted into each other, the user can provide only audio, only text, or both audio and text. In addition to user specification, it can also be read from a local database or downloaded from a cloud database. The embodiment of the present application does not specifically limit the source of the audio and the text.
[0127] In an exemplary scenario, as shown in Figure 4, Figure 4 is a schematic diagram of a method for generating an action of a virtual image provided in an embodiment of the present application. The user inputs audio and text on the terminal side, and the terminal uploads the input audio and text to the server. The server obtains audio 41 and text 42 "My first live broadcast!", where audio 41 is an audio file of the virtual image broadcasting text 42. Audio 41 can be an audio file in any form, such as a WAV file, an MP3 file, an MP4 file, etc.
[0128] After the server obtains the audio and text, it executes the method provided in the embodiment of the present application to generate an action sequence, and the action sequence matches the semantic information of the text. Then, in the process of controlling the virtual image to broadcast the audio, the virtual object is controlled to perform the physical movements indicated by the action sequence, so that the semantic information of the physical movements performed by the virtual object matches the broadcast audio.
[0129] 302. The server determines a sentiment tag of the text based on the text.
[0130] Among them, the emotion tag represents the emotional information expressed by the text, such as happiness, loss, anger, etc. The embodiment of the present application does not specifically limit the content of the emotion tag.
[0131] In some embodiments, a server pre-stores a plurality of candidate emotion tags, and configures a plurality of emotion keywords for each candidate emotion tag, stores a mapping relationship between emotion keywords and emotion tags, and then provides an emotion analysis method based on keyword matching. If the text contains any emotion keyword, then based on the mapping relationship, the emotion tag mapped to the emotion keyword can be queried, and the queried emotion tag is used as an emotion tag of the text; of course, if the text contains multiple emotion keywords, then the emotion tag mapped to each emotion keyword will be used as the emotion tag of the text. It should be noted that if multiple emotion keywords are mapped to the same emotion tag, then the emotion tag of the text also needs to be deduplicated.
[0132] The above sentiment analysis method based on keyword matching has a small amount of calculation and low computational complexity, and the sentiment analysis is fast and efficient.
[0133] In other embodiments, a plurality of candidate emotion tags are pre-stored in the server, and an emotion feature is configured for each candidate emotion tag. Then, text features are extracted from the entire text, and the feature similarity is calculated between the text features and the emotion features of each candidate emotion tag, and the emotion tag with the highest feature similarity is used as the emotion tag of the text.
[0134] Furthermore, considering that sometimes text readings need to be straightforward and free of emotional bias, technicians can also pre-configure a feature similarity threshold. If the feature similarity of all candidate emotion tags is less than the threshold, the emotion tag with the highest feature similarity will not be selected. In this case, the emotion tag will be left blank, or a default emotion tag of "no emotion" will be used as the emotion tag for the text. This can improve the recognition accuracy of emotion tags and ensure that inappropriate emotion tags are not added to emotionless text.
[0135] Furthermore, considering that the emotional composition of text broadcasts can sometimes be complex, with multiple emotions potentially coexisting, if a feature similarity threshold is pre-configured, emotion tags with feature similarity greater than the threshold can be used as the emotion tag for the text. This can further improve the recognition accuracy of emotion tags and provide better performance for text containing a mixture of multiple emotions.
[0136] The number of sentiment tags determined by the above feature similarity-based sentiment analysis method can be 0, 1, or more than 1, and there is no specific limit on the number of sentiment tags. Using feature space similarity to assess the sentiment of the entire text allows for higher accuracy in sentiment analysis compared to keyword matching. This is because some texts may not contain any sentiment keywords themselves, but may express a clear sentiment at the semantic level of the entire text. This can be detected by comparing feature similarity.
[0137] In some other embodiments, a sentiment analysis model is pre-trained in the server, and the text is input into the sentiment analysis model. The sentiment analysis model is used to calculate the matching probability between the text and each candidate sentiment tag. Then, the sentiment analysis model outputs one or more sentiment tags that match the text based on the matching probability between the text and each candidate sentiment tag. In this case, it is necessary to add a "no emotion" sentiment tag to the candidate sentiment tags to cover the recognition accuracy in the absence of emotion. Similarly, the technician can also pre-configure a probability threshold so that an emotion tag with the highest matching probability can be selected for output, where the probability threshold is a value greater than or equal to 0 and less than or equal to 1. Or all emotion tags with a matching probability greater than the probability threshold are output, or the top N (N≥1) emotion tags in the order of matching probability from large to small are output. This embodiment of the present application does not specifically limit this. Optionally, the sentiment analysis model can be a classification model, a decision tree, a deep neural network, a convolutional neural network, a multi-layer perceptron, etc. This embodiment of the present application does not specifically limit this.
[0138] The above sentiment analysis method based on the sentiment analysis model uses machine learning methods to learn the potential mapping relationship between text and sentiment tags, thereby judging the matching probability between the text and each candidate sentiment tag, which can improve the accuracy of sentiment analysis. The embodiment of this application does not specifically limit the sentiment analysis method.
[0139] It should be noted that step 302 is an optional step. If the emotional tag is not considered in the semantic tag, there is no need to perform emotional analysis on the text. The embodiment of the present application does not specifically limit whether text emotional analysis must be performed.
[0140] In an exemplary scenario, still taking Figure 4 as an example, through any of the above sentiment analysis methods, sentiment analysis is performed on the text 42 "My first live broadcast!", and the sentiment label "happy" of the text 42 is obtained, which means that the virtual image needs to be immersed in a happy mood when broadcasting the text 42.
[0141] 303. The server determines at least one word contained in the text based on the text.
[0142] In some embodiments, the server performs word segmentation on the text to obtain a word list of the text, where the word list is used to record at least one word contained in the text, and each word contains at least one character.
[0143] The word segmentation process can be implemented using a word segmentation tool. Different word segmentation tools can be used according to the language of the text. For example, for Chinese text, a Chinese word segmentation tool is used to perform word segmentation to obtain a word list of the Chinese text. For another example, for English text, an English word segmentation tool is used to perform word segmentation to obtain a word list of the English text. The embodiment of the present application does not specifically limit the language of the text, nor does it specifically limit the type of word segmentation tool.
[0144] In an exemplary scenario, still taking Figure 4 as an example, the text 42 "My first live broadcast!" is segmented to obtain a word list {"I", "first time", "live broadcast!"}, where the text 42 contains 3 words, the first word "I" contains 1 character, the second word "first time" contains 3 characters, and the third word "live broadcast!" contains 3 characters.
[0145] 304. The server queries the part-of-speech tag of each word from the part-of-speech table.
[0146] Among them, the part-of-speech tag represents the part-of-speech information of the words in the text, such as subject, verb, state, etc. The embodiment of the present application does not specifically limit the content of the part-of-speech tag.
[0147] In some embodiments, a part-of-speech table is pre-stored in the server, which records candidate part-of-speech tags. Then, for each word obtained by text segmentation, the part-of-speech table is queried, and the vector similarity between the word vector of the word and the tag vector of each part-of-speech tag is calculated, and the part-of-speech tag with the highest vector similarity is used as the part-of-speech tag to which the word belongs.
[0148] In an exemplary scenario, still taking Figure 4 as an example, the text 42 "My first live broadcast!" is segmented to obtain a word list {"I", "first time", "live broadcast!"}. Then, the part-of-speech table is queried to find that the part-of-speech tag of the first word "I" is "subject", the part-of-speech tag of the second word "first time" is "state", and the part-of-speech tag of the third word "live broadcast!" is "verb".
[0149] In steps 303-304 above, a possible implementation method for extracting the part-of-speech tag for each word in the text is provided. This method of querying the part-of-speech table has a small amount of computation, low computational complexity, and fast and efficient part-of-speech analysis. Of course, a part-of-speech analysis model can also be trained, and the text can be input into the part-of-speech analysis model, which will output a series of words and their corresponding part-of-speech tags. This part-of-speech analysis has a higher accuracy rate. This embodiment of the application does not specifically limit the part-of-speech analysis method.
[0150] In the above part-of-speech analysis process, because different body movements are usually made in response to words of different parts of speech when speaking, the different parts of speech will also affect the category or amplitude of the body movements. Considering the part-of-speech tag of each word can better reflect the implicit information of the text at the semantic level.
[0151] It should be noted that step 304 is an optional step. If the part-of-speech tag is not considered in the semantic tag, there is no need to perform part-of-speech analysis on the text (but word segmentation is still required, because only after word segmentation can it be convenient to align words, phonemes, and actions). The embodiment of the present application does not specifically limit whether text part-of-speech analysis must be performed.
[0152] 305. The server determines the sentiment tag and the part-of-speech tag to which the at least one word belongs as a semantic tag of the text.
[0153] The semantic tag represents the part-of-speech information of a word in the text or the sentiment information expressed by the text.
[0154] In some embodiments, the sentiment tag obtained in step 302 and the part-of-speech tag obtained in step 304 are determined as semantic tags of the text, where the number of the semantic tags can be one or more. The embodiment of the present application does not specifically limit the number of semantic tags.
[0155] In an exemplary scenario, still using Figure 4 as an example, text 42 "My first live broadcast!" is segmented to obtain a word list {"I", "first time", "live broadcast!"}. The part-of-speech table shows that the first word "I" has a part-of-speech tag of "subject", the second word "first time" has a part-of-speech tag of "state", and the third word "live broadcast!" has a part-of-speech tag of "verb". Furthermore, sentiment analysis is performed on text 42 to obtain the sentiment tag "happy". Finally, four semantic tags are output: "subject", "state", "verb", and "happy". The above process of analyzing text 42 and extracting semantic tags is called "audio text analysis".
[0156] In steps 302-305, a possible implementation for the server to determine the semantic tags for a text is described, using the example of a semantic tag that considers both part-of-speech tags and sentiment tags. By analyzing the given text, semantically distinct features can be extracted and concisely represented as semantic tags. This facilitates the use of semantic tags as guiding signals during action generation, thereby facilitating the synthesis of highly semantically aligned, smooth, and natural virtual avatar body movements.
[0157] It should be noted that the semantic tag can include at least one of a part-of-speech tag or a sentiment tag. If the semantic tag does not consider the part-of-speech tag, there is no need to execute step 304. If the semantic tag does not consider the sentiment tag, there is no need to execute step 302. The embodiment of the present application does not specifically limit the content of the semantic tag.
[0158] 306. For each word included in the text, the server determines, based on the phoneme associated with the word, the audio segment to which the phoneme belongs from the audio.
[0159] In some embodiments, for each word contained in the text in step 303, the phoneme associated with the word can be determined, wherein the phoneme associated with the word refers to the phoneme that needs to be pronounced to broadcast the word. There can be one or more phonemes associated with each word, and the embodiment of the present application does not specifically limit the number of phonemes. Then, at least one audio frame corresponding to the phoneme is found from the audio, and this at least one audio frame constitutes the audio segment to which the phoneme belongs. In this way, each word can be aligned with an audio segment in the audio through phoneme alignment, thereby aligning the word to the audio segment on the audio timeline.
[0160] In an exemplary scenario, still using Figure 4 as an example, after each word in text 42 is segmented in step 303, phoneme alignment can be performed. That is, the N (N ≥ 1) phonemes that pronounce the word are determined, and at least one audio frame (e.g., frames 2 to 37) in the audio 41 that emits these N phonemes is found. This at least one audio frame is used as the audio segment aligned with the word. The above process can be considered as the process of determining an aligned audio segment from the audio for each word in text 42.
[0161] It should be noted that this step 306 can be executed only after the word segmentation in step 303 is completed, and can be executed in parallel or serially with the extraction of emotion tags in step 302 and the extraction of part-of-speech tags in step 304. The embodiment of the present application does not limit the execution sequence between steps 302, 304 and 306.
[0162] 307. Based on the semantic tag of the word, the server retrieves an action category that matches the semantic tag and action data belonging to the action category from a preset action library.
[0163] The preset action library includes action data of the virtual image belonging to multiple action categories.
[0164] In some embodiments, each semantic tag obtained in step 305 is associated with a word in the text. For the part-of-speech tag in the semantic tag, the part-of-speech tag itself is obtained by querying the part-of-speech table in units of words. Therefore, there is a natural association between the part-of-speech tag and the word. Each word must belong to a part-of-speech tag, but different words may have the same part-of-speech tag; but for the sentiment tag in the semantic tag, since sentiment analysis is performed on the entire text, the context of the entire text can better judge its sentiment tendency, but it is also necessary to find a most matching word in the text for the sentiment tag. For example, if a sentiment analysis method based on keyword matching is used to determine the sentiment tag, the matched sentiment keyword (which must be a word in the text) is directly used as the most matching word for the sentiment tag. If a sentiment analysis method based on feature similarity or sentiment analysis model is used, then when the sentiment tag and the words obtained by each word segmentation are known, the vector similarity between the word vector of the sentiment tag and the word vector of each word is calculated in reverse, and the word with the highest vector similarity is used as the most matching word for the sentiment tag.
[0165] In the above manner, regardless of whether the semantic tag includes a part-of-speech tag or an emotion tag, a best-matching word can be found for each semantic tag. It should be noted that the same word may have one or more semantic tags. For example, still taking Figure 4 as an example, in text 42 "My first live broadcast!", the word "live broadcast!" has two semantic tags, one of which is the part-of-speech tag "verb", and the other is the emotion tag "happy". The embodiment of the present application does not specifically limit the number and type of semantic tags that each word has.
[0166] In this way, for each word in the text, after determining one or more semantic tags to which the word belongs, each semantic tag to which the word belongs is used as an index, and the action category that matches the semantic tag is queried from multiple candidate categories in the preset action library, so that the action data belonging to the action category can be queried.
[0167] The following will take steps A1 to A4 as an example to introduce a possible implementation method of querying action categories based on semantic labels. In this implementation method, whether the semantic label is similar to the candidate category is judged from the feature space.
[0168] A1. The server extracts the semantic features of each semantic tag.
[0169] In some embodiments, for each semantic tag belonging to each word in the text, the server extracts the semantic features of the semantic tag, for example, directly uses the word vector of the semantic tag as the semantic feature, or pre-trains a feature extraction model, inputs the semantic tag into the feature extraction model, processes the semantic tag through the feature extraction model, and outputs the semantic features of the semantic tag. The feature extraction model can be any NLP model. Furthermore, in order to improve the efficiency of feature extraction, the semantic features of all candidate part-of-speech tags and all emotional tags can be pre-extracted, and each part-of-speech tag or emotional tag can be associated with its own semantic features and stored. In this way, for each semantic tag, according to the tag ID (Identification) of the semantic tag, the semantic features stored in association with the tag ID can be directly and quickly queried. This is equivalent to offline calculation of the semantic features of each semantic tag. Only a small amount of query overhead is required in the online action generation stage, and there is no need to calculate the semantic features in real time, which can improve the efficiency of feature extraction.
[0170] In some embodiments, a Key-Value data structure is used to store the tag ID and its semantic features, where the tag ID is the Key (key name) and the semantic feature is the Value (key value). In the online query stage, the tag ID is used as an index to query whether any Key-Value data structure can be hit. If a Key-Value data structure can be hit, the semantic feature stored in the Value is taken out. This semantic feature is the semantic feature of the semantic tag indicated by the tag ID.
[0171] A2. The server queries the preset action library for category features of multiple candidate categories.
[0172] In some embodiments, a preset action library is created and maintained in the server, and the preset action library includes action data of the virtual image belonging to multiple action categories. The construction process of the action library will be introduced in detail in the next embodiment and will not be repeated here. A large amount of action data is stored in the preset action library. In order to facilitate retrieval, these action data are clustered according to the semantic level, thereby being divided into multiple action categories. Each action category has an action set, and the action set stores the action data clustered to the corresponding action category. In some embodiments, the action data can be implemented as multiple frames of 3D skeleton data of the virtual image at consecutive moments when performing the action of this action category.
[0173] Furthermore, all action categories in the preset action library constitute multiple candidate categories of the current semantic label. At this time, the server can calculate the category features for each candidate category. For example, the word vector of the candidate category is used as the category feature of the candidate category. For example, the feature extraction model used in step A1 is reused, and the candidate category is input into the feature extraction model. The candidate category is processed by the feature extraction model and the category feature of the candidate category is output. Here, only the feature extraction model in step A1 is reused as an example to illustrate. It can save the training overhead on the server side and there is no need to retrain a feature extraction model. It can also project the semantic label and action category into the same feature space. Of course, the server side can also train a semantic feature extraction model for the semantic label and a category feature extraction model for the action category, so that the extraction process of semantic features and category features is more targeted, thereby improving the expression ability of semantic features and category features respectively. The embodiments of the present application do not make specific limitations on this.
[0174] Furthermore, to improve feature extraction efficiency, a pre-trained feature extraction model can be used to extract the category features of all action categories (i.e., all candidate categories) in the preset action library. Each action category can then be associated and stored with its own category features. This allows for quick and direct querying of the category features stored in association with each candidate category based on its category ID during the online action generation phase. This is equivalent to calculating the category features of each candidate category offline, thus requiring only a small amount of query overhead for online queries and eliminating the need for real-time category feature calculations, improving feature extraction efficiency.
[0175] In some embodiments, a Key-Value data structure is used to store the category ID and its category features, where the category ID is the Key and the category feature is the Value. In the online query stage, the category ID is used as an index to query whether any Key-Value data structure can be hit. If a Key-Value data structure can be hit, the category feature stored in the Value is taken out. This category feature is the category feature of the candidate category indicated by the category ID.
[0176] It should be noted that in the embodiments of the present application, in order to distinguish the action categories matched by the semantic tags and the action categories used as candidates, the names of candidate categories and action categories are distinguished, that is, candidate categories and action categories are relative to semantic labels, but for the preset action library itself, all categories are action categories supported by the preset action library, and there is no concept of candidate categories.
[0177] A3. The server determines the action category from the multiple candidate categories, and the category feature of the action category meets the similarity condition with the semantic feature.
[0178] The similarity condition represents whether the semantic label is similar to the candidate category.
[0179] In some embodiments, for each semantic tag belonging to each word in the text, the server obtains the semantic features of the semantic tag from step A1, and obtains the category features of all candidate categories in the preset action library from step A2. Then, the feature similarity between the semantic feature and the category features of each candidate category is calculated, and among the multiple candidate categories, the candidate category whose feature similarity meets the similarity condition is selected as the action category that matches the semantic tag, that is, the category feature of the determined action category meets the similarity condition with the semantic feature. Among them, the feature similarity can be cosine similarity, the inverse of the Euclidean distance, etc., and the embodiments of the present application do not specifically limit this.
[0180] In some embodiments, the similarity condition is the highest feature similarity. In this case, it is only necessary to find the candidate category with the highest feature similarity from all candidate categories as the action category that matches the semantic label. This ensures that each semantic label can find an action category that is most similar at the semantic level, and there will be no situation where some semantic labels cannot match the action category. The action category screening process is relatively simple and computationally efficient.
[0181] In other embodiments, the similarity condition is that the feature similarity is greater than a preset similarity threshold, and the preset similarity threshold is a value greater than 0 pre-defined by a technician. If there is only one candidate category that meets the similarity condition, then the only candidate category is used as the action category that matches the semantic label. If there are more than one candidate categories that meet the similarity condition, then the candidate category with the largest feature similarity is selected as the action category that matches the semantic label. If there are 0 candidate categories that meet the similarity condition, that is, all candidate categories do not meet the similarity condition, then step A4 is entered. In this way, by configuring a preset similarity threshold, it is possible to take into account certain situations in which the emotions during certain broadcasts are relatively stable and do not contain specific obvious semantics. In this case, the virtual image is a relatively calm broadcast content and does not need to make body movements with certain semantics (if it does, it may appear exaggerated). In this case, the overall value of each feature similarity is actually low. If the preset similarity threshold is not configured, then the feature similarity with the largest relative value can be directly selected. If the preset similarity threshold is configured, a strategy will be provided in which all candidate categories do not match. At this time, step A4 is entered to directly configure the action category matching the semantic label to a preset action category without special semantics, such as a standing action category, a sitting action category, etc.
[0182] In the above steps A1 to A3, an implementation method is provided for judging whether a semantic label is similar to a candidate category from a feature space. In this way, for any semantic label, an action category that meets similarity conditions with the semantic label in the feature space can be found. By controlling the similarity conditions, it is possible to flexibly control whether to use preset action categories to fill in the situation where the semantic label is not very similar to all candidate categories, thereby improving the recognition efficiency of action categories and improving the controllability of action categories.
[0183] A4. When the category features of the multiple candidate categories and the semantic feature do not meet the similarity condition, the server configures the action category matching the semantic tag as a preset action category.
[0184] In some embodiments, not every semantic tag can necessarily find a matching action category through similarity conditions. If the semantic features of the semantic tag and the category features of all candidate categories do not meet the similarity conditions, it means that the semantic tag and all candidate categories do not match. In this case, a preset action category can be used as the action category that matches the semantic tag to avoid a period of vacancy in the action sequence. The preset action category can be a default action category pre-configured by the technician, such as a standing action category without semantics, or a sitting action category, etc. The preset action category is not specifically limited here, and the technician can also configure different preset action categories for different virtual images.
[0185] In the above steps A1 to A4, a possible implementation method is provided for selecting the action category of each semantic tag based on the semantic tag. The semantic tag of the text can be used as an index to find the action category that best matches the audio at the semantic level in the preset action library. This action category does not simply move with the rhythm of the audio, but is highly adaptable to the semantic information of the audio, and can reflect the emotional tendency and potential semantics of the virtual image in the broadcast audio. In this way, the action data selected from this action category can be used to synthesize a more accurate action sequence for the virtual image.
[0186] In other embodiments, in addition to steps A1 to A4, an action classification model can also be trained, and each semantic label is input into the action classification model. The action classification model is used to predict the matching probability between the semantic label and each candidate category, and the action category with the highest matching probability is output. In this way, it is only necessary to add the above-mentioned preset action categories to the candidate categories, and it can also cover scenarios where there is no need to make semantic body movements during broadcasting, which can further improve the recognition accuracy of action categories.
[0187] It should be noted that since each semantic tag is associated with a word, each word may have multiple semantic tags. To ensure a one-to-one correspondence between words and action categories, multiple matching action categories may exist for a word with multiple semantic tags. In this case, after finding an action category that matches each semantic tag, the action category that matches all of the word's semantic tags is prioritized as the final action category for the word. If no action category matches all of the word's semantic tags, the action category with higher feature similarity is prioritized, or the default action category is directly configured. For example, suppose a word has two semantic tags, a and b. Semantic tag a matches action categories 1 and 2, and semantic tag b matches action categories 1 and 3. Then action category 1 is directly selected as the final action category for the word. However, if semantic tag b matches action categories 3 and 4, the action category with the highest feature similarity is selected from action categories 1 to 4, or the default action category is directly configured as the final action category for the word.
[0188] In an exemplary scenario, still taking Figure 4 as an example, each semantic tag obtained in the audio text analysis stage is used as an index, and an action category matching the semantic tag is selected from the K (K≥2) action categories in the preset action library 43. For example, the three words in the text 42 are "I", "first time", and "live broadcast!". The first word "I" has only one semantic tag "subject", but the semantic tag "subject" does not find a matching action category in the preset action library 43, so it is configured as the preset action category "standing". The second word "first time" has only one semantic tag "state", but the semantic tag "state" finds a matching action category "cute shrug" in the preset action library 43. The third word "live broadcast!" has two semantic tags "verb" (part-of-speech tag) and "happy" (emotion tag), among which the semantic tags "verb" and "happy" jointly lock an action category "happy raising hands", that is, the action category "happy raising hands" matches both the two semantic tags "verb" and "happy", so it is selected as the action category that best matches the third word "live broadcast!".
[0189] The preset action library 43 is also referred to as a dynamic semantic preset action library containing a large amount of action data. This large amount of action data can be collected, publicly available, and legally compliant data of 3D action segments of avatars. For example, each 3D action segment contains multiple frames of 3D skeletal data at consecutive moments. The above process of retrieving action categories based on semantic tags is also referred to as retrieving key poses for each semantic tag (retrieving key actions).
[0190] Furthermore, since the data volume of the preset action library 43 may be very large, each action category can be further divided into multiple subcategories. For example, the action category "raising hands" is further divided into multiple subcategories: "raising one hand", "raising both hands", etc. In an example, as shown in Figure 4, action category 1 contains 10 subcategories, action category 2 contains 3 subcategories, action category 3 contains 6 subcategories... and so on, action category K contains 2 subcategories. There is no specific limitation on whether each action category is divided into subcategories.
[0191] It should be noted that for each semantic label, when each action category contains subcategories, it is also possible to find a subcategory that matches the semantic label from all subcategories of the determined action category by calculating feature similarity in the same way as steps A1 to A3, thereby further improving the degree of matching between the action data used in step 308 and the semantic label at the semantic level.
[0192] 308. The server generates an action segment that matches the audio segment based on the action data corresponding to the word.
[0193] In the above step 307, a unique corresponding action category can be found for each word, which can be summarized into the following situations: 1) The word has a semantic tag. If the semantic tag has an action category that meets the similarity conditions, then this action category is selected. If the semantic tag does not have an action category that meets the similarity conditions, then the preset action category is selected; 2) The word has multiple semantic tags. After each semantic tag selects an action category (including the preset action category) according to step 1), if there is an action category that matches all the semantic tags of the word at the same time, then the action category that matches at the same time is selected. If there are multiple action categories that match at the same time, then the action category that matches at the same time and has the highest feature similarity is selected. If there is no action category that matches all the semantic tags of the word at the same time, then the action category with the largest number of matching semantic tags is selected, or the action category with the highest feature similarity is selected, or the preset action category is selected. The embodiments of the present application do not specifically limit this.
[0194] On the basis of the above, each word has a one-to-one corresponding action category (including preset action categories). Then, according to the corresponding relationship of the audio timeline, each word can find an audio clip in step 306 and an action category in step 307. According to the action data belonging to the action category in the preset action library, an action clip can be synthesized for the word, thereby ensuring that the timestamps of the action clip and the audio clip are aligned and the semantic level is highly adapted.
[0195] Next, a possible action segment synthesis method will be introduced through steps B1 and B2. In this synthesis method, a one-to-one correspondence between audio frames and key action frames can be achieved, so that the timestamps of the two are aligned.
[0196] B1. The server determines at least one key action frame having the highest semantic matching degree with the word from the action data.
[0197] In some embodiments, since there is a one-to-one correspondence between words and action categories, for each word, the server retrieves the action data belonging to the action category corresponding to the word from the preset action library, and then filters the action data to obtain at least one key action frame with the highest semantic match with the word.
[0198] In some embodiments, each action category in the preset action library stores an action set, which is used to store action data belonging to the action category. For example, the action set contains multiple action clips, each action clip contains multiple action frames, and each action frame represents the position of each skeletal key point at a certain moment in the process of the virtual image performing a certain action under the action category, wherein each action clip has its annotated reference audio and reference text. Since in the library building stage, the words in the reference text, the phonemes in the reference audio, and the action frames in the action clips will also be timestamped. Therefore, when comparing the semantic matching degree between words and key action frames, you can first query whether there is any reference text of the action segment in the action set that contains the word. If a reference text is found, then at least one key action frame that matches the word (i.e., timestamp aligned) is directly taken out from the action segment corresponding to the reference text. If no reference text is found, then it is necessary to further calculate the vector similarity between the word vector of the current word and the word vector of each word in each reference text, find the reference text to which the approximate word (usually synonyms and / or near-synonyms) with the highest vector similarity belongs, and take out at least one key action frame that matches the approximate word (i.e., timestamp aligned) from the action segment corresponding to the found reference text.
[0199] In the above process, a possible implementation method for filtering key action frames from the action data of the action set is provided. This method of first detecting repeated words and then detecting similar words can ensure that only when repeated words are not found, similar words need to be detected, thereby reducing the computing overhead of the server. In other embodiments, when detecting similar words, it is also possible not to judge similar words based on vector similarity, but to directly obtain synonyms and / or near-synonyms for the word from the vocabulary first, and then use the synonyms and / or near-synonyms as indexes to query whether a reference text is hit. This also satisfies the requirement that synonyms and / or near-synonyms need to be queried only when repeated words are not found, and can also achieve the effect of saving computing overhead.
[0200] In some other embodiments, a situation is also examined. If each action category in the preset action library is further divided into multiple subcategories, then in the optional method of step 307, a subcategory matching the word can be found in the multiple subcategories of the action category. In this way, when retrieving key action frames, only the action data belonging to the selected subcategory need to be considered, and the action data belonging to the unselected subcategory do not need to be considered. This is equivalent to reducing the query range of the key action frame and further improving the query efficiency of the key action frame. Moreover, the key action frames that are usually queried in a small range, due to the dual matching of the action category (i.e., the major category) and the subcategory (i.e., the minor category), also improve the retrieval accuracy of the key action frame, and can better coordinate and cooperate with the word at the semantic level.
[0201] In other embodiments, only one standard action clip is saved for each action category from the preset action library. Then, it is only necessary to start from the median action frame in the middle of the action clip and sample at least one key action frame closest to the median action frame. In this way, the key action frame can be aligned to the middle of the pre-stored standard action clip. The actions / postures in the middle are often more standard and key.
[0202] B2. The server synthesizes at least one key action frame into an action segment that matches the audio segment based on the audio segment.
[0203] In some embodiments, for each word in the text, after the server queries at least one key action frame based on step B1, it can determine the frame number of the key action frame. In addition, it also determines the number of audio frames of the audio clip aligned with the timestamp of the word in step 306, and then compares the number of audio frames with the number of frames of the key action frame. Optionally, based on the number of audio frames, the key action frame is scaled by a certain proportion to ensure that the final synthesized action clip is the same length as the audio clip in step 306 (that is, the timestamp is aligned). In this case, the key action frame does not need to be cropped or modified, only the playback speed needs to be adjusted. Therefore, overall, more details of the key action frame can be retained, and the complete posture changes of the key action matched by the word can be presented as much as possible.
[0204] In other embodiments, if the difference between the number of audio frames and the number of frames of the key action frame is relatively large, simply adjusting the speed may cause the playback connection to be unsmooth, such as the virtual image suddenly moves slowly, or the virtual image suddenly moves quickly, which will obviously affect the smoothness and naturalness of the movement. Therefore, the embodiment of the present application also provides a method for inserting or cropping key action frames to improve the above-mentioned situations and optimize the smoothness and naturalness of the movement. Below, we will discuss the two situations, namely, situation one where the number of key action frames does not exceed the number of audio frames in the audio clip, and situation two where the number of key action frames exceeds the number of audio frames in the audio clip.
[0205] Case 1: The number of key action frames does not exceed the number of audio frames in the audio clip
[0206] In some embodiments, when the number of frames of the key action frame does not exceed the number of audio frames of the audio segment, the server may insert frames into the at least one key action frame to obtain the action segment having the same length as the audio segment.
[0207] In some embodiments, in order to ensure that the action segment and the audio segment are of equal length, at least one key action frame can be interpolated, for example, one or more intermediate action frames are inserted between any one or more pairs of adjacent key action frames, each intermediate action frame being intermediate action data calculated based on the pair of adjacent key action frames into which it is inserted.
[0208] In some embodiments, a linear interpolation method is used, and the calculation of intermediate action data is actually performed using linear interpolation. For example, i (i ≥ 1) intermediate action frames are inserted between key action frame 1 and key action frame 2. Taking the same skeletal key point of the left shoulder as an example, the skeletal key point is in position θ1 in key action frame 1 and in position θ2 in key action frame 2. Then, it is only necessary to calculate the i intermediate positions of the skeletal key point from position θ1 to position θ2 to obtain the i intermediate positions of the skeletal key point in the i intermediate action frames. Similarly, by calculating i intermediate positions for the skeletal key points of the entire body, it is possible to insert i intermediate action frames. In the linear interpolation method, the skeletal key points change uniformly in a fixed step size in the i intermediate action frames. It can also be considered that the skeletal key points move at a uniform speed. Therefore, it is only necessary to calculate the fixed step size when inserting i intermediate action frames based on the positions in the initial state and the final state (i.e., position θ1 and position θ2), and it is easy to calculate the i intermediate positions. The above linear interpolation method consumes less computing resources, has low computational overhead, fast action segment synthesis, and low waiting delay.
[0209] In other embodiments, a motion adjustment model is pre-trained, and the motion adjustment model is used to perform nonlinear interpolation on the key action frames when the number of frames of the key action frames is less than the number of audio frames in the audio clip. That is, the motion adjustment model is used to learn a nonlinear interpolation mode for the key action frames. This nonlinear interpolation mode may be fitted according to a certain motion curve, or it may be fitted according to the audio rhythm and the motion amplitude to learn the posture change law under this amplitude change. The specific nonlinear interpolation mode learned is determined by the input training samples. After the motion adjustment model is trained, at least one key action frame can be input into the motion adjustment model and controlled by the number of audio frames as a hyperparameter. Then, the motion adjustment model will output at least one intermediate action frame to be inserted, and each intermediate action frame inserted between two adjacent key action frames does not change uniformly according to a fixed step size, but performs non-uniform posture changes according to the nonlinear interpolation mode learned by the motion adjustment model. The embodiments of the present application do not specifically limit whether to use a linear interpolation method. The above nonlinear interpolation method based on the motion adjustment model can improve the mechanical feel that may be brought by the linear interpolation method and optimize the smoothness of the action clip.
[0210] In the above process, when the number of key action frames is less than the number of audio frames, the missing action frames can be supplemented by inserting key action frames, so that the intermediate motion states are supplemented between adjacent key action frames, so that the virtual image in the action clip will move more coherently.
[0211] Case 2: The number of key action frames exceeds the number of audio frames in the audio clip
[0212] In some embodiments, when the number of frames of the key action frame exceeds the number of audio frames, an action segment of the same length as the audio segment is created, and each frame of the action segment is filled with a preset action frame under a preset action category. The preset action category can be a default action category pre-configured by a technician, such as a standing action category without semantics, or a sitting action category, etc. The preset action category is not specifically limited here, and the technician can also configure different preset action categories for different virtual images. The preset action frame is a relatively static action frame pre-configured under the preset action category. For example, when the preset action category is a standing action category, the preset action frame is a standing action frame. When the preset action category is a sitting action category, the preset action frame is a sitting action frame. When the preset action category is maintained, the virtual image usually maintains the same action unchanged for multiple frames.
[0213] In the above process, when the number of key action frames exceeds the number of audio frames, these key action frames are discarded and the preset action frames are used to fill the action clip. This way, there is no need to play the key action frames at a high speed, which will damage the audience experience, and there will be no problems with a certain action clip.
[0214] In other embodiments, when the number of frames of the key action frame exceeds the number of audio frames, the key action frame can also be cropped, such as discarding a part of the key action frames at the beginning and end so that the number of frames of the key action frame after cropping does not exceed the number of audio frames. This avoids using preset action frames to fill an audio segment of a longer word, and the action generation effect is better, but it may destroy the integrity of the key action frame. In this case, it is necessary to improve it through the action smoothing operation in step 309. As for the cropping logic of the first and last key action frames, it can be configured by technical personnel, such as cropping according to the set number of frames, or cropping according to a set ratio. The embodiments of the present application do not specifically limit this.
[0215] Steps B1 and B2 provide a possible method for synthesizing action clips. This method achieves a one-to-one correspondence between audio frames and key action frames, aligning their timestamps. Even when the number of key action frames and audio frames doesn't match, inserting, cropping, or padding with pre-set action frames ensures smooth action clip synthesis and improves efficiency.
[0216] 309. The server generates an action sequence that matches the audio based on each action segment that matches the audio segment of each word. The action sequence is used to control the virtual image to perform actions that match the audio.
[0217] In some embodiments, for each word in the text, a unique corresponding audio clip can be found in step 306, and a unique corresponding action clip can be synthesized in step 308. Therefore, the audio clip in step 306 and the action clip in step 308 can achieve a one-to-one correspondence, with the words as the bridge, and their timestamps aligned. In this way, by simply splicing each action clip in sequence according to the timestamp order of each audio clip, an action sequence can be obtained, and each action clip in the action sequence is guaranteed to be highly compatible with an audio clip in the audio at the semantic level.
[0218] In other embodiments, the spliced action sequence may be smoothed to increase the naturalness and fluency of the connection between different action segments, which will be described in detail below through steps C1 to C2.
[0219] C1. The server splices each action segment that matches each audio segment based on the timestamp sequence of each audio segment to obtain a spliced action sequence.
[0220] In some embodiments, since the audio clips and action clips are linked by words, for each action clip, the timestamp interval of the corresponding audio clip can be found on the audio timeline. Then, each action clip can be spliced together in the order of the timestamp intervals to obtain a spliced action sequence. Alternatively, the spliced action sequence can be directly output to simplify the action synthesis process, or the action smoothing operation in step C2 can be performed to increase the naturalness and fluency of the transitions between different action clips.
[0221] C2. The server performs motion smoothing on each action frame in the spliced action sequence to obtain the action sequence.
[0222] In some embodiments, since some of the spliced action sequence obtained in step C1 may be key action frames, some may be interpolated intermediate action frames, and some may be filled preset action frames, each frame of action data in the spliced action sequence is referred to as an action frame. An action frame may be a key action frame, an intermediate action frame, or a preset action frame, which is not specifically limited in the embodiments of the present application. Next, motion smoothing is performed on each action frame in the spliced action sequence to obtain a final action sequence.
[0223] In some embodiments, a window smoothing method is used to perform global processing on each connected action frame to obtain a globally smoothed action sequence. Wherein, the window smoothing method refers to: taking the skeleton key point as the unit, determining the posture of the same skeleton key point in each action frame, so that a series of posture changes of the skeleton key point in the action sequence can be obtained, so that a posture change broken line can be fitted, and then the posture change broken line is smoothed by the moving window average smoothing algorithm to obtain a posture change curve, and then the posture of the skeleton key point in each action frame is sampled from the posture change curve according to the timestamp, so as to obtain the updated posture of the skeleton key point in each action frame. In this way, when the action category gap between two adjacent action segments is large, the window smoothing method can be used to make the connection between the two adjacent action segments more fluent, coherent and natural, generate an action sequence with better visual effects, and improve the accuracy of action synthesis.
[0224] In other embodiments, in addition to the window smoothing method, other smoothing algorithms can be used to smooth the posture change line, or a posture change curve can be directly machine-fitted on the posture change line, which can also achieve the effect of smooth movement.
[0225] In steps C1 and C2, the mechanically spliced action sequence is smoothed to create a smoother, more coherent, and more natural transition between adjacent action segments, generating a visually appealing action sequence and improving the accuracy of action synthesis. Alternatively, the spliced action sequence can be directly output without smoothing, simplifying the action synthesis process and improving its efficiency.
[0226] In an exemplary scenario, still taking Figure 4 as an example, for text 42 "My first live broadcast!", the first word "I" is matched to the preset action category "standing", the second word "first time" is matched to the action category "cute shrug", and the third word "live broadcast!" is matched to the action category "happy hand raising". Then, three action clips will be synthesized: a standing action clip, a cute shrug action clip and a happy hand raising action clip. The action clip synthesis method is detailed in step 308 and will not be repeated here. Then, the standing action clip, the cute shrug action clip and the happy hand raising action clip are spliced together to obtain a spliced action sequence, and the spliced action sequence is smoothed to obtain the final output action sequence. Figure 4 also outputs a smoothed posture change curve (i.e., action curve), which represents that the posture change curve of the skeletal key points in the output action sequence is relatively smooth and fluent, and can remove the mechanical feel of the limb movements. Optionally, the embodiment of the present application is suitable for synthesizing the body movements of a virtual image, but it is also necessary to combine the facial expressions of the virtual image to generate the final picture, and the picture and audio are combined to generate the final virtual image video (such as a digital human video).
[0227] In steps 306 to 309, a possible implementation method for generating an action sequence of the virtual image based on action data is provided. When there is a large amount of action data for each action category, the representative key action frames with the highest semantic matching degree can be selected, and then a series of action segments can be synthesized and then spliced into an action sequence. This action sequence represents the changes in the body movements of the virtual image at the continuous moments of the broadcast audio, and is used to control the virtual image to perform body movements that match the audio when broadcasting the audio.
[0228] In the above process, each action frame in the final synthesized action sequence is aligned with the timestamp of an audio frame in the audio, so that the action frame reflects the body movements that match the audio frame at the semantic level, greatly improving the adaptability and accuracy of sound and picture, and avoiding mechanical and rigid visual effects. It can improve the simulation and anthropomorphism of the virtual image and optimize the rendering effect of the virtual image.
[0229] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.
[0230] The method provided in the embodiment of the present application uses audio and text as dual-modal driving signals, extracts semantic tags at the semantic level based on the text, and facilitates the retrieval of action categories matching the semantic tags in a preset action library. This action category can be highly adapted to the semantic information of the audio, reflecting the emotional tendency and potential semantics of the virtual image in the broadcast audio, and then retrieves the action data belonging to the action category. Based on the action data, a more accurate action sequence is quickly and efficiently synthesized for the virtual image, which not only improves the action generation efficiency of the virtual image, but also improves the action generation accuracy.
[0231] Moreover, the action sequence can control the virtual image to make body movements that are coordinated with the audio at the semantic level, rather than simply following the rhythm of the audio. This greatly improves the adaptability and accuracy of the sound and picture, and does not produce mechanical and rigid visual effects. It can enhance the simulation and anthropomorphism of the virtual image and optimize the rendering effect of the virtual image.
[0232] In the above action generation scheme, the potential mapping relationship between the audio text and body movements is excavated, and an automated process of generating body movements of virtual images triggered by text and audio dual modes is realized. No human intervention is required, neither real-life performances combined with motion capture systems nor animation repair by animators is required. The machine can quickly and automatically generate the action sequence of the virtual image's body movements given text and audio, replacing the cumbersome motion capture and repair process. It has strong versatility and can be used for the task of generating body movements of virtual images in various scenarios such as games, live broadcasts, animations, and film and television. It has high practicality, and its equipment, manpower, and time costs are greatly reduced. The application is simple, fast, and non-dependent, and the generation of action sequences is high in quality and accuracy.
[0233] Each of the above embodiments details a virtual avatar action generation scheme that can quickly and automatically synthesize a highly semantically matched action sequence from a dual-modal audio and text drive signal without human intervention. The above action generation schemes rely on a pre-built preset action library, and the process of building this preset action library will be described in detail in the embodiments of this application.
[0234] FIG5 is a flowchart of a method for constructing an action library of a virtual character provided by an embodiment of the present application. Referring to FIG5 , this embodiment is executed by a computer device, and is described using a server as an example. The server may be server 102 in the aforementioned implementation environment. This embodiment includes the following steps.
[0235] 501. The server obtains a sample action sequence, reference audio, and reference text for each sample image. The reference text indicates semantic information of the reference audio. The sample action sequence is used to control the sample image to perform an action coordinated with the reference audio.
[0236] Among them, the sample image is a virtual image or real image that is public, collectible and collected in compliance with regulations. For example, the sample image is a virtual image such as an animated character, a virtual anchor, a digital human, etc., or it can be a real image such as an actor, a speaker, an anchor, etc. The embodiments of this application do not specifically limit this.
[0237] Among them, the collection and use of sample action sequences, reference audio and reference text of sample images are in compliance with regulations, and the sample action sequences have one-to-one corresponding reference audio (i.e. dubbing) and reference text (i.e. subtitles or text recognized by audio).
[0238] In some embodiments, the server obtains sample action sequences of multiple sample images, and eliminates low-quality samples that are neither labeled with reference audio nor with reference text. It can further eliminate low-quality samples that do not contain body movements (for example, only the head of the virtual image can be seen from the perspective), and further eliminate low-quality samples that are too short or too long in duration. For example, only sample action sequences with a duration of 1 to 10 seconds are retained. If the sample action sequence has both reference audio and reference text, the three are stored correspondingly. If the sample action sequence only has reference audio, then ASR is performed on the reference audio to obtain the corresponding reference text, and the three are stored correspondingly. If the sample action sequence only has reference text, then dubbing is performed on the reference text (i.e., speech synthesis is performed based on text) to obtain the corresponding reference audio, and the three are stored correspondingly. There is no specific limitation on the number of sample images and the number of sample action sequences here.
[0239] 502. The server divides the sample action sequence into a plurality of sample action segments based on the association between the words in the reference text and the phonemes in the reference audio, where each sample action segment is associated with a word in the reference text and a phoneme in the reference audio.
[0240] In some embodiments, the server processes each sample action sequence as a unit to obtain the reference text and reference audio stored corresponding to the sample action sequence. Furthermore, it can be seen from the previous embodiment that through the phoneme alignment method, the association relationship between the words in the reference text and the phonemes in the reference audio can be established. Then, based on the association relationship, the sample action sequence can be divided into multiple sample action segments.
[0241] In some embodiments, a possible method for dividing sample action segments is introduced through the following steps D1 to D2.
[0242] D1. For each word in the reference text, the server determines, based on the phoneme associated with the word, the sample audio segment associated with the phoneme from the sample audio.
[0243] The above step D1 is similar to step 306 in the previous embodiment and will not be described again here.
[0244] D2. The server divides the sample action sequence into multiple sample action segments based on the timestamp interval of each sample audio segment, and each sample action segment is aligned with the timestamp interval of a sample audio segment.
[0245] In some embodiments, for each sample audio segment, the start timestamp of the first audio frame and the end timestamp of the last audio frame in the sample audio segment can be found on the audio timeline. The start timestamp and the end timestamp constitute a timestamp interval. Since the reference audio, the reference text itself and the sample action sequence are timestamp-aligned, the sample action sequence can be directly divided according to the timestamp interval of each sample audio segment, which can be divided into multiple sample action segments and ensure that the timestamp interval of each sample action segment is aligned with the timestamp interval of the sample audio segment.
[0246] 503. The server clusters each sample action segment of each sample image based on the action features of the sample action segment to obtain multiple action sets, each action set indicating action data belonging to the same action category and belonging to different sample images.
[0247] In some embodiments, the server executes step 502 to divide each sample action sequence into multiple sample action segments, thereby obtaining a series of sample action segments from different sample images or different sample action sequences. Then, the action features of each sample action segment are extracted. Optionally, an action feature extraction model is trained, and the sample action segment is input into the action feature extraction model. The sample action features are processed by the action feature extraction model to output the action features of the sample action segment.
[0248] Furthermore, based on the action features extracted from each sample action clip, each sample action clip is clustered based on a clustering algorithm to form multiple action sets, each action set represents an action category, and each action set contains action data belonging to the corresponding action category (that is, each sample action clip clustered into this action category), wherein the clustering algorithm includes but is not limited to: KNN (K-Nearest Neighbor) clustering algorithm, K-means clustering algorithm, hierarchical clustering algorithm, etc.
[0249] In an exemplary scenario, taking the K-means clustering algorithm as an example, the clustering process of sample action segments is described. The K-means clustering algorithm is an iterative clustering analysis algorithm, and its steps are as follows: all sample action segments are divided into K action categories. First, randomly select K sample action segments as the initial clustering centers of each of the K action categories. Then, calculate the distances between each of the remaining sample action segments and the K initial clustering centers (in fact, calculate the distances between action features), and assign each of the remaining sample action segments to the clustering center closest to it. The clustering centers and the remaining sample action segments assigned to them represent an action set. Each time a new sample action segment is assigned to the action set, the clustering center of the action set will be recalculated based on all the existing sample action segments. The above process will be continuously repeated until a certain termination condition is met. The termination conditions include but are not limited to: no (or the minimum number of) sample action segments are reassigned to different action sets, no (or the minimum number of) clustering centers change anymore, or the sum of squared errors of K-means clustering is locally minimized, etc. The embodiments of the present application do not specifically limit the termination conditions of the K-means clustering algorithm.
[0250] In an exemplary scenario, as shown in FIG. 6, FIG. 6 is a schematic diagram of a method for creating an action library provided by an embodiment of the present application. Taking a single sample action sequence as an example, the reference text 61 and reference audio 62 of the sample action sequence are obtained. For example, the reference text 61 is "The first day of getting to know everyone, happy". Then, using a word segmentation tool, the reference text 61 is segmented into 5 words: "get to know", "everyone", "of", "the first day", "happy". Then, using a part-of-speech table, the part-of-speech tags of each word are queried. For example, the part-of-speech tag of "get to know" is "v (verb)", and the part-of-speech tag of "the first day" is "TIME (time)", etc. Then, using a phoneme alignment tool, the start frame number and end frame number of the sample audio segment aligned with each word are identified. For example, the sample audio segment of "get to know" is frames 2 to 37. Through the above operations, a quadruple [word, start frame number, end frame number, part-of-speech tag] can be constructed for each word. For example, the quadruple of the word "get to know" is ['get to know', 2, 37, 'v']. By splicing the quadruples of each word, a part-of-speech sequence can be obtained. Then, according to the sample audio segments of each word, the sample action sequence is divided into 4 sample action segments. Among them, since the sample action segment of the word "of" has too short duration, the sample action segments of the words "everyone" and "of" are combined into one sample action segment, and the words, sample audio segments, and sample action segments are aligned in time stamps. Then, after dividing each sample action sequence into multiple sample action segments in the above manner, each sample action segment is input into a clustering algorithm to obtain the action sets of each of the K action categories, where K is an integer greater than or equal to 2.
[0251] In other embodiments, the action set of each action category can be further subdivided into multiple subcategories in a similar manner. The clustering process of the subcategories is the same as the clustering process of the action categories, and will not be repeated here. By clustering, massive action data can be divided into multiple action categories, and it is ensured that the action data within each action category has a certain similarity, while the action data between different action categories have a certain difference. In this way, it is considered that each action category can represent an action semantics, that is, the action data belonging to different action categories are different from each other at the semantic level.
[0252] 504. The server constructs an action library based on the multiple action sets.
[0253] In some embodiments, the server directly constructs an action library based on the K action sets clustered in step 504. The action library includes K action sets, that is, action data of the avatar belonging to K action categories.
[0254] In some embodiments, the category features of the action categories to which the K action sets belong are also calculated and stored, thereby simplifying the creation process of the action library and accelerating the efficiency of the action library.
[0255] In other embodiments, further data cleaning can be performed on the K action sets clustered in step 504 to filter out outlier samples that deviate from the cluster center in each action set, thereby increasing the similarity of each sample action segment within the same action category and reducing the similarity of each sample action segment within different action categories. The following describes the data cleaning process for a single action set, taking steps E1 to E4 as an example.
[0256] E1. The server obtains, for each action set, a category feature of the action category indicated by the action set. The category feature is the average action feature of each sample action segment in the action set.
[0257] In some embodiments, for each action set clustered in step 504, an average action feature is calculated based on the action features of each sample action segment in the action set as the category feature of the action category indicated by the action set. This category feature represents the cluster center of the action set.
[0258] E2. The server determines a contribution score of the action feature of each sample action segment in the action set to the category feature, where the contribution score represents the degree of matching between the sample action segment and the action category.
[0259] Among them, although each sample action clip belongs to an action category, the degree of matching between different sample action clips and action categories may be different. The degree of matching is used to measure whether the action performed by the sample action clip is standard. For example, the category feature of the action category is the average action feature of multiple sample action clips in the action set, but the action features of some sample action clips in the action set are more similar to the average action feature, indicating that the action performed by the sample action clip belonging to the action category is more standard, while the action features of some sample action clips are not very similar to the average action feature, indicating that although the sample action clip also performs an action belonging to the action category, the action performed is not standard enough. Therefore, the contribution score represents the standardization of the sample action clip relative to the action category.
[0260] In some embodiments, for each sample action segment in the action set, the contribution score of the action feature of the sample action segment to the category feature in step E1 is calculated. Optionally, the feature similarity between the action feature and the category feature is directly calculated, and then the feature similarity of each sample action segment in the entire action set is exponentially normalized to obtain the contribution score of each sample action segment (referring to the feature similarity after exponential normalization). In this way, by using the feature similarity after exponential normalization as a measurement indicator of the contribution score, the calculation complexity of the contribution score can be reduced and the calculation efficiency of the contribution score can be improved.
[0261] In other embodiments, a contribution score measurement indicator based on the intra-class variance after excluding the individual itself (also called N-1 variance) is provided. In this way, the intra-class variance characterizes the contribution of the excluded individual to the entire cluster, that is, it reflects the contribution score of the excluded sample action segment to the entire action set. The contribution score has better performance and more accurate measurement dimensions. When the contribution score is larger, the action of the sample action segment is more standard. When the contribution score is smaller, the action of the sample action segment is less standard. Below, the method of obtaining the intra-class variance (i.e., a possible contribution score) of a single sample action segment will be described in detail through steps E21 to E22.
[0262] E21. The server obtains, for any sample action segment in the action set, an action score of each remaining action segment except the sample action segment, where the action score represents the degree of similarity between the remaining action segments and the category feature.
[0263] The remaining action segments refer to sample action segments in the action set other than the sample action segment.
[0264] In some embodiments, for each sample action segment in the action set, the feature similarity between the action feature of the sample action segment and the category feature in step E1 is calculated, and then the feature similarity of each sample action segment in the entire action set is exponentially normalized to obtain an action score for each sample action segment (referring to the feature similarity after exponential normalization). Then, the current sample action segment is excluded, and the action score of each remaining action segment other than the sample action segment is determined.
[0265] E22. The server determines the intra-class variance after excluding the sample action segment based on the action score of each remaining action segment, and determines the intra-class variance as the contribution score of the sample action segment.
[0266] In some embodiments, the server calculates the average of the action scores of all remaining action clips obtained in step E21, uses the average as an average action score, and then determines the intra-class variance after excluding the sample action clip based on the average action score and the action score of each remaining action clip, and determines the intra-class variance as the contribution score of the sample action clip.
[0267] In an example, assuming that the action set contains N sample action segments, excluding the Nth sample action segment, the remaining action segments refer to the 1st to the N-1th sample action segments. The intra-class variance (also called N-1 variance) is obtained by the following formula:
[0268] Among them, S N-1 Characterizes the intra-class variance of the Nth sample action segment, i is an integer greater than or equal to 1 and less than or equal to N-1, x i Characterizes the action score of the i-th sample action clip, Represents the average action score, where the average action score refers to the average of the action scores of the remaining N-1 action segments.
[0269] In the above process, a measurement indicator of the contribution score is provided based on the intra-class variance after excluding the individual itself (also called N-1 variance). In fact, after excluding a specified individual, the intra-class variance of the remaining individuals is calculated. The larger the intra-class variance, the smaller the influence of the excluded individual on the deviation from the cluster, and the greater the influence of the remaining individuals on the deviation from the cluster. Therefore, the intra-class variance can well measure the contribution of the excluded individual to the entire cluster, that is, it reflects the contribution score of the excluded sample action segment to the entire action set. The performance of its contribution score is better and the measurement dimension is more accurate. When the contribution score is larger, the action of the sample action segment is more standard. When the contribution score is smaller, the action of the sample action segment is more non-standard. Then it is necessary to consider eliminating non-standard sample action segments (that is, sample action segments with low contribution scores). This facilitates data cleaning within each action category.
[0270] E3. The server removes sample action segments whose contribution scores meet the removal criteria from the action set.
[0271] In some embodiments, the server can sort each sample action segment in the action set in descending order of contribution scores and remove the sample action segment at the bottom of the sort. In this way, each data cleaning process only discards the sample action segments that have the least impact on cluster deviation, thus avoiding the accidental deletion of high-quality sample action segments.
[0272] In other embodiments, the server can also sort each sample action segment in the action set in descending order of contribution score, and remove the sample action segments that are in the last j positions in the sorting. In this way, each data cleaning will discard the j sample action segments that have a smaller impact on cluster deviation. By flexibly controlling the value of j, the data cleaning rate of the action set can be finely controlled. Where j is an integer greater than or equal to 1.
[0273] In an example, as shown in Figure 7, Figure 7 is a data cleaning principle diagram of an action set provided by an embodiment of the present application. For an action set of a certain action category, after excluding the first sample action segment, the intra-class variance of the remaining N-1 action segments is calculated, and the contribution score of the first sample action segment is obtained as 0.2. The above operation is repeated for each sample action segment to calculate the contribution score of each sample action segment. Then, each sample action segment is sorted in descending order of the contribution score. Then, the sample action segment at the end of the sort is eliminated, for example, the sample action segment at the end with a contribution score of 0.02 is eliminated.
[0274] E4. The server updates the category feature and the contribution score based on the eliminated action set, iterates the elimination operation multiple times, and stops the iteration when the iteration stop condition is met.
[0275] In some embodiments, since one (or more) sample action segments with low contribution scores are eliminated in step E3, the cluster center, i.e., the category feature, must be recalculated due to the change in the number of samples in the action set. Therefore, the category feature is updated in a similar manner to step E1. Correspondingly, due to the change in the category feature, the contribution score of each sample action segment must also be recalculated. Therefore, the contribution score is updated in a similar manner to step E2. Then, based on the similar manner to step E3, the sample action segments whose contribution scores meet the elimination criteria are further eliminated according to the updated contribution scores. Steps E1 to E3 are iteratively executed until the iteration stop condition is met, thereby obtaining a relatively pure and high-quality action set. The iteration stop condition includes but is not limited to: the number of iterations reaches the number threshold, which is an integer greater than 0; or, the sample capacity of the action set is reduced to a preset capacity, which is an integer greater than or equal to 1; or, the contribution score of the last ranked segment is greater than the contribution threshold, which is a value greater than or equal to 0. The embodiment of the present application does not specifically limit the iteration stop condition.
[0276] In the above steps E1 to E4, since the action set directly formed by clustering is relatively rough, there may be some actions with large intra-class differences. Such actions need to be eliminated to avoid affecting the clustering accuracy of the action category, thereby providing a method for data cleaning, data filtering, or data purification for each action set. Finally, the action library constructed based on the cleaned action set has better action generation effect and higher usability. The entire iterative sorting and screening process can be self-supervised without manual intervention. Therefore, the database construction stage can also be automated, with low database construction cost and high database construction efficiency.
[0277] Steps 501-504 above detail the process for building an action library to support the avatar's action generation solution. In some embodiments, given that the action library cannot remain static, it is often necessary to expand or add new action data. The following describes the process for adding a new action sequence to the library, using steps F1-F4 as an example.
[0278] F1. The server obtains the newly added reference audio and newly added reference text associated with any newly added action sequence outside the action library.
[0279] The newly added reference text indicates semantic information of the newly added reference audio, and the newly added action sequence is used to control the corresponding sample image to perform an action coordinated with the newly added reference audio.
[0280] Step F1 is similar to step 501 and will not be described again here.
[0281] F2. The server divides the newly added action sequence into multiple newly added action segments based on the association between the words in the newly added reference text and the phonemes in the newly added reference audio.
[0282] Each newly added action segment is associated with a word in the newly added reference text and a phoneme in the newly added reference audio.
[0283] Step F2 is similar to step 502 and will not be described again here.
[0284] F3. The server determines, for each newly added action segment, based on the action features of the newly added action segment and from multiple action sets in the action library, a target action set to which the newly added action segment belongs.
[0285] In some embodiments, for each newly added action segment, the action features of the newly added action segment are calculated based on the same method as step 503, and then the distance between the action features of the newly added action segment and the category features of each action set is calculated, and the newly added action segment is assigned to the target action set that is closest to it.
[0286] F4. The server adds the newly added action segment to the target action set, updates the category feature and the contribution score, and removes sample action segments whose contribution scores meet the removal criteria from the target action set.
[0287] In some embodiments, after the newly added action segment is assigned to the target action set, due to the change in the number of samples in the target action set, its cluster center, i.e., the category feature, must be recalculated. Therefore, the category feature is recalculated in the same manner as step E1. Correspondingly, due to the change in category feature, the contribution score of each sample action segment (including the newly added action segment) must also be recalculated. Therefore, based on the same manner as step E2, the contribution score is recalculated. Then, based on the same manner as step E3, according to the calculated new contribution score, the sample action segments whose contribution scores meet the elimination criteria are continued to be eliminated.
[0288] In an example, as shown in Figure 8, Figure 8 is a data supplement principle diagram of a new action clip provided in an embodiment of the present application. For the target action set into which the new action clip falls, assuming that two new action clips are added, the contribution scores of the two new action clips are calculated to be 0.7 and 0.04 respectively based on the same method as step E2, then these two new action clips are included, and each sample action clip in the entire target action set is reordered (in reverse order) according to the contribution score, and the sample action clip at the last position after reordering is eliminated, for example, the last new action clip with a contribution score of 0.04 is eliminated.
[0289] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.
[0290] The method provided in the embodiment of the present application divides the sample action sequence into a series of sample action segments according to the guidance of reference text and reference audio, and then uses a clustering method to divide the sample action segments into multiple action categories. Each action category has an action set to store the action data clustered into this action category. In this way, a complete action library with multiple action categories can be constructed, so that the action data belonging to different action categories can be distinguished at the semantic level, which is convenient for subsequent input into the action generation process. The semantic label is used as an index to detect the most matching action category, thereby improving the efficiency and accuracy of action generation.
[0291] The above action library establishment solution provides a mechanism for automated learning semantic production, automatic classification, and automatic screening, which facilitates the automated removal of low-quality samples and allows the addition of new samples to any action category at any time. It only requires the use of contribution scores to re-clean the action data in the action category, ensuring the high quality of the action library and improving the uniformity of action data in each action category.
[0292] FIG9 is a schematic diagram of the structure of a device for generating a virtual image's motion according to an embodiment of the present application. As shown in FIG9 , the device includes:
[0293] An acquisition module 901 is configured to acquire audio and text of the avatar, wherein the text indicates semantic information of the audio;
[0294] An analysis module 902 is configured to determine a semantic tag of the text based on the text, where the semantic tag represents at least one of part-of-speech information of a word in the text or sentiment information expressed by the text;
[0295] A retrieval module 903 is configured to retrieve an action category that matches the semantic tag and action data belonging to the action category from a preset action library, wherein the preset action library includes action data of the avatar belonging to multiple action categories;
[0296] The generation module 904 is used to generate an action sequence for the virtual image based on the action data, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
[0297] The device provided in the embodiment of the present application uses audio and text as dual-modal driving signals, extracts semantic tags at the semantic level based on the text, and facilitates the retrieval of action categories matching the semantic tags in a preset action library. This action category can be highly adapted to the semantic information of the audio, reflecting the emotional tendency and potential semantics of the virtual image in the broadcast audio, and then retrieves the action data belonging to the action category. Based on the action data, a more accurate action sequence is quickly and efficiently synthesized for the virtual image, which not only improves the action generation efficiency of the virtual image, but also improves the action generation accuracy.
[0298] Furthermore, the action sequence can control the virtual image to make body movements that are coordinated with the audio on the semantic level, rather than simply following the rhythm of the audio. This greatly improves the adaptability and accuracy of sound and picture, and does not produce mechanical and rigid visual effects. It can enhance the simulation and anthropomorphism of the virtual image and optimize the rendering effect of the virtual image.
[0299] In some embodiments, the analysis module 902 is used to: determine the sentiment tag of the text based on the text; determine at least one word contained in the text based on the text; query the part-of-speech tag of each word from the part-of-speech table; and determine the sentiment tag and the part-of-speech tag of the at least one word as the semantic tag of the text.
[0300] In some embodiments, the retrieval module is used to retrieve, for each word contained in the text: based on the semantic tag to which the word belongs, an action category matching the semantic tag from the preset action library; and retrieve action data belonging to the action category from the preset action library.
[0301] In some embodiments, based on the apparatus composition of FIG9 , the generating module 904 includes:
[0302] a determining unit configured to determine, for each word included in the text, based on a phoneme associated with the word, an audio segment to which the phoneme belongs from the audio;
[0303] a segment generating unit, configured to generate an action segment matching the audio segment based on the action data corresponding to the word and the audio segment;
[0304] The sequence generating unit is configured to generate an action sequence matching the audio based on each action segment matching the audio segment of each word.
[0305] In some embodiments, based on the apparatus composition of FIG9 , the fragment generation unit includes:
[0306] a determination subunit, configured to determine, from the action data, the at least one key action frame having the highest semantic matching degree with the word;
[0307] The synthesis subunit is configured to synthesize the at least one key action frame into the action segment matching the audio segment based on the audio segment.
[0308] In some embodiments, the synthesis sub-unit is used to: when the number of frames of the key action frame does not exceed the number of audio frames of the audio clip, insert at least one key action frame to obtain the action clip with the same length as the audio clip; when the number of frames of the key action frame exceeds the number of audio frames, create an action clip with the same length as the audio clip, and fill each frame of the action clip with a preset action frame under a preset action category.
[0309] In some embodiments, the sequence generation unit is used to: splice each action segment matching each audio segment based on the timestamp order of each audio segment to obtain a spliced action sequence; and perform action smoothing on each action frame in the spliced action sequence to obtain the action sequence.
[0310] In some embodiments, the retrieval module 903 is used to: extract the semantic features of each semantic tag; query the category features of multiple candidate categories in the preset action library; determine the action category from the multiple candidate categories, and the category features of the action category meet the similarity conditions with the semantic features.
[0311] In some embodiments, the retrieval module 903 is further configured to: when none of the category features of the multiple candidate categories and the semantic feature meet the similarity condition, configure the action category matching the semantic tag as a preset action category.
[0312] All the above optional technical solutions can be combined in any way to form optional embodiments of the present disclosure, and will not be described in detail here.
[0313] It should be noted that the aforementioned embodiments of the avatar motion generation device, when generating avatar body movements, illustrate the division of the aforementioned functional modules only as an example. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the computer device can be divided into different functional modules to perform all or part of the functions described above. Furthermore, the avatar motion generation device and the avatar motion generation method embodiments provided in the aforementioned embodiments share the same concept. The specific implementation process is detailed in the avatar motion generation method embodiments and will not be further elaborated here.
[0314] FIG10 is a schematic diagram of a structure of a device for constructing a virtual character action library according to an embodiment of the present application. As shown in FIG10 , the device includes:
[0315] The sample acquisition module 1001 is used to acquire a sample action sequence, reference audio, and reference text for each sample image, wherein the reference text indicates semantic information of the reference audio, and the sample action sequence is used to control the sample image to perform an action corresponding to the reference audio;
[0316] a segment division module 1002 for dividing the sample action sequence into a plurality of sample action segments based on associations between words in the reference text and phonemes in the reference audio, each sample action segment being associated with a word in the reference text and a phoneme in the reference audio;
[0317] A clustering module 1003 is configured to cluster each sample action segment of each sample image based on the action features of the sample action segment to obtain a plurality of action sets, each action set indicating action data of different sample images clustered into the same action category;
[0318] The construction module 1004 is configured to construct an action library based on the multiple action sets.
[0319] The device provided in the embodiment of the present application divides the sample action sequence into a series of sample action segments according to the guidance of reference text and reference audio, and then uses a clustering method to divide the sample action segments into multiple action categories. Each action category has an action set to store all action data clustered into this action category. In this way, a complete action library with multiple action categories can be constructed, so that the action data belonging to different action categories can be distinguished at the semantic level, which is convenient for subsequent input into the action generation process. The semantic label is used as an index to detect the most matching action category, thereby improving the efficiency and accuracy of action generation.
[0320] In some embodiments, the segment division module 1002 is used to: for each word in the reference text, based on the phoneme associated with the word, determine the sample audio segment associated with the phoneme from the sample audio; based on the timestamp interval of each sample audio segment, divide the sample action sequence into multiple sample action segments, each sample action segment is aligned with the timestamp interval of a sample audio segment.
[0321] In some embodiments, based on the device composition of FIG10 , the device further includes:
[0322] A feature acquisition module is used to obtain, for each action set, a category feature of the action category indicated by the action set, where the category feature is the average action feature of each sample action segment in the action set;
[0323] a determination module, configured to determine a contribution score of the action feature of each sample action segment in the action set to the feature of the category, the contribution score representing the degree of matching between the sample action segment and the action category;
[0324] A removal module is used to remove sample action segments whose contribution scores meet the removal conditions from the action set;
[0325] The iteration module is used to update the category feature and the contribution score based on the action set after elimination, iterate the elimination operation multiple times, and stop the iteration when the iteration stopping condition is met.
[0326] In some embodiments, the determination module is used to: obtain the action score of each remaining action fragment except the sample action fragment in any sample action fragment in the action set, and the action score represents the degree of similarity between the remaining action fragment and the category feature; based on the action score of each remaining action fragment, determine the intra-class variance after excluding the sample action fragment, and determine the intra-class variance as the contribution score of the sample action fragment.
[0327] In some embodiments, the elimination module is used to sort each sample action segment in the action set in descending order of contribution scores, and eliminate the sample action segment at the bottom of the sorting.
[0328] In some embodiments, the sample acquisition module 1001 is further configured to: for any newly added action sequence outside the action library, acquire a newly added reference audio and a newly added reference text associated with the newly added action sequence;
[0329] The segment division module 1002 is further configured to: divide the newly added action sequence into a plurality of newly added action segments based on the association between the words in the newly added reference text and the phonemes in the newly added reference audio;
[0330] The clustering module 1003 is further configured to: for each newly added action segment, determine, based on the action features of the newly added action segment, the target action set to which the newly added action segment belongs from the multiple action sets in the action library;
[0331] The construction module 1004 is further configured to: add the newly added action segment to the target action set, update the category feature and the contribution score, and remove sample action segments whose contribution scores meet the removal criteria from the target action set.
[0332] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.
[0333] It should be noted that the apparatus for constructing a virtual character's action library provided in the above embodiment only uses the division of the aforementioned functional modules as an example when constructing the action library. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for constructing a virtual character's action library provided in the above embodiment and the embodiment of the method for constructing a virtual character's action library are based on the same concept. The specific implementation process is detailed in the embodiment of the method for constructing a virtual character's action library, and will not be repeated here.
[0334] FIG11 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. As shown in FIG11 , the computer device 1100 may have relatively large differences due to different configurations or performances. The computer device 1100 includes one or more processors (Central Processing Units, CPUs) 1101 and one or more memories 1102, wherein the memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the methods for generating the action of a virtual image or the methods for constructing the action library of a virtual image provided in the above-mentioned various embodiments. Optionally, the computer device 1100 also has components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The computer device 1100 also includes other components for implementing the functions of the device, which will not be described in detail here.
[0335] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program. The at least one computer program can be executed by a processor in a computer device to implement the methods for generating avatar actions or constructing an avatar action library in the above-described embodiments. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device.
[0336] In an exemplary embodiment, a computer program product is also provided, comprising one or more computer programs stored in a computer-readable storage medium. One or more processors of a computer device can read the one or more computer programs from the computer-readable storage medium and execute the one or more computer programs, thereby enabling the computer device to perform the method for generating an action of an avatar or the method for constructing an action library of an avatar in the above-described embodiments.
[0337] Those skilled in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. Optionally, the program is stored in a computer-readable storage medium. Optionally, the above-mentioned storage medium is a read-only memory, a disk or an optical disk, etc.
[0338] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A method for generating an action of a virtual image, applied to a computer device, the method comprising: Acquire audio and text of the avatar, the text indicating semantic information of the audio; Based on the text, determining a semantic tag of the text, wherein the semantic tag represents at least one of part-of-speech information of a word in the text or sentiment information expressed by the text; Retrieving an action category matching the semantic tag and action data belonging to the action category from a preset action library, wherein the preset action library includes action data of the avatar belonging to multiple action categories; Based on the action data, an action sequence of the virtual image is generated, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
2. The method according to claim 1, wherein: Determining the semantic tag of the text based on the text includes: Based on the text, determining a sentiment tag of the text; Based on the text, determining at least one word contained in the text; Query the part-of-speech tag of each of the words from the part-of-speech table; The sentiment tag and the part-of-speech tag to which the at least one word belongs are determined as semantic tags of the text.
3. The method according to claim 1, wherein: The retrieving, from a preset action library, an action category matching the semantic tag and action data belonging to the action category, comprises: For each word contained in the text: Based on the semantic tag to which the word belongs, searching the preset action library for an action category matching the semantic tag; The action data belonging to the action category is retrieved from the preset action library.
4. The method according to claim 3, wherein: The step of generating an action sequence of the virtual image based on the action data comprises: For each word included in the text: based on the phoneme associated with the word, determining from the audio the audio segment to which the phoneme belongs, and, based on the action data corresponding to the word and the audio segment, generating an action segment matching the audio segment; Based on each action segment matched with the audio segment of each word, the action sequence matched with the audio is generated.
5. The method according to claim 4, wherein: The step of generating an action segment matching the audio segment based on the action data corresponding to the word and the audio segment comprises: Determining at least one key action frame having the highest semantic matching degree with the word from the action data; Based on the audio segment, the at least one key action frame is synthesized into the action segment matching the audio segment.
6. The method according to claim 5, wherein: The synthesizing, based on the audio segment, the at least one key action frame into the action segment matching the audio segment comprises: When the number of the key action frames does not exceed the number of audio frames of the audio segment, inserting the at least one key action frame to obtain the action segment having the same length as the audio segment; When the number of the key action frames exceeds the number of the audio frames, an action segment having the same length as the audio segment is created, and each frame of the action segment is filled with a preset action frame under a preset action category.
7. The method according to claim 4, wherein: For each action segment matched with the audio segment of each word, generating the action sequence matched with the audio segment comprises: Based on the timestamp sequence of each audio clip, each action clip matching each audio clip is spliced to obtain a spliced action sequence; Perform motion smoothing on each action frame in the spliced action sequence to obtain the action sequence.
8. The method according to claim 1, wherein: The retrieving the action category matching the semantic tag and the action data belonging to the action category from the preset action library comprises: Extracting semantic features of the semantic tags; Querying the category features of multiple candidate categories in the preset action library; The action category is determined from the multiple candidate categories, and the category feature of the action category meets a similarity condition with the semantic feature.
9. The method according to claim 8, wherein: The method further comprises: In a case where the category features of the plurality of candidate categories and the semantic features do not meet the similarity condition, the action category matching the semantic label is configured as a preset action category.
10. A method for constructing an action library of a virtual image, applied to a computer device, the method comprising: Acquire a sample action sequence, a reference audio, and a reference text of each sample image, wherein the reference text indicates semantic information of the reference audio, and the sample action sequence is used to control the sample image to perform an action coordinated with the reference audio; Based on the association relationship between the words in the reference text and the phonemes in the reference audio, the sample action sequence is divided into a plurality of sample action segments, each sample action segment being associated with a word in the reference text and a phoneme in the reference audio; Based on the action features of the sample action clips, clustering each sample action clip of each sample image to obtain a plurality of action sets, each action set indicating action data belonging to the same action category and belonging to different sample images; An action library is constructed based on the multiple action sets.
11. The method according to claim 10, wherein: The step of dividing the sample action sequence into a plurality of sample action segments based on the association between the words in the reference text and the phonemes in the reference audio comprises: For each word in the reference text, based on the phoneme associated with the word, determine from the sample audio the sample audio segment associated with the phoneme; Based on the timestamp interval of each sample audio segment, the sample action sequence is divided into a plurality of sample action segments, and each sample action segment is aligned with the timestamp interval of a sample audio segment.
12. The method according to claim 10, wherein: The method further comprises: For each action set, obtaining a category feature of an action category indicated by the action set, wherein the category feature is an average action feature of each sample action segment in the action set; Determine a contribution score of an action feature of each sample action segment in the action set to the category feature, wherein the contribution score represents a matching degree between the sample action segment and the action category; Eliminating, from the action set, sample action segments whose contribution scores meet the elimination condition; Based on the action set after elimination, the category feature and the contribution score are updated, the elimination operation is iterated multiple times, and the iteration is stopped when the iteration stopping condition is met.
13. The method according to claim 12, wherein: Determining the contribution score of the action feature of each sample action segment in the action set to the category feature includes: For any sample action segment in the action set, obtaining an action score of each remaining action segment except the sample action segment, wherein the action score represents a degree of similarity between the remaining action segments and the category feature; Based on the action score of each remaining action segment, the intra-class variance after excluding the sample action segment is determined, and the intra-class variance is determined as the contribution score of the sample action segment.
14. The method according to claim 12, wherein: The sample action segments whose contribution scores meet the elimination conditions are eliminated from the action set, and the obtained action set includes: Each sample action segment in the action set is sorted in descending order of contribution scores, and the sample action segment at the bottom of the sorting is removed.
15. The method according to claim 12, wherein: The method further comprises: For any newly added action sequence outside the preset action library, obtaining newly added reference audio and newly added reference text associated with the newly added action sequence; Based on the association between the words in the newly added reference text and the phonemes in the newly added reference audio, the newly added action The sequence is divided into multiple additional action segments; For each newly added action segment, based on the action features of the newly added action segment, determining the target action set to which the newly added action segment belongs from the plurality of action sets in the preset action library; The newly added action segment is added to the target action set, the category feature and the contribution score are updated, and the sample action segments whose contribution scores meet the elimination condition are eliminated from the target action set.
16. A device for generating a motion of a virtual image, the device comprising: An acquisition module, used to acquire audio and text of the avatar, wherein the text indicates semantic information of the audio; An analysis module, configured to determine a semantic tag of the text based on the text, wherein the semantic tag represents at least one of part-of-speech information of a word in the text or sentiment information expressed by the text; A retrieval module, used to retrieve an action category matching the semantic tag and action data belonging to the action category from a preset action library, wherein the preset action library includes action data of the avatar belonging to multiple action categories; A generation module is used to generate an action sequence for the virtual image based on the action data, and the action sequence is used to control the virtual image to perform actions coordinated with the audio.
17. A device for constructing an action library of a virtual image, the device comprising: A sample acquisition module, used to acquire a sample action sequence, a reference audio and a reference text of each sample image, wherein the reference text indicates semantic information of the reference audio, and the sample action sequence is used to control the sample image to perform an action in coordination with the reference audio; A segment division module, configured to divide the sample action sequence into a plurality of sample action segments based on the association relationship between the words in the reference text and the phonemes in the reference audio, each sample action segment being associated with a word in the reference text and a phoneme in the reference audio; A clustering module, for clustering each sample action segment of each sample image based on the action features of the sample action segments, to obtain a plurality of action sets, each action set indicating action data belonging to the same action category and belonging to different sample images; A construction module is used to construct an action library based on the multiple action sets.
18. A computer device, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the method for generating a virtual image action as described in any one of claims 1 to 9; or, the method for constructing a virtual image action library as described in any one of claims 10 to 15.
19. A computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the method for generating a virtual image action as described in any one of claims 1 to 9; or the method for constructing a virtual image action library as described in any one of claims 10 to 15.
20. A computer program product, comprising at least one computer program, wherein the at least one computer program is loaded and executed by a processor to implement the method for generating a virtual image action as described in any one of claims 1 to 9; or the method for constructing a virtual image action library as described in any one of claims 10 to 15.
Citation Information
Patent Citations
Virtual image generation method and device, storage medium and electronic equipment
CN112650831A
Facial information generation method and device
CN114513678A
Action generation method and device, electronic equipment and storage medium
CN114911973A
Method for generating role expression animation based on artificial intelligence semantic analysis
CN115147521A
Virtual image action generation method and device and action library construction method and device
CN116958342A
Cited By
Motion video generation method, related device and medium
CN119229218A
Action video generation method, related device and medium
CN119229218B
Method for synthesizing video, electronic device and computer program product
CN119323625A
