A method and system for generating virtual character animations

Abstract

The embodiment of the present specification provides a virtual character animation generation method, which comprises: acquiring a music frame sequence, each music frame in the music frame sequence comprising music frame information; generating a corresponding bone action sequence based on the music frame sequence through a motion generation model, and generating a virtual character animation based on the bone action sequence and a corresponding bone model.

Description

Technical Field

[0001] This specification relates to the field of animation technology, and in particular to a method and system for generating virtual character animations. Background Technology

[0002] Virtual characters generally refer to virtual images appearing in games, animations, and other works. With the development of 5G technology and image processing technology, virtual characters have been widely used in various fields such as live streaming and instant messaging services. Virtual character animations include a large number of virtual character movements, and in some scenes, virtual characters need to perform corresponding actions according to music, such as dancing to music or performing character fighting moves. All of this makes generating virtual character animations quite complex (for example, modelers need to first bind motion capture equipment to real people and generate virtual character animations by capturing the real people's movements, which requires a lot of complex manual operation) and makes it difficult to generate high-quality virtual character animations.

[0003] Therefore, there is an urgent need for methods and systems for generating virtual character animations to achieve the automatic and efficient generation of high-quality virtual character animations. Summary of the Invention

[0004] One embodiment of this specification provides a method for generating virtual character animation. The method includes: acquiring a music frame sequence, wherein each music frame in the music frame sequence includes music frame information; generating a corresponding skeletal motion sequence based on the music frame sequence using a motion generation model; and generating a virtual character animation based on the skeletal motion sequence and the corresponding skeletal model.

[0005] In some embodiments, the music frame information includes one or more of the following: beat information, volume information, and audio information of the music frame.

[0006] In some embodiments, the skeletal motions in the skeletal motion sequence include the skeletal position and skeletal rotation parameters of at least one bone corresponding to a music frame.

[0007] In some embodiments, generating the corresponding skeletal motion sequence based on the music frame sequence includes: generating a music frame encoding vector sequence based on the music frame sequence; and generating the skeletal motion sequence based on the music frame encoding vector sequence.

[0008] In some embodiments, generating the skeletal motion sequence based on the music frame encoding vector sequence includes: obtaining the starting skeletal motion and the stopping skeletal motion of the music frame sequence; and obtaining the skeletal motion sequence based on the music frame encoding vector sequence, the starting skeletal motion, and the stopping skeletal motion.

[0009] In some embodiments, the motion generation model includes an encoding network, a bidirectional recurrent neural network, and a decoding network; generating a corresponding skeletal motion sequence based on the music frame sequence includes: generating a music frame encoding vector sequence based on the music frame sequence using the encoding network; obtaining an initial skeletal motion sequence based on the music frame encoding vector sequence, the starting skeletal motion, and the stopping skeletal motion using the bidirectional recurrent neural network; and obtaining the skeletal motion sequence based on the music frame encoding vector sequence and the initial skeletal motion sequence using the decoding network.

[0010] In some embodiments, generating a virtual character animation based on the skeletal motion sequence and the corresponding skeletal model includes: obtaining at least one fitted skeletal motion between two skeletal motions in the skeletal motion sequence using a data fitting method; obtaining a skeletal motion animation based on the skeletal motion sequence and the fitted skeletal motion; and generating the virtual character animation based on the skeletal motion animation and the skeletal model.

[0011] In some embodiments, the method for obtaining the action generation model includes: acquiring at least one training sample and an initial action generation model, wherein the training sample includes music samples and their corresponding action samples; and iteratively updating the parameters of the initial action generation model based on the at least one training sample to obtain the action generation model.

[0012] One embodiment of this specification provides a virtual character animation generation system, the system comprising: an acquisition module for acquiring a music frame sequence, wherein each music frame in the music frame sequence includes music frame information; an action generation module for generating a corresponding skeletal action sequence based on the music frame sequence using an action generation model; and an animation generation module for generating virtual character animation based on the skeletal action sequence and the corresponding skeletal model.

[0013] One embodiment of this specification provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes the virtual character animation generation method provided in this embodiment. Attached Figure Description

[0014] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0015] Figure 1 These are schematic diagrams illustrating application scenarios of the virtual character animation generation system according to some embodiments of this specification;

[0016] Figure 2 This is a schematic diagram of the processing device according to some embodiments of this specification;

[0017] Figure 3 This is an exemplary flowchart of a virtual character animation generation method according to some embodiments of this specification;

[0018] Figure 4 This is an exemplary structural block diagram of a virtual character animation generation model shown in some embodiments of this specification;

[0019] Figure 5 This is a structured flowchart of a virtual character animation generation model training method according to some embodiments of this specification;

[0020] Figure 6 This is a structural schematic diagram of key nodes of a skeletal model shown in some embodiments of this specification. Detailed Implementation

[0021] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0022] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0023] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0024] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0025] Figure 1 These are exemplary schematic diagrams illustrating application scenarios of a virtual character animation generation system according to some embodiments of this specification.

[0026] like Figure 1 As shown, in the application scenario, it may include server 110, network 120, terminal device 130 and storage device 140.

[0027] The Virtual Character Animation Generation System 100 is primarily used for generating animations of virtual characters. A virtual character can refer to a virtual image appearing in games, animations, or other works. In some embodiments, a virtual character can be a two-dimensional character animation and / or a three-dimensional character model. For example, a virtual character can be a two-dimensional virtual live-streaming image generated using Live2D technology. Another example is a three-dimensional virtual image generated using 3D modeling technology. Virtual character animation refers to the continuous movements of a virtual character, and its presentation corresponds to the type of virtual character. For example, when the virtual character is a two-dimensional image, virtual character animation can refer to a sequence of image frames of the virtual character. When the virtual character is a three-dimensional image, virtual character animation can refer to the continuous movements of a three-dimensional model. The Virtual Character Animation Generation System 100 has been widely used in various virtual character production scenarios, such as animated dramas, movie animations, virtual idol live streams, or 3D games. In some embodiments, the Virtual Character Animation Generation System 100 can generate animations of virtual characters based on music by implementing the virtual character animation generation methods and / or processes disclosed in this specification.

[0028] Server 110 may include processing device 112 for performing virtual character animation generation methods. In some embodiments, server 110 may be local or remote. For example, server 110 may locally connect to terminal device 130 to obtain information and / or data sent by terminal device 130. As another example, server 110 may remotely receive information and / or data sent by terminal device 130 via network 120. In some embodiments, server 110 may be implemented on a cloud platform. By way of example only, cloud platforms may include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, internal clouds, multi-tiered clouds, etc., or any combination thereof.

[0029] Network 120 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the virtual character animation generation system 100 (e.g., server 110, terminal device 130, storage device 140) can transmit information to other components of the virtual character animation generation system 100 via network 120. For example, terminal device 130 can send music files to server 110 via network 120 to generate virtual character animations. As another example, server 110 can store virtual character animations. In some embodiments, network 120 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 120 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, etc. In some embodiments, network 120 may include one or more network access points. For example, network 120 may include wired or wireless network access points, such as base stations and / or network switching points 120-1, 120-2, ..., through which one or more components of the access point system can be connected to network 120 to exchange data and / or information.

[0030] Terminal device 130 can be used for human-computer interaction in the virtual character animation generation system 100. In some embodiments, terminal device 130 can be a portable device with data acquisition, storage, and / or transmission functions, such as tablet computer 130-1, smartphone 130-2, laptop computer 130-3, other mobile terminal 130-4, etc., or any combination thereof. In some embodiments, terminal device 130 may include one or more processing engines (e.g., a single-core processing engine or a multi-core processor). By way of example only, processing engines may include one or more combinations of central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), controller, microcontroller unit, reduced instruction set computer (RISC), microprocessor, etc. In some embodiments, an operator can input music through terminal device 130 or select music from a music library so that processing device 112 can generate corresponding virtual character animations based on the music. In some embodiments, the terminal device 130 can acquire virtual character animations and display the acquired virtual character animations to users or viewers.

[0031] Storage device 140 can store data and / or instructions. In some embodiments, storage device 140 can store data obtained / acquired by terminal device 130 and / or server 110, such as music, skeletal models, motion generation models, virtual character animations, etc. In some embodiments, storage device 140 can store data and / or instructions used by server 110 to perform or use in order to accomplish the exemplary methods described herein. In some embodiments, storage device 140 may include mass storage, removable storage, volatile read-write storage, read-only storage (ROM), etc., or any combination thereof. Exemplary mass storage may include disks, optical disks, solid-state drives, etc. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tapes, etc. Exemplary volatile read-only storage may include random access memory (RAM). Exemplary RAM may include dynamic RAM (DRAM), double-rate synchronous dynamic RAM (DDRSDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero-capacitance RAM (Z-RAM), etc. Exemplary ROMs may include mask ROMs (MROMs), programmable ROMs (PROMs), erasable programmable ROMs (PEROMs), electronically erasable programmable ROMs (EEPROMs), optical disc ROMs (CD-ROMs), and digital universal disk ROMs, etc. In some embodiments, the storage device 140 may be implemented on a cloud platform. By way of example only, a cloud platform may include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, internal clouds, multi-tiered clouds, etc., or any combination thereof.

[0032] In some embodiments, storage device 140 may be connected to network 120 to communicate with one or more components of the virtual character animation generation system 100 (e.g., server 110, terminal device 130). One or more components of image processing 100 may access data or instructions stored in storage device 140 via network 120. In some embodiments, storage device 140 may be directly connected to or communicate with one or more components of the virtual character animation generation system 100 (e.g., server 110, terminal device 130, etc.). In some embodiments, storage device 140 may be part of server 110.

[0033] Figure 2 This is a block diagram of a virtual character animation generation system according to some embodiments of this specification. The virtual character animation generation system 200 can be deployed on the processing device 112. The virtual character animation generation system 200 may include an acquisition module 210, an action generation module 220, and an animation generation module 230.

[0034] The acquisition module 210 can be used to acquire a music frame sequence, wherein each music frame in the music frame sequence includes music frame information.

[0035] In some embodiments, the acquisition module 210 can automatically generate a corresponding music frame sequence based on the music input by the operator in the terminal device 130 and / or storage device 140.

[0036] In some embodiments, the music frame information may include one or more of the following: beat information, volume information, and audio information of the music frame. The motion generation module 220 can generate a corresponding skeletal motion sequence based on the music frame sequence using a motion generation model.

[0037] In some embodiments, the skeletal motions in a skeletal motion sequence may include the bone position and bone rotation parameters of at least one bone corresponding to a music frame.

[0038] In some embodiments, the operation of generating a corresponding skeletal motion sequence based on a music frame sequence in the motion generation module 220 may include: generating a music frame encoding vector sequence based on the music frame sequence; and generating a skeletal motion sequence based on the music frame encoding vector sequence.

[0039] In some embodiments, the operation of generating a skeletal motion sequence based on a music frame encoded vector sequence may include: obtaining a starting skeletal motion and a stopping skeletal motion of the music frame sequence; and obtaining a skeletal motion sequence based on the music frame encoded vector sequence, the starting skeletal motion, and the stopping skeletal motion.

[0040] In some embodiments, the action generation model in the action generation module 220 may include an encoding network, a bidirectional recurrent neural network, and a decoding network. Based on this action generation model, the operation of generating a corresponding skeletal action sequence based on a music frame sequence in the action generation module 220 may include: generating a skeletal action sequence based on a music frame encoded vector sequence by: generating a music frame encoded vector sequence based on the music frame sequence using the encoding network; obtaining an initial skeletal action sequence based on the music frame encoded vector sequence, a start skeletal action, and a stop skeletal action using the bidirectional recurrent neural network; and obtaining the skeletal action sequence based on the music frame encoded vector sequence and the initial skeletal action sequence using the decoding network.

[0041] The animation generation module 230 can generate virtual character animations based on skeletal motion sequences and corresponding skeletal models.

[0042] In some embodiments, the animation generation module 230 may generate virtual character animation based on a skeletal motion sequence and a corresponding skeletal model by: obtaining at least one fitted skeletal motion between two skeletal motions in the skeletal motion sequence through a data fitting method; obtaining skeletal motion animation based on the skeletal motion sequence and the fitted skeletal motion; and generating virtual character animation based on the skeletal motion animation and the skeletal model.

[0043] In some embodiments, the virtual character animation generation system 200 may further include a training module 240, which is used to train an initial motion generation model to obtain an motion generation model.

[0044] In some embodiments, the training method of the action generation model in the training module 240 may include: obtaining at least one training sample and an initial action generation model, wherein the training sample includes music samples and their corresponding action samples; and iteratively updating the parameters of the initial action generation model based on at least one training sample to obtain the action generation model.

[0045] It should be noted that the above description of the virtual character animation generation system and its modules is for convenience only and should not be construed as limiting this specification to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, the acquisition module, action generation module, animation generation module, and training module disclosed in this specification may be different modules within a single system, or a single module may implement the functions of two or more of the aforementioned modules. For example, various modules may share a storage module, or each module may have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0046] This specification provides a method and system for generating virtual character animations based on some embodiments. The animation of a virtual character can be decomposed into a sequence of skeletal movements and a skeletal model. The sequence of skeletal movements is automatically generated based on music, and then the virtual character animation is automatically generated based on this sequence. This achieves high-quality generation of virtual character animations, reduces labor costs, and improves the efficiency of virtual character animation generation.

[0047] Figure 3 This is an exemplary flowchart of a virtual character animation generation method according to some embodiments of this specification. In some embodiments, process 300 may be executed by processing device 112. Figure 3 As shown, process 300 may specifically include steps 310 to 330.

[0048] Step 310: Obtain the music frame sequence. In some embodiments, step 310 may be performed by the acquisition module 210.

[0049] A music frame refers to audio data obtained by sampling or dividing the audio signal of music according to a preset time length (which can be called frame length, such as 25ms, 30ms, etc.). It can contain music information within the corresponding preset time length, and the music information included in the music frame can be called music frame information.

[0050] A music frame sequence can refer to the sequence of music frames that make up a piece of music. For example, if each music frame is 30ms long, a 30s piece of music can be represented as a sequence of 1000 consecutive music frames. In some embodiments, a music frame sequence can be obtained by dividing the music into frames.

[0051] In some embodiments, music and / or music frame sequences can be acquired through terminal device 130. For example, an operator can upload music through terminal device 130, and the music can be processed by terminal device 130 or processing device 112 to obtain the corresponding music frame sequence. As another example, an operator can input a calling command through terminal device 130, and terminal device 130 can send the command to processing device 112 to control processing device 112 to call the corresponding music frame sequence from storage device 140.

[0052] In some embodiments, music frames can be associated with the beat of the music. For example, when performing frame segmentation on the music, the beat points of the music can be used as the segmentation points of the music frames, and a music frame can contain audio signal data for one beat. Thus, each bone movement in the skeletal motion sequence determined based on the music frame sequence corresponds to the beat point of the music, thereby making the generated virtual character animation more rhythmic.

[0053] In some embodiments, the music frame sequence can be a sequence of music frames corresponding to various types of music. For example, when a virtual character acts as a virtual DJ, the music is generally electronic music, and various types of music can refer to electronic music with different rhythms. In some embodiments, various types of music can be categorized according to their style and characteristics. These types of music can include different types of music such as classical music, pop music, blues, rock and roll, and jazz.

[0054] In some embodiments, music frame information may include one or more of the following: beat information, volume information, audio information, and other music-related information. Beat refers to the organization of fixed unit time values ​​and dynamic patterns in a piece of music, also known as rhythm. The beat information (also known as rhythm information) of a music frame may include the number of beats per unit time for the music frame, and the unit of beat information may be BPM (Beat Per Minute). Volume information refers to the sound intensity of the music, and the unit of volume may be decibels. Audio information refers to the sound frequency information of the music, and the unit of audio may be Hertz.

[0055] In some embodiments, after acquiring music, the music can be parsed to obtain music frame information. For example, the beat information, volume information, and other information of the music frames can be stored in the music encapsulation information, and the audio information can be music waveform segments corresponding to each music frame. In some embodiments, music and / or music frame sequences can be processed by recognition algorithms to obtain music frame information. For example, the beat information can be confirmed by the recognition algorithm, then the music frame sequence can be determined based on the beat information, and finally the volume information and audio information can be determined according to the shape and amplitude of the music waveform in the music frame. The recognition algorithm may include spectrum analysis algorithms, envelope extraction algorithms, periodicity detection algorithms, etc.

[0056] Step 320: Generate a corresponding skeletal motion sequence based on the music frame sequence using the motion generation model. In some embodiments, step 320 can be performed by the motion generation module 220.

[0057] In some embodiments, the action generation model can be stored in storage device 140, and step 320 can be implemented by calling the action generation model through action generation module 220. The music frame sequence can be used as input to the action generation model, and the skeletal motion sequence can be used as output. In some embodiments, action generation module 220 can be a machine learning model. For example, the action generation model can include artificial neural networks such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), and GAN (Generative Adversarial Networks), as well as related modified artificial neural networks. In particular, the action generation model can include a Transformer model. For more details on the action generation model, please refer to further details of step 320. Figure 4 And related explanations.

[0058] Virtual characters can be various virtual human figures, animals, plants, etc. Each virtual character can have its corresponding skeleton and skeletal model. The skeleton can include the virtual character's body skeleton and key nodes of the torso. A skeletal model can refer to a model composed of at least one body skeleton and / or at least one key node. The skeletal model of a virtual character can reflect the virtual character's torso structure. For example, in the skeletal model of a virtual character, the hand can include 15 key nodes (palm, thumb first joint, thumb second joint, index finger first joint, index finger second joint, index finger third joint, middle finger first joint, middle finger second joint, middle finger third joint, ring finger first joint, ring finger second joint, ring finger third joint, little finger first joint, little finger second joint, little finger third joint, etc.). The key nodes in the skeletal model can be represented by the two-dimensional or three-dimensional contours corresponding to each joint, or by nodes and lines.

[0059] In some embodiments, the skeletal model of a virtual character can be determined according to actual needs such as the type of virtual character (e.g., gender, identity), characteristics of the virtual character (e.g., age, height), image resolution, and playback medium.

[0060] In some embodiments, when the virtual avatar is used for a virtual DJ character, the skeletal model may include 15 key nodes, as detailed in this application. Figure 6 The diagram shows the structural schematic of key nodes in the skeletal model. Figure 6 As shown, the skeletal model can include the head, spine, hips, left hand, left forearm, left arm, right hand, right forearm, right arm, left thigh, left leg, left foot, right thigh, right leg, and right foot. Each node name represents the correspondence between the node and the skeletal model, or the correspondence between nodes and the skeletal model. For example, the head node represents the position of the head in the skeletal model. Similarly, the bones between the spine node and the hip nodes represent the position and shape of the spine in the skeletal model.

[0061] Skeletal motion refers to the skeletal posture of a virtual character. A skeletal motion sequence can be a sequence of multiple skeletal actions that reflects changes in skeletal posture. In some embodiments, skeletal actions correspond to music frames, representing the skeletal actions of the virtual character corresponding to that music frame. The skeletal motion sequence can be a sequence of multiple skeletal actions corresponding to multiple music frames. In some embodiments, when the beat points of the music are used as the dividing points of the music frames, each skeletal action in the skeletal motion sequence can correspond to each beat point of the music, thereby enabling the virtual character's actions to accurately match the beat points and improving the adaptability of the virtual character's actions to the music rhythm.

[0062] The representation methods and parameters for skeletal motion can be determined according to actual needs. Skeletal motion can include one or more of various parameters related to bone posture. In some embodiments, considering that skeletal motion is mainly guided by position, rotational posture, etc., skeletal motion can be represented by bone position and bone rotation parameters. Correspondingly, the skeletal motion in the skeletal motion sequence can include the bone position and bone rotation parameters of at least one bone corresponding to the music frame. The at least one bone is at least one bone included in the skeletal model of the virtual character. In some embodiments, skeletal motion can also be directly represented as bone position and node orientation, wherein node orientation can include at least one of the orientation of the bone model corresponding to the bone node (e.g., the orientation of the palm in the left hand node), direction of motion, and direction of motion acceleration.

[0063] In some embodiments, skeletal motion can include the parameters and values ​​of the skeletal motion corresponding to a certain music frame, which can be represented by vectors. A skeletal motion sequence can include multiple vectors corresponding to multiple skeletal motions. In some embodiments, a skeletal motion can include at least one position parameter and at least one rotation parameter of at least one bone (e.g., at least one bone included in the skeletal model of a virtual character). For example, when the virtual character is a three-dimensional character, the vector of a skeletal motion can include multiple position parameters corresponding to multiple coordinate axes (e.g., x-axis, y-axis, z-axis), and multiple parameters representing the rotation state of the bone (e.g., the four rotation parameters included in the quaternion in three-dimensional rotation, or rotation parameters such as Euler angles). When the skeletal motion sequence includes T skeletal motions corresponding to T music frames, and a skeletal motion includes 3 position parameters and 4 rotation parameters of n bones, the skeletal motion corresponding to each frame can be represented as an n*(3+4) dimensional vector, and the skeletal motion sequence can be represented as T n*(3+4) dimensional vectors.

[0064] In some embodiments, the motion generation module 230 can first generate a music frame encoding vector sequence based on the music frame sequence, and then generate a skeletal motion sequence based on the music frame encoding vector sequence. This improves the motion generation model's ability to parse music frame sequences. The music frame encoding vector sequence includes multiple music frame encoding vectors corresponding to multiple music frames in the music frame sequence.

[0065] In some embodiments, each music frame encoding vector in the music frame encoding vector sequence can characterize the feature information of the music frame. For example, the music frame encoding vector may include the volume features, beat features, and audio features of the music frame.

[0066] In some embodiments, the action generation model may include an encoding network, and the music frame encoding vector can be generated through the encoding network. For example, the encoding network in a transformer, an RNN encoding model, etc., can be used to implement the encoding network model. In some embodiments, the encoding network may include one or more encoders (e.g., a transformer encoding network including multiple encoders).

[0067] In some embodiments, the encoding network may include a machine learning model that incorporates a self-attention mechanism (also referred to as autocorrelation), such as an RNN model with an attention mechanism, a Transformer model, etc. Correspondingly, each music frame encoding vector may include the correlation between that music frame and other music frames, and the music frame encoding vector sequence may include the correlation between each music frame. For example, for a music frame sequence of T frames, the music frame encoding vector sequence may be represented as a T×T matrix, where each row of the matrix represents the correlation between the music frame corresponding to that row / column and other music frames, and the music frame encoding vector may be the value of a row or a column in the matrix.

[0068] Based on this, changes in the features of music frames within a music frame sequence can be reflected through a music frame encoding vector sequence. For example, when a music frame sequence includes 20 music frames, the operator can perform a music frame adjustment operation (e.g., adjusting the volume) at the 10th frame. After processing by an encoding network, the output music frame encoding vector sequence shows that the first 10 frames are volume-related, and the last 10 frames are volume-related, while the first 10 frames and the last 10 frames are not volume-related. This correlation can be parsed in subsequent operations, thus ensuring that the skeletal motion corresponding to the 10th frame corresponds to the volume adjustment action. For details regarding music frame adjustment operations, please refer to this application. Figure 4 Related descriptions.

[0069] In some embodiments, the position vector representation of each music frame in the music frame sequence can be obtained first; then, a music frame encoding vector sequence can be generated by an encoding network based on the music frame sequence and the position vector representation of each music frame in the music frame sequence.

[0070] In some embodiments, to improve the accuracy of music frame encoding vectors and reduce the computational load of the encoding process, feature extraction can be performed on the music frames before determining the music frame encoding vector sequence based on the music frame sequence. This yields music frame features for each music frame, resulting in a music frame feature sequence corresponding to the music frame sequence. Specifically, generating the music frame encoding vector sequence based on the music frame sequence can include first generating a music frame feature sequence based on the music frame sequence, and then generating the music frame encoding vector sequence based on the music frame feature sequence. In some embodiments, feature extraction of music frames can be implemented using feature extraction networks, such as NN (Neural Networks) and CNN.

[0071] In some embodiments, the motion generation model may further include a decoding network for determining a sequence of skeletal motions based on the music frame encoding vectors. In some embodiments, the decoding network may be implemented using various network models capable of processing music frame encoding vectors to obtain corresponding skeletal motions, such as deep neural networks, transformer decoding networks, etc. In some embodiments, the decoding network may include one or more decoders (e.g., a transformer decoding network including multiple decoders).

[0072] In some embodiments, the starting and stopping skeletal movements of the music frame sequence can also be determined. The stopping skeletal movement of the current music frame sequence can also serve as the starting skeletal movement of the next music frame sequence, ensuring the continuity of the virtual character's movements when switching between multiple music segments. In some embodiments, generating a skeletal movement sequence based on the music frame encoding vector sequence can include obtaining the skeletal movement sequence based on the music frame encoding vector sequence, the starting skeletal movement, and the stopping skeletal movement.

[0073] In some embodiments, the starting and stopping skeletal movements can be set based on the music information of the music frame sequence and / or actual needs. In some embodiments, the motion generation model may further include a bidirectional recurrent neural network, which can then generate a skeletal motion sequence based on the music frame encoded vector sequence, the starting skeletal movement, and the stopping skeletal movement. More details about motion generation models and generating skeletal motion sequences can be found in [reference needed]. Figure 4 And its related descriptions.

[0074] Step 330: Generate virtual character animation based on the skeletal motion sequence and the corresponding skeletal model. In some embodiments, step 330 can be performed by the animation generation module 230.

[0075] In some embodiments, individual skeletal movements from a skeletal motion sequence can be overlaid onto corresponding skeletal models, and these movements can be spliced ​​together to obtain virtual character animation. Specifically, individual skeletal movements from a skeletal motion sequence can be spliced ​​together to obtain skeletal motion animation, which can then be applied to the virtual character model (e.g., importing the skeletal motion animation into a computer program, which then applies it to the virtual character model) to obtain virtual character animation. In some embodiments, overlaying individual skeletal movements from a skeletal motion sequence onto corresponding skeletal models can be achieved using skeletal skinning technology, which binds the virtual character's bones (key nodes) to corresponding positions on the skeletal model, thereby converting skeletal motion parameters into movements of the skeletal model.

[0076] In some embodiments, skeletal motion sequences can be supplemented with additional skeletal motions. Specifically, for two adjacent skeletal motion sequences, one or more additional skeletal motions can be added between the two adjacent skeletal motion sequences to make the virtual character animation generated based on the supplemented skeletal motion sequences smoother and more fluid.

[0077] In some embodiments, supplementing one or more skeletal movements between two adjacent skeletal movements can be achieved through data fitting. Fitting can be understood as fitting a data curve or function based on known data points to achieve data smoothing. According to the data fitting method, a data curve or function corresponding to a skeletal movement can be fitted based on multiple skeletal movements in a skeletal movement sequence, thereby obtaining one or more supplementary skeletal movements between two adjacent skeletal movements based on this data curve or function. Then, a virtual character animation can be obtained based on the skeletal movement sequence and the fitted skeletal movements (e.g., first obtaining the skeletal movement animation, then applying the skeletal movement animation to the virtual character model to obtain the virtual character animation). In this specification, the supplementary skeletal movements obtained according to the data fitting method can be referred to as fitted skeletal movements.

[0078] In some embodiments, the data fitting method may include Gaussian fitting, polynomial fitting, Hermite curve fitting, etc.

[0079] In some embodiments, the order of some steps in the method for generating virtual character animation described above can be changed. For example, the animation corresponding to each music frame can be determined first, and then the animation of the intermediate frames can be fitted based on the animation of each music frame to obtain the overall virtual character animation.

[0080] Figure 4 This is an exemplary structural block diagram of an action generation model provided in some embodiments of this specification.

[0081] like Figure 4 As shown, the action generation model 400 may include an encoding network 410, a bidirectional neural network 420, and a decoding network 430.

[0082] The encoding network 410 can be used to generate a music frame encoded vector sequence based on a music frame sequence. In some embodiments, the encoding network 410 can consist of transformer encoders, and may include multiple cascaded transformer encoders to obtain deeper music information from the music frame sequence. Specifically, the input to the first transformer encoder is the music frame sequence, the output to the last transformer encoder is the music frame encoded vector sequence, and the input to intermediate transformer encoders is the encoded output of the previous transformer encoder.

[0083] For each transformer encoder layer, the encoder input can be processed through position encoding, a multi-head attention layer, and a feedforward neural network to obtain the encoder output. The multi-head attention layer allows the output of this layer to include the autocorrelation of its input. The transformer encoder can also incorporate other structures as needed, such as normalization layers and stacking layers.

[0084] The bidirectional recurrent neural network 420 can be used to obtain an initial skeletal action sequence based on the music frame encoded vector sequence, the starting skeletal action, and the stopping skeletal action. Specifically, the bidirectional recurrent neural network 420 refers to a recurrent neural network that can acquire memories from past time points and information from future time points; that is, the determination of the initial skeletal action at a certain time t is influenced by both the starting and stopping skeletal actions. In some embodiments, the bidirectional recurrent neural network 420 may include a bidirectional long short-term memory (BLSTM) recurrent neural network or a similar neural network model. The internal structure of the bidirectional long short-term memory recurrent neural network can be found in [reference needed]. Figure 4 .

[0085] The decoding network 430 can be used to obtain a skeletal motion sequence based on the music frame encoded vector sequence and the initial skeletal motion sequence. In some embodiments, the decoding network 430 can be composed of transformer decoders, and may include multiple cascaded transformer decoders. The input to the first transformer decoder is the music frame encoded vector sequence and the initial skeletal motion sequence, the output of the last transformer decoder is the skeletal motion sequence, and the input to intermediate transformer decoders is the output of the previous transformer decoder.

[0086] For each transformer decoder layer, the decoder can perform a mask multi-head attention mechanism on the input data (such as the decoding output of the previous layer, or a sequence of music frame encoded vectors, or a sequence of music frame encoded vectors and an initial skeletal motion sequence) to calculate the relationship between the current skeletal motion and the adjusted skeletal motion. Then, the calculation result is combined with the music frame encoded vector sequence and subjected to a multi-head attention mechanism to calculate the relationship between the currently adjusted skeletal motion and the encoded vectors. Finally, the decoder output is obtained through a feedforward neural network and a softmax regression function. The transformer decoder can also be configured with other structures as needed, such as normalization layers, stacking layers, etc.

[0087] In this embodiment, the skeletal motion output by the motion generation model accurately reflects the correlation between music frame sequences, exhibiting a high degree of correlation with the music frame sequences. Furthermore, the generation of skeletal motion is influenced by the starting and ending skeletal motions, resulting in a more coherent sequence of skeletal motions and facilitating the splicing of multiple music frame sequences.

[0088] In some embodiments, when a virtual character is used as a virtual DJ, the virtual character's actions can include two categories: improvisational performance and technical actions. Dance actions refer to the virtual DJ performing movements such as dancing and swaying according to the melody and rhythm of the music during the main part of a song. Technical actions refer to the technical actions used by the virtual DJ to improve the playback effect when playing music. Technical actions can include one or more of the following: setting a hot cue, adjusting pitch, adjusting volume, and adjusting the equalizer (EQ). The virtual DJ's technical actions correspond to the operator's manipulation of the original music's music frames. Setting a hot cue can be understood as setting a memory point (playback point) in the music and jumping the music's playback position to that point. Adjusting pitch can be understood as adding effects to the music (e.g., raising or lowering the pitch). Adjusting the equalizer can be understood as adjusting the various frequencies of sound in the music; for example, channels 63 and 160 in the equalizer represent low-frequency information and are related to the performance of percussion instruments in the music.

[0089] In some embodiments, the operator can directly input a sequence of music frames containing music frame operation instructions. During encoding, the music frame sequence containing these instructions can be used to identify the operation instructions based on internal correlations, which are then reflected in the music frame encoding vector sequence. For example, when adjusting the volume, the adjusted music frame volume may be lower than the previous music frame volume. This difference is reflected in the correlation between the various music frame sequences, and thus in the music frame encoding vector sequence. During subsequent decoding, this correlation can be used to deduce that the virtual character has performed the technical action of adjusting the volume.

[0090] In some embodiments, to reduce training difficulty, after determining the original music, the operator can input music frame operation instructions in real time, and the processing device 112 determines the music frame sequence based on the original music and the music frame operation instructions. In some embodiments, the position corresponding to the music frame operation instructions can be used as the sequence segmentation point to divide the processed original music into multiple music frame sequences. Thus, the starting and stopping skeletal movements of the music frame sequence can correspond to the technical movements indicated by the music frame operation instructions, thereby achieving a smooth connection between technical movements and dance movements.

[0091] Figure 5 This is an exemplary flowchart of an action generation model training method according to some embodiments of this specification. Figure 5 As shown, process 500 may specifically include steps 510 and 520. In some embodiments, process 500 may be executed by processing device 112.

[0092] Step 510: Obtain at least one training sample and an initial motion generation model. The training samples include music samples and their corresponding motion samples. The initial motion generation model can be an unadjusted model or a motion generation model 400 that has not yet met the requirements. The number of training samples can be selected based on factors such as the accuracy requirements of the motion generation model and the actual application scenario.

[0093] In some embodiments, various types of music can be acquired as music samples. For each music sample, the corresponding music frame sequence can be obtained. Furthermore, based on the music parameters of the music in the music sample library, virtual character animations can be artificially created and the motion parameters of the animations can be exported to obtain motion samples corresponding to the music samples (e.g., virtual character actions corresponding to each music frame, or virtual character animations (including multiple virtual character actions) corresponding to a piece of music (including multiple music frames)), thereby establishing a motion sample library.

[0094] In some embodiments, music samples can be classified based on their type, similarity to music parameters such as bpm, and training samples can be constructed based on the classification results. For example, music samples can first be classified based on their music frame information to obtain classification results, where the classification results can include electronic music with different bpm. Then, based on the classification results and the motion samples corresponding to each music sample in each category, a motion sample library corresponding to each category is generated.

[0095] Music samples of the same category can correspond to the same action sample library. The trained action generation model has a high recognition ability for this type of music. At the same time, it also has a certain transfer and adaptation ability for music of similar types.

[0096] In some embodiments, to improve the ability of the action generation model to represent music adjustment operations (technical actions), the music samples can be processed to introduce music adjustment operations and expand the sample range. Specifically, one or more music samples to be processed can be obtained first. Then, the music adjustment operations of the music samples to be processed are determined based on the music samples to be processed, such as simulating the changes in frequency and volume at random positions in the middle of a single piece of music to simulate the changes in the actual DJing process, expanding the dataset samples, and classifying them accordingly, as well as the corresponding action samples.

[0097] In some embodiments, to simulate DJ improvisation, the aforementioned music adjustment operation may include at least one random music frame operation instruction randomly inserted into the music sample to be processed. In response to the music adjustment operation, a technical action corresponding to the music frame operation instruction may be added to the corresponding action sample. For example, for a 3-minute music sample to be processed, a hot cue operation can be performed at 50s to rewind the music to 45s, the pitch can be adjusted at 1 minute 23s, and the volume can be adjusted at 2 minutes. In the corresponding action sample, the virtual DJ may perform a hot cue action at 50s, perform a pitch adjustment action at 1 minute 23s, and perform a volume adjustment action at 2 minutes.

[0098] In some embodiments, the music samples after music adjustment operations can be categorized to expand the number of music samples in each category. In some embodiments, music adjustment operations may include operations that adjust the type of music samples, such as mixing, adjusting bpm, etc., thereby further expanding the number of music samples in each category.

[0099] Step 520: Iteratively update the parameters of the initial action generation model based on the training samples to obtain the action generation model.

[0100] In some embodiments, the music frame sequence corresponding to a music sample can be input into an initial motion generation model. The initial motion generation model outputs a skeletal motion sequence, which can serve as the label for the corresponding music sample. Each iteration adjusts or updates the model parameters based on the comparison between the model output and the label, minimizing the difference between the model output and the label. In some embodiments, a loss function can be determined based on the difference between the model output and the label, and the model parameters can be adjusted or updated based on this loss function. In some embodiments, the skeletal motion label values ​​for each music frame can be manually labeled beforehand. In some embodiments, the method for iteratively updating the model parameters can include conventional model training methods such as stochastic gradient descent.

[0101] Therefore, this specification packages the entire action generation model into an initial action generation model. The parameters of each action generation model are determined simultaneously during the training of the initial action generation model, reducing engineering complexity. It also avoids the need for multiple modules (e.g., Figure 4 The macroscopic target bias caused by the inconsistency of training targets in the encoding network 410, bidirectional recurrent neural network 420 and decoding network 430 is eliminated, thereby improving the overall performance of the human posture prediction system.

[0102] As can be seen from the foregoing, some embodiments of this specification may bring beneficial effects including but not limited to: (1) automatically generating virtual character animations through music frame sequences, reducing labor costs; (2) introducing the internal correlation of music frame sequences to analyze the impact of technical operations on music frame sequences, thereby determining the technical operations of the virtual DJ; (3) introducing starting and stopping skeletal movements to make the connection between various music frame sequences more coherent; (4) packaging and training the motion generation model as a whole, reducing the complexity of the project, avoiding macroscopic target deviations caused by inconsistent training objectives in multiple modules, thereby improving the overall performance of the human posture prediction system. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced can be any one or a combination of the above, or any other possible beneficial effects.

[0103] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0104] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0105] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0106] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0107] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0108] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0109] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for generating virtual character animation, comprising: obtaining a sequence of music frames, each music frame in the sequence of music frames comprising music frame information, the sequence of music frames comprising music frame operation instructions; generating a corresponding sequence of skeletal actions based on the sequence of music frames by an action generation model, the action generation model comprising an encoding network, a bidirectional recurrent neural network, and a decoding network, comprising: generating a sequence of music frame encoding vectors from the sequence of music frames by the encoding network; generating the sequence of skeletal actions from the sequence of music frame encoding vectors, comprising: obtaining a starting skeletal action and a stopping skeletal action of the sequence of music frames, the starting skeletal action and the stopping skeletal action being set according to music information of the sequence of music frames and / or actual requirements, the starting skeletal action and the stopping skeletal action corresponding to a technical action indicated by the music frame operation instructions, the technical action comprising one or more of setting a hot memory point, adjusting a pitch, adjusting a volume, and adjusting a music equalizer; obtaining the sequence of skeletal actions from the sequence of music frame encoding vectors, the starting skeletal action, and the stopping skeletal action, comprising: obtaining an initial sequence of skeletal actions from the sequence of music frame encoding vectors, the starting skeletal action, and the stopping skeletal action by the bidirectional recurrent neural network; obtaining the sequence of skeletal actions from the sequence of music frame encoding vectors and the initial sequence of skeletal actions by the decoding network; and generating virtual character animation based on the sequence of skeletal actions and a corresponding skeletal model.

2. The method of claim 1, wherein the music frame information comprises one or more of beat information, volume information, and audio information of a music frame.

3. The method of claim 1, wherein a skeletal action in the sequence of skeletal actions comprises a skeletal position and a skeletal rotation parameter of at least one skeleton corresponding to a music frame.

4. The method of claim 1, wherein generating virtual character animation based on the sequence of skeletal actions and a corresponding skeletal model comprises: obtaining at least one fitted skeletal action between two skeletal actions in the sequence of skeletal actions by a data fitting method, the data fitting method comprising Gaussian fitting, polynomial fitting, or Hermite curve fitting; obtaining skeletal action animation based on the sequence of skeletal actions and the fitted skeletal action; and generating the virtual character animation based on the skeletal action animation and the skeletal model.

5. The method of claim 1, wherein the action generation model is obtained by: obtaining at least one training sample and an initial action generation model, wherein the training sample comprises a music sample and a corresponding action sample, the training sample being constructed by classifying music samples according to a type of the music sample and a similarity of a bpm of the music sample; and updating parameters of the initial action generation model based on the at least one training sample to obtain the action generation model.

6. A system for generating virtual character animation, comprising: ​ An acquisition module is configured to acquire a music frame sequence, each music frame in the music frame sequence comprising music frame information, and the music frame sequence comprising music frame operation instructions; An action generation module is configured to generate a corresponding skeleton action sequence based on the music frame sequence by using an action generation model, the action generation model comprising an encoding network, a bidirectional recurrent neural network, and a decoding network, and comprising: generating a music frame encoding vector sequence according to the music frame sequence by using the encoding network; generating the skeleton action sequence according to the music frame encoding vector sequence, and comprising: acquiring a starting skeleton action and a stopping skeleton action of the music frame sequence, the starting skeleton action and the stopping skeleton action being set according to music information of the music frame sequence and / or actual requirements, the starting skeleton action and the stopping skeleton action corresponding to technical actions indicated by the music frame operation instructions, the technical actions comprising one or more of setting a hot memory point, adjusting a pitch, adjusting a volume, and adjusting a music equalizer; obtaining the skeleton action sequence according to the music frame encoding vector sequence, the starting skeleton action, and the stopping skeleton action, and comprising: obtaining an initial skeleton action sequence based on the music frame encoding vector sequence, the starting skeleton action, and the stopping skeleton action by using the bidirectional recurrent neural network; obtaining the skeleton action sequence based on the music frame encoding vector sequence and the initial skeleton action sequence by using the decoding network; and an animation generation module configured to generate a virtual character animation based on the skeleton action sequence and a corresponding skeleton model.

7. A computer readable storage medium, the storage medium storing computer instructions, when a computer reads the computer instructions in the storage medium, the computer executes the method according to any one of claims 1-5.

Images