Animation control information construction method and device, equipment, medium and product

Abstract

The application relates to an animation control information construction method and device, equipment, medium and product, the method comprising: acquiring music, determining audio segments corresponding to each music measure in a structural unit of the music; matching candidate audio segments similar to the audio segments of each music measure from a material database, obtaining preset motion control information corresponding to each candidate audio segment; forming a full-quantity combination path through ordered full permutation between motion control information of all music measures in the structural unit, searching for an optimal combination path of the structural unit in the full-quantity combination path; and forming motion control information contained in the optimal combination path of the structural unit into a control information unit corresponding to the structural unit, which is used for controlling a digital person to generate corresponding dance animation. The dance animation generated by the application has smooth motion of the digital person, and can be coordinated and synchronized with the emotion and rhythm of the music.

Description

Technical Field

[0001] This application relates to digital human virtual technology, and more particularly to a method for constructing animation control information and its apparatus, equipment, medium, and product. Background Technology

[0002] Currently, generating digital human dances from music is a research hotspot in the fields of music information processing, computer graphics, and robotics.

[0003] Early research primarily focused on simple choreography based on the principles of dance movement, requiring significant human intervention. With the rise of deep learning, Transformer models employing a full attention mechanism emerged for cross-modal mapping between music and dance, enabling automatic dance generation from music. However, due to the lack of explicit constraints on preserving generated poses within spatial constraints and the use of autoregressive generation, this model generates many strange and clipping movements, and sometimes produces stuck, unusable dance actions, thus rendering it unsuitable for practical applications.

[0004] The later-released Bailando model addressed these issues by using the VQ-VAE model to map dance movements into a dance subspace and employing a GPT network to generate dance sequences. This solved the problems of strange movements and getting stuck, but issues such as clipping in animation and unattractive movements still existed.

[0005] Therefore, it is essential to find an intelligent choreography approach that generates smooth dance movements that align with musical emotions and rhythms, in order to address the dance movements required by the current "metaverse" for digital humans. Summary of the Invention

[0006] The purpose of this application is to solve the above-mentioned problems by providing a method for constructing animation control information and corresponding devices, equipment, non-volatile readable storage media, and computer program products.

[0007] According to one aspect of this application, a method for constructing animation control information is provided, comprising the following steps:

[0008] Acquire music and determine the audio segments corresponding to each musical measure within the structural unit of the music;

[0009] Candidate audio segments similar to the audio segments of each music bar are matched from the material database, and preset motion control information corresponding to each candidate audio segment is obtained. The motion control information includes multiple information frames for controlling the digital human to produce corresponding dance movements.

[0010] The full combination path is formed by the ordered arrangement of the motion control information of all music measures within the structural unit, and the optimal combination path of the structural unit is searched in the full combination path.

[0011] The motion control information contained in the optimal combination path of the structural unit is used to form the corresponding control information unit of the structural unit, which is used to control the digital human to generate corresponding dance animation.

[0012] According to another aspect of this application, an animation control information construction apparatus is provided, comprising:

[0013] The format processing module is configured to acquire music and determine the audio segments corresponding to each musical measure within the structural unit of the music.

[0014] The material matching module is configured to match candidate audio segments that are similar to the audio segments of each music bar from the material database, and obtain preset motion control information corresponding to each candidate audio segment. The motion control information includes multiple information frames for controlling the digital human to produce corresponding dance movements.

[0015] The material selection module is configured to form a full combination path by the orderly arrangement of the motion control information of all music measures within the structural unit, and to search for the optimal combination path of the structural unit within the full combination path;

[0016] The unit construction module is configured to construct the motion control information contained in the optimal combination path of the structural unit into a corresponding control information unit for controlling the digital human to generate corresponding dance animations.

[0017] According to another aspect of this application, an animation control information construction device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the animation control information construction method described in this application.

[0018] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the animation control information construction method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0019] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0020] Compared with existing technologies, this application has several technological advantages, including but not limited to:

[0021] First, this application performs structural and rhythmic analysis on the audio data of the music, dividing the audio data into multiple musical measures according to structure and rhythm. Based on the audio data of each musical measure, multiple candidate audio segments are selected from a pre-selected material database using semantic matching. Then, the corresponding preset motion control information of these candidate audio segments is used to construct a control information unit corresponding to each structural unit, which can be used to control the digital human. Since the materials in the material database are pre-selected, high-quality motion control information corresponding to candidate audio segments can be matched. This ensures that when the control information unit is used to control the digital human to generate corresponding dance animations, the corresponding dance animation movements are smooth, and the emotions expressed and the rhythm implemented can achieve a high-quality matching effect with the music, avoiding abnormal phenomena such as clipping and stuttering in the resulting dance animation.

[0022] Secondly, after matching multiple similar candidate audio segments for each music measure to determine each candidate audio segment, this application will perform full permutation and combination of the candidate audio segments of each music measure in each structural unit as a unit to obtain multiple combination paths. Then, based on the full number of combination paths, the best combination path will be selected to achieve the optimal selection of combination paths and ensure that the dance animation obtained accordingly has the best quality.

[0023] Furthermore, this application obtains the motion control information required for dance animation by selectively retrieving existing motion control information from the material database through semantic matching. This method involves low computational load and does not require excessive reliance on neural network models to perform a large amount of computation to determine the motion control information. This also saves on model training costs and sample preparation costs, making it more economical and cost-effective. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the network architecture of an exemplary application environment for this application;

[0026] Figure 2 This is a schematic diagram illustrating an exemplary digital human and the distribution of its skeletal key points according to this application;

[0027] Figure 3This is a flowchart illustrating one embodiment of the animation control information construction method of this application;

[0028] Figure 4 This is a schematic diagram illustrating the process of segmenting audio segments from music into measures in an embodiment of this application.

[0029] Figure 5 This is a flowchart illustrating the process of matching multiple action control information corresponding to each music bar from the material database in an embodiment of this application.

[0030] Figure 6 This is a flowchart illustrating the process of evaluating and determining the optimal combination path for each structural unit in an embodiment of this application.

[0031] Figure 7 This is a schematic diagram illustrating the process of generating corresponding dance animations based on music animation control files in this application embodiment;

[0032] Figure 8 A schematic diagram of several exemplary image frames in a dance animation generated from the animation control file of this application;

[0033] Figure 9 This is a schematic block diagram of the animation control information construction device of this application;

[0034] Figure 10 This is a schematic diagram of the structure of an animation control information construction device used in this application. Detailed Implementation

[0035] Please see Figure 1 The network architecture adopted in an exemplary application scenario of this application includes a terminal device 80, a media server 81, and an application server 82. The application server 82 can be used to deploy a live streaming service. The media server 81 can be used to generate a video stream corresponding to the digital human based on motion control information. The terminal device 80 allows broadcasters or viewers to log into the live streaming room supported by the live streaming service. The broadcaster can drive the media server to generate a dance animation of the digital human by playing music, and then push it to the live streaming room for viewers to view as a video stream.

[0036] Specifically, when a user accesses the live streaming service provided by the application server 82 from their terminal device 80, enters the corresponding live streaming room, and starts playing music, the media server can call the corresponding animation control file based on the music. The media server generates a dance animation for the digital human based on the motion control information provided in the animation control file corresponding to each musical measure of the music. Under the coordination of the application server 82, the dance animation is pushed to the live streaming room in the form of a video stream, thereby providing a live streaming service for the digital human through the network.

[0037] Other exemplary application scenarios may flexibly utilize various network architectures to realize various music-based digital human dance animation services. For example, some augmented reality, virtual reality, and 3D game scenarios can all become application scenarios of this application. The technical solution of this application is used to generate corresponding animation control files for music. When the music is played, the animation control files are used to generate digital human dance animations, and the dance animations are played synchronously with the music.

[0038] The animation control file corresponds to the rhythm settings of the music. Specifically, it provides corresponding motion control information for each musical measure consisting of two strong beats. The motion control information includes information frames corresponding to each image frame in the digital human dance animation to be generated. Each information frame stores control data describing the digital human in the corresponding image frame within the dance animation. This control data is described by motion vectors or static position data of the digital human relative to preset skeletal key points. The specific data format can be standardized to adapt to the output format of the motion generation model or arbitrarily preset. For example, in one embodiment, each information frame can provide the rotation values ​​of each preset skeletal key point on the three axes of the image coordinate system and the offset of the digital human's position. In another embodiment, each information frame can provide the position coordinates of each preset skeletal key point in the image coordinate system. Regardless of the form, in one information frame, the control data of all skeletal key points can be represented as an information matrix, the dimension of which can be represented as: joints * dimensions, where joints represents the total number of joints to be controlled, using Euler angles. In the illustrated embodiment, it is the sum of the actual number of joints of the digital human plus a digital human position offset. In the embodiment using absolute coordinates to represent joints, it can be the actual number of joints of the digital human. Dimensions usually correspond to the actual dimensions of the coordinate system, i.e., three dimensions: x, y, and z axes. Therefore, assuming 25 image frames are generated per second, the motion control information within one second can be represented as an information matrix of 25 * joints * dimensions. In actual calculation, for each music measure, its duration is multiplied by the frame rate (frameRates) per second to determine its specific duration (time). Thus, the motion control information for each music measure can be represented as: frameRates * time * joints * dimensions. It can be seen that as long as the data content of the information frames is defined in advance, the motion control information can be constructed using any parsable data organization method.

[0039] The key skeletal features of the digital human, such as Figure 2 The example mainly includes various skeletal joints distributed in the body, limbs, and hands of the digital human. By displacing these skeletal key points in the three-dimensional image coordinate system, the corresponding body parts of the digital human can be driven to produce corresponding motion effects. By progressively adjusting the positional information of the same set of skeletal key points in multiple information frames, the image frames of the digital human with gradual motion changes can be controlled accordingly. By playing the video stream composed of these image frames in an orderly manner, the motion effects of the digital human body parts corresponding to the corresponding skeletal key points can be visually presented.

[0040] The digital human can be pre-modeled so that, according to the position information of each skeletal key point in the information frame, the image frame position and viewpoint of the body parts corresponding to each skeletal key point can be adjusted. This allows for the subsequent generation of corresponding digital human image frames through rendering, forming a dance animation from sequentially occurring image frames. The animation control information corresponding to the audio data of a piece of music can be used to drive the digital human to generate the corresponding dance animation in real time, or it can be stored as an animation control file corresponding to the music. When the music is played, the animation control file is then called to drive the digital human to generate the corresponding dance animation. Furthermore, depending on actual needs, corresponding background images or other foreground images can be added when generating the digital human's image frames; this can be flexibly implemented by those skilled in the art.

[0041] Please see Figure 3 According to an animation control information construction method provided in this application, in one embodiment, the method includes the following steps:

[0042] Step S1100: Obtain music and determine the audio segments corresponding to each music measure within the structural unit of the music;

[0043] When it is necessary to generate corresponding animation control information for a piece of music, the music can be determined first, and then the audio data of the music can be segmented using the structural and rhythmic information of the music, dividing the audio data into audio segments corresponding to each musical measure in each structural unit of the music.

[0044] It is understandable that music is a structured data file. A piece of music typically contains different structural units such as an intro, verse, chorus, bridge, and outro. While the verse, chorus, and bridge often interweave and repeat, their basic structure is composed of multiple different types of structural units. Therefore, the structural information of the music can be obtained in advance to determine its structure. This structural information can be obtained through structural analysis of the music, and any mature and feasible method can be used for this analysis. For music that provides its own sheet music file, the structural information recorded in that file can also be directly accessed.

[0045] Every piece of music has its rhythmic information, usually represented in sheet music as a beat marker, such as 2 / 4 or 4 / 4 time. The numerator represents the number of beats, and the denominator represents the unit duration. Therefore, the beat marker indicates how many beats are triggered within a unit duration. Music maintains its rhythm by repeating beats. The analysis of rhythmic information in music can be implemented using any mature and feasible method. In this application, the audio data corresponding to the unit duration used to represent the beat is considered a musical measure. Generally, beats are divided into strong beats and weak beats, and sometimes may include secondary weak beats. Strong beats are also called drumbeats, and the interval between two adjacent strong beats represents a musical measure. Therefore, based on the rhythmic information of the music, using the timestamps of two drumbeats, an audio segment is segmented from the audio data. This audio segment is the audio data of the musical measure defined by the two drumbeats.

[0046] To facilitate organization and utilization, in the process of determining the audio segments corresponding to each musical measure of the music, the structural information of the music can be combined, and each musical measure can be processed separately for each structural unit of the music to obtain the audio segments corresponding to each musical measure. Then, an association mapping relationship can be established between these audio segments and their respective structural units to facilitate subsequent targeted processing.

[0047] Step S1200: Match candidate audio segments similar to the audio segments of each music bar from the material database, and obtain preset motion control information corresponding to each candidate audio segment. The motion control information includes multiple information frames for controlling the digital human to produce corresponding dance movements.

[0048] This application provides motion control information templates through a material database. Specifically, the material database stores pre-selected audio segments and their corresponding motion control information. These audio segments can originate from multiple pieces of music, and these pieces of music already possess high-quality animation control information required to drive the digital human to generate dance animations. Thus, following the same method as step S1100 of this application, the audio segment corresponding to each music bar and its corresponding motion control information can be obtained. Each audio segment corresponds to a music bar setting, and its motion control information contains multiple information frames. Using each information frame in the motion control information to drive the digital human to generate image frames corresponding to each information frame, a corresponding dance animation can be generated from these image frames. This dance animation maintains a coordinated correspondence with the audio segments in terms of emotion and rhythm, and has excellent image quality without abnormal phenomena such as clipping or stuttering.

[0049] Therefore, the motion control information of these audio segments in the material database actually serves as a template, which can be used as basic material to construct motion control information for other music. The motion control information of these audio segments can be pre-generated manually or generated using a deep learning model and then manually reviewed and selected; in short, the higher the quality, the better. Of course, in terms of emotion, style, and rhythm, the dance animation generated by the audio segments and their motion control information should be unified and coordinated.

[0050] To facilitate subsequent processing, during the construction of the material database, a preset feature extraction model can be used to extract the deep semantic representation of each audio segment. The deep semantic representation is then stored in the material database as a vector, associated with its corresponding audio segment, for later retrieval.

[0051] The feature extraction model can be implemented using a neural network model suitable for processing audio feature information, such as a convolutional neural network model, a recurrent neural network model, a long short-term memory network, or a Transformer encoder, which can be selected by those skilled in the art as needed. To enable the feature extraction model to have feature representation capabilities, in one embodiment, a classifier can be added after the neural network model. Then, the audio feature information of the prepared audio data is input into the neural network model, which performs feature representation to obtain a deep semantic representation. This representation is then input into the classifier for emotion or style classification. The classification loss of the classifier's classification result is calculated using the corresponding real emotion or style labels. The classification loss value is used to correct the weight parameters of the entire training network. By iteratively training the entire training network with massive amounts of training samples, the feature extraction model is continuously trained to convergence until it reaches a convergent state based on the classification loss value. The feature extraction model can then be used to extract the corresponding deep semantic representation for the audio segments of this application. Generally, the deep semantic representation is represented as a vector with a defined dimension, such as a 32-dimensional vector.

[0052] Before feeding the audio data into the feature extraction model, it is usually necessary to extract audio features from the audio data to obtain corresponding audio feature information. Therefore, in one embodiment, conventional audio preprocessing methods are used, including pre-emphasis, framing, windowing, short-time Fourier transform, Mel-scale filtering, and logarithmic calculation, to obtain the corresponding Mel-cepstral coefficients as audio feature information, which can be represented as a sequence vector. In other embodiments, the audio data can also be represented as constant Q transform features, chromatographic features (Chroma), etc. Any other feature representation that can represent the style-invariant features of the audio data does not affect the inventive spirit of this application.

[0053] Based on the aforementioned material database, material template resources can be provided for the audio segments corresponding to each musical measure of each structural unit of the audio. The method of obtaining the corresponding material template resources from the material database for each musical measure's audio segment can be achieved by implementing semantic matching.

[0054] Specifically, for each music measure that needs to be matched with the source template resources, the audio data of that music measure undergoes similar audio preprocessing to obtain its corresponding audio feature information. This information is then input into the feature extraction model for feature representation, resulting in its corresponding deep semantic representation, which is also a vector with a defined dimension, such as a 32-dimensional vector. Based on the deep semantic representation of the music segment of the music measure, the data distance between the deep semantic representation of the music segment and the deep semantic representation of each audio segment in the source database can be calculated one-to-one. This data distance is expressed as a semantic similarity score, used to measure the degree of semantic similarity between the two deep semantic representations. Thus, the degree of semantic similarity between each music measure's audio segment and each audio segment in the source database is determined through various semantic similarity scores.

[0055] The data distance between vectors can be calculated using any mature and feasible algorithm, such as the cosine similarity algorithm, vector dot product algorithm, Euclidean distance algorithm, Jaccard coefficient algorithm, Pearson correlation coefficient algorithm, etc. For example, in this embodiment, it is recommended to use the cosine similarity algorithm to determine the data distance between two vectors and directly express the data distance as a semantic similarity form.

[0056] For each audio segment of a music measure, each audio segment in the material database has a corresponding semantic similarity. Therefore, based on a preset number or a preset threshold, multiple audio segments with high semantic similarity can be selected from the material database for each music measure as candidate audio segments corresponding to each music measure.

[0057] Through semantic matching, each musical measure of the music has multiple candidate audio segments that are similar to it. Since each candidate audio segment has pre-provided preset action control information in the material database, multiple candidate action control information that are suitable for each musical measure of the music are actually determined. These candidate action control information can be regarded as a list of candidate actions corresponding to the corresponding musical measure.

[0058] Step S1300: The full combination path is formed by the orderly arrangement of the motion control information of all music measures in the structural unit, and the optimal combination path of the structural unit is searched in the full combination path.

[0059] For each structural unit of the music, each musical measure has multiple candidate audio segments corresponding to it, that is, multiple selectable action control information. Therefore, the action control information of multiple musical measures actually forms a many-to-many relationship, which is an ordered full permutation relationship. Each permutation can form a combination path. However, a piece of music only needs one optimal combination path to satisfy it. Therefore, for each structural unit, there needs to be a corresponding strategy to determine an optimal combination path from its full range of combination paths.

[0060] One exemplary implementation of determining the optimal combination path is to dynamically plan the path within each structural unit according to the time sequence of the music measures. Starting from the first music measure, the optimal motion control information is selected. Then, based on the optimal motion control information determined in the previous music measure, the optimal motion control information for the next music measure is searched, and so on, until the optimal motion control information for the last music measure is determined. The optimal motion control information corresponding to each music measure is sequentially spliced ​​together to form the optimal motion control information set corresponding to the entire structural unit. In effect, this determines the final matching motion control information for each music measure in the entire structural unit, and thus matches the corresponding motion control information for the entire structural unit.

[0061] In another exemplary embodiment of determining the optimal combination path, the multiple motion control information corresponding to each music measure within each structural unit can be used as units. All these units in all music measures are then subjected to a full permutation and combination operation at the data level to obtain a full set of combination paths. Then, each combination path is evaluated to determine the overall loss cost of each path. Finally, the optimal combination path is searched based on the lowest overall loss cost. The evaluation algorithm can aim to achieve the best possible expression of the emotion and rhythm of the corresponding structural unit with optimal motion effects, and can be flexibly determined as needed. Specifically, this can be broken down into the following two steps: Step S1311: For each combination path of the structural unit, evaluate the overall loss cost of the dance animation corresponding to each combination path; Step S1312: Determine the combination path with the minimum overall loss cost as the optimal combination path for the corresponding structural unit.

[0062] Step S1400: The motion control information contained in the optimal combination path of the structural unit is used to form the control information unit corresponding to the structural unit, which is used to control the digital human to generate corresponding dance animation.

[0063] Once the optimal combination path for each structural unit is determined, the motion control information for each musical measure within that structural unit is also determined. This motion control information can be spliced ​​together in the order of its corresponding musical measures to form the control information unit corresponding to that structural unit. This control information can be stored as the animation control file corresponding to that structural unit and used to control the digital human to generate corresponding dance animations.

[0064] By extension, for a piece of music, by organizing the control information units of multiple structural units in an orderly manner according to the temporal relationship of the structural units in its structural information, all the motion control information of the music can be obtained. This information can be stored as an animation control file corresponding to the entire piece of music and used to control digital humans to produce corresponding dance animations.

[0065] All motion control information determined for a structural unit or a piece of music can be applied to a digital human in real time, causing it to move synchronously with the music and rendering corresponding image frames, which are then included in the video stream and pushed to the terminal device for display. Alternatively, it can be stored as animation control files, which are then called when needed in specific business scenarios to drive the digital human to generate corresponding actions, thereby obtaining image frames and video streams that are pushed to the terminal device for display.

[0066] As can be seen from the above embodiments, this application has multiple technical advantages, including but not limited to:

[0067] First, this application performs structural and rhythmic analysis on the audio data of the music, dividing the audio data into multiple musical measures according to structure and rhythm. Based on the audio data of each musical measure, multiple candidate audio segments are selected from a pre-selected material database using semantic matching. Then, the corresponding preset motion control information of these candidate audio segments is used to construct a control information unit corresponding to each structural unit, which can be used to control the digital human. Since the materials in the material database are pre-selected, high-quality motion control information corresponding to candidate audio segments can be matched. This ensures that when the control information unit is used to control the digital human to generate corresponding dance animations, the corresponding dance animation movements are smooth, and the emotions expressed and the rhythm implemented can achieve a high-quality matching effect with the music, avoiding abnormal phenomena such as clipping and stuttering in the resulting dance animation.

[0068] Secondly, after matching multiple similar candidate audio segments for each music measure to determine each candidate audio segment, this application will perform full permutation and combination of the candidate audio segments of each music measure in each structural unit as a unit to obtain multiple combination paths. Then, based on the full number of combination paths, the best combination path will be selected to achieve the optimal selection of combination paths and ensure that the dance animation obtained accordingly has the best quality.

[0069] Furthermore, this application obtains the motion control information required for dance animation by selectively retrieving existing motion control information from the material database through semantic matching. This method involves low computational load and does not require excessive reliance on neural network models to perform a large amount of computation to determine the motion control information. This also saves on model training costs and sample preparation costs, making it more economical and cost-effective.

[0070] Based on any embodiment of this application, please refer to Figure 4 Acquire music, determine the audio segments corresponding to each musical measure within the structural unit of the music, including:

[0071] Step S1110: Obtain the audio data of the music;

[0072] The music can be specified by a user, such as a streamer in a live broadcast room, or it can be called one by one during the batch processing of music in the music library. Once the music is determined, its corresponding audio data can be retrieved from the music library. To facilitate standardized processing, the audio data can be converted to a specific format if necessary, such as a waveform file with the .wav extension.

[0073] Step S1120: Perform structural analysis on the audio data of the music to obtain its structural information, the structural information including the time node information of each structural unit of the music;

[0074] Based on the audio data of the music, structural analysis can be performed to obtain the structural information of the music. Of course, before performing structural analysis, the audio features of the music's audio data are usually extracted and encoded for further processing. These audio features can be constant Q transform features (CQT), Mel-spectral features, chromatographic features, etc., which can be selected as needed by those skilled in the art.

[0075] When performing music structure analysis, the start and end points of similar segments in the audio data of the music can be determined first to segment the music and identify multiple musical segments. Then, each musical segment is labeled to indicate which structural unit it belongs to, thereby identifying specific structural units such as the intro, verse, chorus, bridge, and outro of a piece of music. If the music has already been segmented at this stage, subsequent segmentation into measures can be directly based on these musical segments.

[0076] Specifically, in one implementation of music structure analysis, orthogonal linear discriminant analysis (OLDA) can be used to segment the audio data of the music. Then, two-dimensional Fourier magnitude coefficients (FMC2D) can be used to label the audio segments on the product, thereby distinguishing the various structural units of the music, and thus determining their corresponding time node information.

[0077] In another embodiment of music structure analysis, machine learning algorithms are used, specifically clustering algorithms such as K-Means. Through clustering algorithms, the audio features of the audio data are clustered to find the most frequently repeated frames or segments, and their corresponding time node information, namely the start and end timestamps, is determined. These can then be identified as the structural units corresponding to the chorus. After determining the chorus, based on the structural characteristics of the song, the time node information of the preceding verses, as well as the intro, outro, bridge, and other structural units, can be inferred.

[0078] In another embodiment, an autocorrelation algorithm can be used to perform music structure analysis. The idea is to match a music segment in a song with the whole song. The best matching segment can be considered the most representative segment in the song, thus identifying the structural unit corresponding to the chorus. In this way, all structural units of the whole song can be inferred, and the corresponding time node information can be obtained.

[0079] Step S1130: Perform rhythm analysis on the audio data of the music to obtain its rhythm information, the rhythm information including the drum beat information of the music;

[0080] Based on the audio data of the music, rhythm analysis can be performed to obtain its corresponding rhythm information. As mentioned earlier, rhythm information mainly provides the timestamps of each drumbeat in the music, that is, the drumbeat information. To facilitate rhythm analysis, the music data can also be encoded by extracting audio features according to specific needs, and then the analysis can be performed based on the encoded information.

[0081] In one embodiment of rhythm analysis, after representing and reasoning the audio feature encoding information of the music's audio data based on a deep learning model such as RNNDownbeatProcessor, the rhythm information in the music can be accurately detected, thereby enabling direct and rapid acquisition of the music's rhythm information in an end-to-end manner.

[0082] In another embodiment, various traditional detection algorithms can also be used to determine the rhythm information of the music, such as beat detection algorithms based on spectral flux, audio rhythm detection algorithms (Onset Detection), etc.

[0083] Step S1140: Based on the time node information of each structural unit and its corresponding drum beat information, determine each musical measure within each structural unit, and segment the audio segments corresponding to each musical measure from the audio data.

[0084] After determining the structural and rhythmic information of the music, the audio data of each structural unit can be segmented based on the time node information of each structural unit in the structural information. Furthermore, for each structural unit, the audio data between two adjacent drum beats can be independently segmented as an audio segment corresponding to a musical measure based on the drum beat information in the rhythmic information. Thus, the corresponding audio segment is determined for each musical measure in each structural unit of the music, refining the audio data of the entire music to the granularity of musical measures.

[0085] As can be seen from the above embodiments, by performing structural and rhythmic analysis on the music, and then segmenting the audio data of the music according to the corresponding structural and rhythmic information, down to the granularity of music bars, the corresponding audio segments of each music bar in each structural unit are determined. This makes it easier to match similar audio segments in the material database on a bar-by-bar basis to further obtain pre-made motion control information. Since each audio bar usually corresponds to a basic choreography movement, this actually refines the acquisition of motion control information to the granularity of basic movements, making it possible to perform machine choreography based on pre-made basic movements.

[0086] Based on any embodiment of this application, please refer to Figure 5 The system matches candidate audio segments similar to the stated audio segment from the material database and obtains preset motion control information corresponding to each candidate audio segment, including:

[0087] Step S1210: Calculate the semantic similarity between the deep semantic representation of the audio segments of each music measure and the deep semantic representation of the audio segments pre-stored in the material database;

[0088] As mentioned earlier, each audio segment in the material database is determined according to the duration of a musical measure, and its corresponding deep semantic representation is pre-determined and stored using a feature extraction model. Similarly, for each audio segment within each musical measure of each structural unit in the music, its corresponding deep semantic representation can also be obtained using the same feature extraction model. These deep semantic representations are all represented in vector form. Therefore, any data distance algorithm can be used to calculate the vector-to-vector semantic similarity between the deep semantic representation of a musical segment in the music and the deep semantic representation of each musical segment in the material database.

[0089] For example, taking the cosine similarity algorithm used in the data distance algorithm as an example, the semantic similarity of vectors x and y can be directly calculated and determined according to the following formula:

[0090]

[0091] It's easy to understand that the greater the semantic similarity, the more similar the two audio segments are; otherwise, the less similar they are.

[0092] Step S1220: For each music measure's audio segments, select multiple audio segments from the material database as candidate audio segments based on the semantic similarity.

[0093] It is not difficult to understand that, relative to the audio segment A of each musical measure in the music, i For example, each audio segment B in the material database j All can obtain the corresponding semantic similarity, but for each audio segment A i In this case, it is only necessary to select a subset of candidate audio segments. Therefore, audio segment A can be selected based on a preset number or a preset threshold. i Multiple candidate audio data were selected from the material database.

[0094] In one embodiment, to meet the need for filtering using a preset number K, for audio segment A i First, based on the corresponding semantic similarity, each audio segment B in the material database is sorted from largest to smallest. j Sort the audio segments, then extract the top K audio segments B from the sorted list. j As the audio segment A i The candidate audio segments are thus used to constitute the audio segment A. i Corresponding candidate audio segment list ik .

[0095] In another embodiment, to meet the need for filtering using a preset threshold, for audio segment Ai Using the threshold and each audio segment B in the material database j The semantic similarity is compared to obtain all audio segments B whose semantic similarity is higher than the threshold. j Kist constitutes the candidate audio segment list ik .

[0096] Step S1230: Obtain preset action control information corresponding to each candidate audio segment from the material database to form a candidate action list for the corresponding music measure.

[0097] After the above processing, a list of candidate audio segments corresponding to each musical measure in each structural unit of the music is obtained. ik Because each audio segment B in the material database j Each candidate audio segment list is pre-provided with corresponding motion control information that can be used to drive the data human to generate dance animations. Therefore, the candidate audio segment list actually represents the candidate motion control information list, or simply the candidate motion list, which can be denoted as Motions. ik To distinguish them. Of course, the candidate action list includes Motions. ik Alternatively, it may simply include the identifiers of the motion control information for each candidate audio segment to simplify the representation without affecting the inventive spirit of this application.

[0098] As can be seen from the above embodiments, by using the material database, candidate audio segments are first matched for each musical measure in the music, and then the matching motion control information is determined based on the candidate audio segments, thus realizing cross-modal mapping. Since the audio segments and their motion control information in the material database are pre-made, the motion control information drives the digital human to generate corresponding dance animations with good image quality and no abnormal phenomena such as clipping or stuttering. Moreover, the dance animations and their corresponding audio segments are coordinated and correspond in terms of emotion and rhythm. Therefore, it can be ensured that the candidate motion control information obtained for each musical measure is also a high-quality resource that is more coordinated and consistent with the audio data of that musical measure in terms of emotion and rhythm, thereby ensuring that a high-quality work can be obtained when producing dance animations for the music in the future.

[0099] Based on any embodiment of this application, please refer to Figure 6 The complete combination path is formed by the ordered full arrangement of the motion control information of all musical measures within the structural unit. The optimal combination path for the structural unit is searched within this complete combination path, including:

[0100] Step S1321: For each combined path, determine the deep semantic representation of the audio segment belonging to the music and the deep semantic representation of the audio segment belonging to the material database, and use the similarity loss value as the first loss cost.

[0101] After determining the full combination path for each structural unit, each combination path corresponds to each musical measure within the structural unit, pointing to an audio segment A belonging to the music. i This audio segment A i The audio segment B belonging to the material database that corresponds to this music measure in the combined path. j The semantic similarity loss value between the two can measure the similarity loss cost between them. Therefore, the similarity loss value corresponding to each musical measure in each combination path can be determined by the following formula, which serves as the first loss cost λ. cons The formula is shown below:

[0102] λ cons =1-cos_sim(m i ,m o )

[0103] Where, m i This indicates the audio segment A corresponding to the musical measure. i Deep semantic representation, m o This indicates the audio segment B belonging to the material database corresponding to the music measure. j The deep semantic representation of .

[0104] In an embodiment that evaluates the entire combination path to determine a total first loss cost, the first loss costs corresponding to each music measure can be aggregated to represent the total loss cost. In an embodiment based on dynamically searching for the optimal motion control information for each music measure, the first loss cost corresponding to each candidate audio segment can be calculated for each music measure, and then aggregated into a total loss cost. This total loss cost is then used to filter out the optimal motion control information corresponding to the next music measure, and so on, to help determine an optimal combination path.

[0105] Step S1322: For each combined path, determine the similarity loss value between the last information frame in the earlier action control information and the first information frame in the later action control information in each pair of adjacent action control information, and use it as the second loss cost.

[0106] As mentioned earlier, each motion control information contains multiple information frames corresponding to the generation of multiple image frames. Each information frame determines the motion state generated by the digital human. It is easy to understand that, in two temporally adjacent music bars within the same combination path, the closer the semantics of the last information frame in the motion control information corresponding to the first music bar (which is temporally earlier) and the first information frame in the motion control information corresponding to the second music bar (which is temporally later), the more coherent the two are, and the more natural the generated motion effect is. Conversely, the effect is worse. To measure the effect of both, this effect can be described as the second loss cost λ. trans The following formula is used for calculation:

[0107] λ trans =1-cos_sim(d i ,d i+1 )

[0108] Where, d i This indicates the last information frame in the motion control information corresponding to the music measure with the earlier timing sequence, d i+1 It follows d in time sequence. i What follows is actually the first information frame in the motion control information corresponding to the music measure that comes later in the time sequence.

[0109] It can be seen that by calculating the semantic similarity between two adjacent information frames, a second loss cost can be obtained to measure the smoothness of the connection between the action control information of two consecutive music bars.

[0110] In an embodiment that determines a total second loss cost by evaluating the entire combination path, the second loss costs corresponding to each musical measure can be aggregated to represent the total loss cost. In an embodiment based on dynamically searching for the optimal motion control information for each musical measure, the second loss cost between each candidate motion control information for the next musical measure and its previous musical measure can be calculated one by one, and then aggregated into a total loss cost. This total loss cost is then used to filter out the optimal motion control information corresponding to the next musical measure, and so on, to help determine an optimal combination path.

[0111] Step S1325: Integrate multiple loss costs, including the first loss cost and the second loss cost of each combined path, to obtain the total loss cost of the combined path, and determine the combined path corresponding to the lowest total loss cost as the optimal combined path.

[0112] In the embodiment of dynamically searching to determine the optimal combination path, after determining the first loss cost and the second loss cost corresponding to each action control information of the next musical measure for each musical measure in the corresponding structural unit, the total loss cost of each action control information of the next musical measure can be determined. The exemplary formula for calculating the total loss cost C is as follows:

[0113] C=α1*λ cons +α2*λ trans

[0114] Among them, α1 and α2 are weight parameters that can be flexibly set as needed.

[0115] Based on the above formula, for each musical measure in a structural unit, the overall loss cost of splicing each motion control information with the previous musical measure can be determined. It's easy to understand that the motion control information corresponding to the lowest overall loss cost is the one with the best splicing effect, because its corresponding audio segment is semantically closest to the audio segment belonging to the music pointed to by the musical measure, and its first information frame is also closest to the last information frame in the motion control information of the previous musical measure. Using this dynamic search method, the best motion control information is searched for one by one for each musical measure in the structural unit. The path formed by splicing these best motion control information is the optimal combination path.

[0116] In the embodiment of comprehensively evaluating the overall loss cost of each combination path, the method of determining each specific loss cost is the same as described above, except that the calculation unit is different. That is, based on each combination path, the specific loss cost corresponding to each music measure is calculated and then summarized into the first loss cost, second loss cost, etc. corresponding to each combination path. Finally, all specific loss costs are summarized to obtain the overall loss cost corresponding to each combination path, and then the combination path corresponding to the lowest overall loss cost is determined as the optimal combination path.

[0117] It should be noted that for each structural unit, the first musical measure does not have a preceding musical measure. When applying the above formula to calculate various specific loss costs, the initial value can be used instead.

[0118] As can be seen from the above embodiments, when constructing the motion control information corresponding to each structural unit of the music, the various loss costs that can be obtained from the multiple motion control information corresponding to each musical measure in the structural unit are calculated, and the optimal combination path with the lowest overall loss cost is evaluated. This achieves the goal of the structural unit selecting the most similar set of motion control information from the material database, so that the corresponding structural unit can drive the digital human to generate the corresponding dance animation. By extension, the entire piece of music can also be obtained by splicing the motion control information sets of all structural units according to its structural information. Since the continuity of motion control information between adjacent musical measures and the similarity of audio semantics are fully considered in the process of determining the optimal combination path, after the obtained set of motion control information is used to drive the digital human to generate the corresponding dance animation, the dance animation movements are more coherent and the emotions and rhythms are more coordinated and unified with the music.

[0119] Based on any embodiment of this application, please refer to [further details]. Figure 6 Before determining the optimal combined path by aggregating multiple loss costs, including the first and second loss costs of each combined path, to obtain the total loss cost of the combined path, the following steps S1323 and / or S1324 can be set as follows:

[0120] In one embodiment, in order to ensure the diversity of dance, the selection of dance movements should be as diverse as possible. Therefore, the following steps can be set: Step S1323: For each combination path, count the number of repetitions of the dance movements indicated by the information frames in all its motion control information, and use it as the third loss cost fused into the overall loss cost.

[0121] Specifically, a cost function for repetitive dance movements can be added to calculate the third loss cost λ corresponding to the control information of each movement in each musical measure. repe As shown below:

[0122] λ repe =count({d i-P+1 ,d i-P+2 ,…,d i-1},d i )

[0123] The count function counts the number of times an action is repeated, and P is the window size for counting the repeated actions.

[0124] As can be seen, by sliding the window, we can count the number of information frames that indicate repeated actions of the digital human in the structural unit after each action control information of the current music measure is combined with the previously determined action control information. That is, the number of action repetitions. The higher the number, the worse the diversity, and the lower the number, the richer the diversity.

[0125] In an embodiment that evaluates the entire combination path to determine a total third loss cost, the third loss cost corresponding to each musical measure can be summarized to represent it. In an embodiment based on dynamically searching for the optimal motion control information for each musical measure, the third loss cost can be calculated by combining each motion control information in the next musical measure with all previous motion control information and summarizing it into a total loss cost. This total loss cost is then used to filter out the optimal motion control information corresponding to the next musical measure, and so on, to help determine an optimal combination path.

[0126] With the introduction of the third loss cost, the formula for the total loss cost C corresponding to a combined path can be transformed as follows:

[0127] C=α1*λ cons +α2*λ trans +α3*λ repe

[0128] Similarly, α1, α2, and α3 are all empirical parameters used as weights.

[0129] It can be seen that the settings in step S1323 can further enrich the diversity of dance movements and make the obtained dance animation effect better.

[0130] In another embodiment, step S1324 involves calculating the distance between the dance position indicated by the information frame in all motion control information of each combined path and the preset midpoint, as the fourth loss cost to be fused into the overall loss cost.

[0131] Generally, dance involves movement around a central point. Therefore, we add a movement constraint to the dance path, limiting the digital human's range of motion to a specific area. This also encourages the system to search for symmetrical dance units to offset the overall range of motion. Accordingly, for each motion control information in the next musical measure, the fourth loss cost λ is calculated using the following formula. range :

[0132]

[0133] Taking an embodiment that uses Euler angles to represent the motion control information of a digital human as an example, in the information frame of the motion control information, there exists an offset amount representing the digital human's deviation from a preset midpoint. This offset amount can then be used as Distance(d j The fourth loss cost is obtained by combining the motion control information of each movement in the next music measure with the motion control information of each previously determined music measure, and then calculating the offset of the information frames in the combined motion control information. This offset represents the distance the digital human's dance position deviates from the preset midpoint. It is easy to understand that a larger fourth loss cost indicates a larger overall distance from the midpoint.

[0134] In an embodiment that evaluates the entire combination path to determine a total fourth loss cost, the fourth loss cost corresponding to each musical measure can be summarized to represent it. In an embodiment based on dynamically searching for the optimal motion control information for each musical measure, the fourth loss cost can be calculated by combining each motion control information in the next musical measure with all previous motion control information and summarizing it into a total loss cost. This total loss cost is then used to filter out the optimal motion control information corresponding to the next musical measure, and so on, to help determine an optimal combination path.

[0135] With the introduction of the fourth loss cost, the formula for the total loss cost C corresponding to a combined path can be transformed as follows:

[0136] C=α1*λ cons +α2*λ trans +α4*λ range

[0137] Similarly, α1, α2, and α4 are all empirical parameters used as weights.

[0138] It can be seen that the setting of step S1324 can make the dance animation obtained under the selected motion control information have good stage symmetry, thus making the dance effect better.

[0139] In another embodiment, steps S1323 and S1324 are introduced simultaneously, and the calculation formula for the overall loss cost is changed accordingly as follows:

[0140] C=α1*λ cons +α2*λ trans +α3*λ repe +α4*λ range

[0141] It is evident that by simultaneously introducing multiple specific loss costs to evaluate and determine the overall loss cost corresponding to the combined path, we can fully consider various factors related to dance aesthetics, such as the emotional similarity, smoothness, diversity, and symmetry between dance movements, to obtain the optimal combined path for each structural unit. This allows us to select the best motion control information corresponding to each musical measure to generate high-quality dance animation.

[0142] Based on any embodiment of this application, please refer to Figure 7 After constructing the action control information unit corresponding to the structural unit from the optimal combination path of the structural unit, the following is included:

[0143] Step S1500: Based on the sorting relationship of the structural units in the structural information of the music, construct the control information units of each structural unit into an animation control file;

[0144] As mentioned earlier, after determining the set of motion control information corresponding to each structural unit, these sets of motion control information constitute the control information unit corresponding to the structural unit. By organizing these control information units in an orderly manner, an animation control file corresponding to the entire piece of music can be formed.

[0145] Specifically, since the structural information of the music contains time node information about the ordering relationship of its various structural units, this relationship can be used to sequentially splice together the corresponding control information units of each structural unit according to their order. This yields all the motion control information corresponding to all the musical measures of a piece of music. All the motion control information can be stored as an animation control file, which can be used to drive a digital human to generate corresponding dance animations.

[0146] Step S1600: For the animation control file, perform frame interpolation processing on the last information frame in the motion control information of the structure unit with the earlier timing and the first information frame in the motion control information of the structure unit with the immediately following timing.

[0147] When generating the animation control file, considering that in two adjacent structural units, there is one information frame connected one after the other, that is, between the last information frame of the structural unit with the earlier timing and the first information frame of the structural unit with the later timing, the dance movements indicated by each other may not be able to transition completely smoothly. In order to make the connection between the two smoother, frame interpolation can be performed based on such two connected information frames.

[0148] Taking the case of using Euler angles to represent the information frame of a digital human as an example, considering that Euler angles cannot be used for interpolation, quaternion interpolation is used to splice and synthesize adjacent dance units. The specific operation can be carried out as follows:

[0149] 1) Transfer the information frame from Euler angles The formula for converting to quaternion (w,x,y,z) is as follows:

[0150]

[0151]

[0152]

[0153]

[0154] 2) For the last frame d of the adjacent structural unit i and the first frame d i+1 Frame interpolation is used to make the transition animation smooth and fluid. The frame interpolation method can be spherical linear interpolation, and the number of interpolated frames depends on the difference in motion between the two frames that need to be transitioned. The difference in motion between the two frames is obtained by calculating the distance between the quaternions (a, b) of the two frames, as shown below:

[0155] c = a * b -1

[0156] quat_distance = arccos(cw)

[0157] The formula for spherical linear interpolation is expressed as follows:

[0158]

[0159] Where θ is the angle between a and b, and t is the percentage position where interpolation is needed.

[0160] 3) Convert quaternions to Euler angles. After quaternion interpolation, a smooth dance can be obtained. Converting to Euler angles makes it easier to render using tools such as Unity. The formula for converting quaternions to Euler angles is:

[0161]

[0162] θ=arcsin(2(wy-zx))

[0163]

[0164] Therefore, frame interpolation can make the transition between two adjacent structural units more natural.

[0165] Step S1700: For two structural units in the animation control file that have the same properties but different times, process the motion control information of the two structural units into a repeating relationship or a mirror relationship.

[0166] To ensure that the dance and music share the same structure, for structural units of the same nature in the music, such as verses or choruses, identical control information units can be used for structural units of the same nature appearing at different positions in the temporal sequence of the music. Alternatively, the motion control information of the same structural unit appearing at the first and second positions can be processed into a mirror image relationship. That is, the motion represented by the information frame in the motion control information appearing at the second position is transformed by transforming the vectors in the information frame, so that the motion is converted into a mirror image of the motion represented by the information frame in the motion control information appearing at the first position. This gives the generated dance a certain structural aesthetic. Of course, for two structural units of the same nature but at different times, identical control information units can also be used to make the generated dance animation perform completely repetitive actions.

[0167] Step S1800: Drive the digital human to generate corresponding dance animations according to the animation control file.

[0168] After splicing the various structural units of the music according to its structural information, complete motion control information corresponding to the music is obtained. This motion control information is stored as an animation control file, which is then associated with the music. When a corresponding digital human dance animation needs to be generated for the music, the animation control file is simply called, and each information frame is applied to the corresponding digital human to control the digital human to perform the corresponding movements. Then, the corresponding image frames are rendered to obtain the corresponding motion image. By generating corresponding image frames one by one for each information frame in the animation control file, the ordered encoding results of these image frames form a video stream that can be pushed to terminal devices such as live streaming platforms for playback.

[0169] As can be seen from the above embodiments, based on the structural information of the music, the control information units generated by this application for each structural unit are organically spliced ​​together, and frame interpolation is performed as needed to compile the animation control file of the music. This animation control file can be provided in association with the music and used to generate corresponding digital human dance animations for the music. Moreover, the animation control file can be reused by multiple digital humans, so that when the machine choreographs the dance animation of the music, it can obtain a dance animation work with smooth movements, natural emotions, and coordinated rhythm. Figure 8 The image content of several image frames in a dance animation generated by a digital human driven by the animation control file according to this application is shown for reference.

[0170] Please see Figure 9According to one aspect of this application, an animation control information construction device includes a format processing module 1100, a material matching module 1200, a material selection module 1300, and a unit construction module 1400. The format processing module 1100 is configured to acquire music and determine the audio segments corresponding to each musical measure within a structural unit of the music. The material matching module 1200 is configured to match candidate audio segments similar to the audio segments of each musical measure from a material database, and obtain preset motion control information corresponding to each candidate audio segment. The motion control information includes multiple information frames for controlling a digital human to generate corresponding dance movements. The material selection module 1300 is configured to form a full combination path by orderly arranging the motion control information of all musical measures within the structural unit, and search for the optimal combination path of the structural unit within the full combination path. The unit construction module 1400 is configured to construct a control information unit corresponding to the structural unit from the motion control information contained in the optimal combination path of the structural unit, for controlling the digital human to generate corresponding dance animation.

[0171] Based on any embodiment of this application, the format processing module 1100 includes: a data acquisition submodule, configured to acquire the audio data of the music; a structure analysis submodule, configured to perform structure analysis on the audio data of the music to obtain its structure information, the structure information including the time node information of each structural unit of the music; a rhythm analysis submodule, configured to perform rhythm analysis on the audio data of the music to obtain its rhythm information, the rhythm information including the drum beat information of the music; and a segmentation submodule, configured to determine each musical measure within each structural unit according to the time node information of each structural unit and its corresponding drum beat information, and segment the audio segments corresponding to each musical measure from the audio data.

[0172] Based on any embodiment of this application, the material matching module 1200 includes: a similarity calculation submodule, configured to calculate semantic similarity based on the deep semantic representation of the audio segments of each music measure and the deep semantic representation of the audio segments pre-stored in the material database; a segment filtering submodule, configured to filter multiple audio segments in the material database as candidate audio segments for each music measure based on the semantic similarity; and a segment calling submodule, configured to obtain preset action control information corresponding to each candidate audio segment from the material database to form a candidate action list for the corresponding music measure.

[0173] Based on any embodiment of this application, the material selection module 1300 includes: a cost evaluation submodule, configured to evaluate the overall loss cost of the dance animation corresponding to each combination path for each combination path of the structural unit; and a path selection submodule, configured to determine the combination path with the minimum overall loss cost as the optimal combination path for the corresponding structural unit.

[0174] Based on any embodiment of this application, the material selection module includes: a first evaluation submodule, configured to, for each combined path, determine the similarity loss value between the deep semantic representation of the corresponding audio segment belonging to the music and the deep semantic representation of the corresponding audio segment belonging to the material database, as a first loss cost; a second evaluation submodule, configured to, for each combined path, determine the similarity loss value between the last information frame in the earlier action control information and the first information frame in the later action control information, as a second loss cost; and a fusion evaluation submodule, configured to, fuse multiple loss costs including the first and second loss costs of each combined path, to obtain the total loss cost of the combined path, and determine the combined path corresponding to the lowest total loss cost as the optimal combined path.

[0175] Based on any embodiment of this application, the material selection module further includes the following third evaluation submodule and / or fourth evaluation submodule, wherein the third evaluation submodule is configured to, for each combined path, count the number of repetitions of dance movements indicated by information frames in all its motion control information, as the third loss cost fused into the overall loss cost; the fourth evaluation submodule is configured to, for each combined path, count the distance of the dance position indicated by information frames in all its motion control information from the preset midpoint, as the fourth loss cost fused into the overall loss cost.

[0176] Based on any embodiment of this application, the animation control information construction device of this application includes: a file construction module, configured to construct an animation control file by constructing the control information units of each structural unit according to the sorting relationship of the structural units in the structural information of the music; a smoothing processing module, configured to perform frame interpolation processing on the last information frame of the motion control information of the structural unit with the earlier timing and the first information frame of the motion control information of the structural unit with the immediately following timing in the animation control file; a symmetry optimization module, configured to process the motion control information of two structural units with the same nature but different timing in the animation control file into a repetitive relationship or a mirror relationship; and an animation generation module, configured to drive a digital human to generate a corresponding dance animation according to the animation control file.

[0177] Another embodiment of this application also provides an animation control information construction device. For example... Figure 10 The diagram shows the internal structure of an animation control information construction device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement an animation control information construction method.

[0178] The processor of the animation control information construction device provides computing and control capabilities to support the operation of the entire device. The memory of the device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the animation control information construction method of this application. The network interface of the device is used for communication with a terminal.

[0179] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the animation control information construction device applied thereto. The specific animation control information construction device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In this embodiment, the processor is used to execute... Figure 9 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the animation control information construction device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0181] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the animation control information construction method of any embodiment of this application.

[0182] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0184] In summary, this application employs a low-cost implementation method to generate the motion control information required for music-driven digital humans to produce dance animations, and makes the movements of the digital humans in the generated dance animations relatively smooth and coordinated with the emotions and rhythm of the music.

Claims

1. A method for constructing animation control information, characterized in that, include: Acquire music, perform structural analysis and rhythm analysis on the audio data of the music, and combine the time node information of each structural unit obtained by the structural analysis with the drum beat information obtained by the rhythm analysis to divide and determine the audio segments corresponding to each musical measure within the structural unit of the music; Semantic similarity is calculated from the deep semantic representation of the audio segments of each music measure in the material database and the deep semantic representation of the audio segments stored in the material database. Candidate audio segments that are similar to the audio segments of each music measure are selected, and preset motion control information corresponding to each candidate audio segment is obtained. The motion control information includes multiple information frames for controlling the digital human to produce corresponding dance movements. The total combination path is formed by the orderly arrangement of the motion control information of all music measures in the structural unit. For each combination path of the structural unit, the total loss cost of the dance animation corresponding to each combination path is evaluated, and the combination path with the minimum total loss cost is determined as the optimal combination path of the corresponding structural unit. The motion control information contained in the optimal combination path of the structural unit is used to form the corresponding control information unit of the structural unit, which is used to control the digital human to generate corresponding dance animation.

2. The animation control information construction method according to claim 1, characterized in that, Acquire music, and perform structural and rhythmic analysis on the audio data of the music, including: Obtain the audio data of the music; Structural analysis is performed on the audio data of the music to obtain its structural information, which includes the time node information of each structural unit of the music. The audio data of the music is subjected to rhythm analysis to obtain its rhythm information, which includes the drum beat information of the music; Based on the time node information of each structural unit and its corresponding drum beat information, each musical measure within each structural unit is determined, and the corresponding audio segments of each musical measure are segmented from the audio data.

3. The animation control information construction method according to claim 1, characterized in that, Semantic similarity is calculated from the deep semantic representation of each music bar's audio segment in the material database and the deep semantic representation of pre-stored audio segments in the material database. Candidate audio segments similar to the audio segments are selected, and preset motion control information corresponding to each candidate audio segment is obtained, including: Semantic similarity is calculated based on the deep semantic representation of the audio segments of each music measure and the deep semantic representation of the audio segments pre-stored in the material database; For each music measure's audio segments, multiple audio segments from the material database are selected as candidate audio segments based on the semantic similarity. The preset action control information corresponding to each candidate audio segment is obtained from the material database to form a candidate action list for the corresponding music measure.

4. The animation control information construction method according to claim 1, characterized in that, For each combined path of the structural unit, evaluate the loss cost of the dance animation corresponding to each combined path, including: For each combined path, the deep semantic representation of the corresponding audio segment belonging to the music is determined, and the similarity loss value between it and the deep semantic representation of the corresponding audio segment belonging to the material database is determined, which is used as the first loss cost. For each combined path, the similarity loss value between the last information frame in the earlier action control information and the first information frame in the later action control information is determined as the second loss cost. By aggregating multiple loss costs, including the first and second loss costs of each combined path, the total loss cost of the combined path is obtained, and the combined path corresponding to the lowest total loss cost is determined as the optimal combined path.

5. The animation control information construction method according to claim 4, characterized in that, Before fusing multiple loss costs, including the first and second loss costs of each combined path, to obtain the total loss cost of the combined path, and determining the combined path corresponding to the lowest total loss cost as the optimal combined path, the process includes any one or more of the following steps: For each combined path, the number of repetitions of dance movements indicated by information frames in all its motion control information is counted and used as the third loss cost fused into the overall loss cost. For each combined path, the distance of the dance position indicated by the information frame in all its motion control information from the preset midpoint is counted and used as the fourth loss cost to be integrated into the overall loss cost.

6. The method for constructing animation control information according to any one of claims 1 to 5, characterized in that, After constructing the corresponding control information unit for the structural unit from the motion control information contained in the optimal combination path of the structural unit, it includes: Based on the ordering relationship of structural units in the structural information of the music, the control information units of each structural unit are constructed into an animation control file; For the animation control file, frame interpolation is performed on the last information frame of the motion control information of the structure unit with the earlier timing and the first information frame of the motion control information of the structure unit with the immediately following timing. For two structural units in the animation control file that have the same properties but different times, the motion control information of the two structural units is processed into a repetitive relationship or a mirror relationship; The digital human is driven to generate corresponding dance animations based on the animation control file.

7. An animation control information construction device, characterized in that, include: The format processing module is configured to acquire music, perform structural analysis and rhythm analysis on the audio data of the music, and use the time node information of each structural unit obtained by the structural analysis and the drum beat information obtained by the rhythm analysis to divide and determine the audio segments corresponding to each musical measure within the structural unit of the music. The material matching module is configured to calculate semantic similarity from the material database based on the deep semantic representation of the audio segments of each music bar and the deep semantic representation of the audio segments pre-stored in the material database, filter out candidate audio segments similar to the audio segments of each music bar, and obtain preset motion control information corresponding to each candidate audio segment. The motion control information includes multiple information frames for controlling the digital human to produce corresponding dance movements. The material selection module is configured to form a full combination path by the orderly arrangement of the motion control information of all music measures within the structural unit. In the full combination path, for each combination path of the structural unit, the overall loss cost of the dance animation corresponding to each combination path is evaluated, and the combination path with the minimum overall loss cost is determined as the optimal combination path of the corresponding structural unit. The unit construction module is configured to construct the motion control information contained in the optimal combination path of the structural unit into a corresponding control information unit for controlling the digital human to generate corresponding dance animations.

8. The animation control information construction device according to claim 7, characterized in that, The material selection module includes: The first evaluation submodule is configured to determine, for each combined path, the similarity loss value between the deep semantic representation of the audio segment belonging to the music and the deep semantic representation of the audio segment belonging to the material database, as the first loss cost. The second evaluation submodule is configured to determine, for each combined path, the similarity loss value between the last information frame in the earlier action control information and the first information frame in the later action control information in each pair of adjacent action control information, as the second loss cost. The fusion evaluation submodule is configured to fuse multiple loss costs, including the first and second loss costs of each combined path, to obtain the total loss cost of the combined path, and determine the combined path corresponding to the lowest total loss cost as the optimal combined path.

9. An animation control information construction device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

Images