Animation generation method, system and computer device

Abstract

The embodiment of the present application provides an animation generation method, system and computer device. In this embodiment, the target music is first divided into at least two music clips, and each of the music clips is input into a pre-trained music recognition model in turn for music feature recognition to obtain the music feature information of each of the music clips, and then the animation clips matching each of the music clips are searched from a pre-established animation library according to the music feature information of each of the music clips, and finally the animation corresponding to the target music is generated according to the animation clips matching each of the music clips. In this way, by using a large number of music-animation data sets, a deep learning model is used to extract music features, and a feature matching search algorithm is used to process animations, so that the entire animation matching and generation process can be controlled, the scalability of the animation generation method is improved, and the generated animation is more natural and reasonable, which can enhance the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence and animation processing, and more specifically, to an animation generation method, system and computer equipment. Background Art

[0002] With the continuous development and application of computer technology, computer-based animation production or creation has received widespread attention and application in various scenarios. In some specific application scenarios, it is necessary to automatically generate dance animations according to music. For example, in the application scenarios of music and dance games, it is often necessary to generate corresponding dance animations according to music.

[0003] The inventors have studied and analyzed some common solutions for generating dance animations based on music and found that most of the current animation generation methods have problems such as poor scalability and the generated animations are not natural and reasonable enough, which affects the user experience. Summary of the invention

[0004] Based on the above, in order to at least partially solve the above technical problems, in a first aspect, an embodiment of the present application provides an animation generation method, the method comprising:

[0005] Divide the target music into at least two music segments;

[0006] Inputting each of the music clips into a pre-trained music recognition model in turn to perform music feature recognition, thereby obtaining music feature information of each of the music clips, wherein the music feature information includes style information and rhythm information of the music clip;

[0007] Searching for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips;

[0008] Animation corresponding to the target music is generated according to the animation segments matching the music segments.

[0009] Based on an implementation manner of the first aspect, the method further includes a step of training a pre-selected deep learning model to obtain the music recognition model, the step comprising:

[0010] Establishing a sample music library, wherein the sample music library may include sample music of different styles;

[0011] Each piece of sample music in the sample music library is divided into a plurality of sample music segments, and corresponding music feature information is annotated for each sample music segment to obtain a music feature label for each sample music segment, wherein the music feature information corresponding to the sample music segment includes style information and rhythm information;

[0012] The sample music library is used to train a pre-selected deep learning model to obtain a music recognition model for recognizing music feature information of music.

[0013] According to an implementation of the first aspect, a pre-selected deep learning model is trained using the sample music library to obtain a music recognition model for recognizing music feature information of music, including:

[0014] Inputting each of the sample music clips into the deep learning model in turn, and using the deep learning model to identify and obtain music feature information of the sample music clips;

[0015] Calculating the loss function value of the deep learning model based on the identified music feature information of the sample music clip and the music feature label of the sample music clip;

[0016] The model parameters of the deep learning model are iteratively optimized according to the loss function value until the model training termination condition is met, and the trained deep learning model is used as the music recognition model.

[0017] Based on an implementation manner of the first aspect, the method further includes a step of pre-establishing the animation library, the step comprising:

[0018] Obtaining animations that match each piece of sample music in the sample music library;

[0019] Segmenting each of the animations to obtain a plurality of animation segments, wherein each animation segment has the same duration as a sample music segment of the sample music corresponding to the animation;

[0020] Performing feature annotation on each of the animation clips to obtain animation feature information corresponding to each of the animation clips, wherein the animation feature information includes style information and rhythm information of the animation clip;

[0021] Each animation clip marked with the animation feature information is saved in a set database to obtain the animation library.

[0022] In an implementation manner of the first aspect, searching for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips includes:

[0023] For any target music clip, searching the animation library for a plurality of animation clips matching the music clip according to the style information of the target music clip;

[0024] Searching at least one candidate animation segment matching the target music segment from the plurality of animation segments according to the rhythm information of the target music segment;

[0025] A target animation segment matching the target music segment is determined according to motion information of the at least one candidate animation segment, wherein the motion information includes position information, speed information, rotation speed and angular speed information for characterizing a target object in the candidate animation segment.

[0026] In an implementation manner based on the first aspect, determining a target animation segment matching the target music segment according to the motion information of the at least one candidate animation segment includes:

[0027] Obtaining the motion information of the last preset number of frames of the animation segment matched by the previous music segment of the target music segment as an index;

[0028] According to the index, one of the candidate animation clips whose motion information of the previous preset number of frames is closest to the index is searched as the target animation clip matching the target music clip.

[0029] According to an implementation of the first aspect, the method further includes:

[0030] The target animation clip is scaled by using a B-spline interpolation method so that the duration of the target animation clip is consistent with that of the target music clip.

[0031] In a second aspect, an embodiment of the present application further provides an animation generation system, which is applied to a computer device, and the animation generation system includes:

[0032] A segmentation module, used for segmenting the target music into at least two music segments;

[0033] A recognition module, used to sequentially input each of the music clips into a pre-trained music recognition model to perform music feature recognition, and obtain music feature information of each of the music clips, wherein the music feature information includes style information and rhythm information of the music clip;

[0034] A search module, used for searching for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips;

[0035] A generation module is used to generate animations corresponding to the target music according to the animation clips matching the music clips.

[0036] According to an implementation manner of the second aspect, the search module is specifically used for:

[0037] For any target music clip, searching the animation library for a plurality of animation clips matching the music clip according to the style information of the target music clip;

[0038] Searching at least one candidate animation segment matching the target music segment from the plurality of animation segments according to the rhythm information of the target music segment;

[0039] A target animation segment matching the target music segment is determined according to motion information of the at least one candidate animation segment, wherein the motion information includes position information, speed information, rotation speed and angular speed information for characterizing a target object in the candidate animation segment.

[0040] In a third aspect, an embodiment of the present application further provides a computer device, comprising a machine-readable storage medium and one or more processors, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the one or more processors, the above-mentioned animation generation method is implemented.

[0041] Based on the above content of the embodiment of the present application, relative to the prior art, the animation generation method, system and computer equipment provided by the embodiment of the present application, the animation generation method, system and computer equipment provided by the embodiment of the present application, by using a large amount of music-animation data sets, do not adopt an end-to-end music-animation generation model, but adopt a deep learning model (music recognition model) to extract music features, adopt a feature matching search algorithm to process the animation, can be corrected according to the user's ideas during the search process, realize the controllability of the entire animation matching and generation process, and the animation can be a high-quality animation found and corrected from the action library, and it can be ensured that the output animation is lossless. Therefore, the animation generation method in the present embodiment has good scalability, and the generated animation is also more natural and reasonable, which can enhance the user's experience. In addition, as the scale of the action library is gradually improved and becomes larger, the animation generation effect is correspondingly better. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flowchart of an animation generation method provided in an embodiment of the present application.

[0044] Figure 2 yes Figure 1 Schematic diagram of the implementation process of step S200.

[0045] Figure 3 yes Figure 1 Schematic diagram of the implementation process of step S300.

[0046] Figure 4 It is a schematic diagram of segmenting the target music provided in this embodiment.

[0047] Figure 5 It is a schematic diagram of scaling the matched animation clip provided in an embodiment of the present application.

[0048] Figure 6 It is a schematic diagram of the functional modules of the animation generation system provided in the embodiment of the present application.

[0049] Figure 7 It is a schematic diagram of a computer device for implementing the above-mentioned animation generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0053] In the description of the present application, the terms “first”, “second”, “third”, etc. are only used to distinguish the descriptions and should not be understood as indicating or implying relative importance.

[0054] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0055] Based on the technical problems mentioned in the aforementioned background technology, the inventor of the present application has discovered that there are currently two common ways to generate dance animations based on music. One is to use a music library and an action library, where the user selects a piece of music from the set music library, and then the system selects the corresponding dance animation from the action library based on the correspondence, thereby ensuring the consistency of the music and the dance animation in terms of duration, style, rhythm, etc. However, this solution must require that the music played by the user be included in the music library. Music that does not exist in the music library cannot be matched to the corresponding dance animation, and the scalability is poor. If new music needs to be added, the corresponding dance animation must be added simultaneously.

[0056] Another approach is to use deep learning methods to directly model the relationship between music and animation. This approach usually requires the collection of a large amount of paired music and animation data, training of deep learning models, and the goal of inputting music into the model and outputting animation. However, this approach has poor interpretability and controllability, and the generated animation is often not natural and reasonable enough. It is also difficult to manually control the entire dance animation generation process. In addition, the model cannot achieve 100% fit with the original music and dance animation, so the output animation is also lossy.

[0057] The implementation scheme of the embodiment of the present application will be exemplarily introduced below with reference to the accompanying drawings.

[0058] like Figure 1 , which is a flowchart of an animation generation method provided in an embodiment of the present application. In this embodiment, the animation generation method can be executed and implemented by a computer device. It should be understood that the order of some of the steps included in the animation generation method provided in this embodiment can be interchanged according to actual needs during actual implementation, or some of the steps can be omitted or deleted, and this embodiment does not specifically limit this.

[0059] Combine the following Figure 1 Each step of the animation generation method of this embodiment is described in detail by way of example. Figure 1 As shown, the method may include the relevant contents described in the following steps S100 to S400.

[0060] Step S100, dividing the target music into at least two music segments.

[0061] In this embodiment, the target music can be segmented into music clips according to the beat or measure of the target music, and the segmentation position can be between two nearest beat points. The target music can be a complete song or a processed song clip. Preferably, in this embodiment, the target music has a fixed BPM (beats per minute), so that each segmented music clip has the same duration. As an example, for example Figure 4 As shown, the target music may be segmented in beats or measures. For example, each measure may include four beats. This embodiment does not limit the specific segmentation method.

[0062] Step S200, input each of the music clips into the pre-trained music recognition model in turn to perform music feature recognition, and obtain music feature information of each of the music clips. In this embodiment, the music feature information includes, but is not limited to, style information and rhythm information of the music clip.

[0063] In a possible implementation of this embodiment, Figure 2 As shown, the music recognition model can be implemented in advance through machine learning through the following steps S210-S230.

[0064] Step S210: Establish a sample music library, wherein the sample music library may include a large number of sample music of different styles, for example, sample music of different styles such as rock, pop, classical, etc.

[0065] Step S220, dividing each sample music in the sample music library into a plurality of sample music segments, and marking corresponding music feature information for each sample music segment to obtain a music feature label for each sample music segment.

[0066] For example, in this embodiment, each piece of sample music can be divided into sections of sample music fragments according to beats or measures, and the position of the division can be in the middle of two nearest beat points. Among them, the music feature information can be marked for each sample music fragment by machine marking or manual marking, and the music feature information can include, but is not limited to, the style information and rhythm information of the corresponding sample music fragment, wherein the style information can be described by natural words such as rock, pop, classical, etc., and the rhythm information can be represented by a feature vector (such as a one-hot 8-dimensional vector). Preferably, in this embodiment, each piece of sample music has a fixed bpm (beats per minute), so that the duration of each sample music fragment divided is the same. At the same time, the style information of the sample music fragments corresponding to the same sample music should be the same, but the rhythm information can be different.

[0067] Step S230, using the sample music library to train a pre-selected deep learning model to obtain a music recognition model for identifying music feature information of music.

[0068] For example, in a possible implementation, each of the sample music clips can be input into the deep learning model in turn, and the deep learning model can identify the music feature information of the sample music clip; then, the loss function value of the deep learning model is calculated based on the identified music feature information of the sample music clip and the music feature label of the sample music clip; finally, the model parameters of the deep learning model are iteratively optimized according to the loss function value until the model training termination condition is met, and the trained deep learning model is used as the music recognition model. Among them, the model training termination condition can be that the loss function value is less than a set threshold, or that the number of iterative training reaches a set number, which is not limited in this embodiment.

[0069] In addition, in this embodiment, inputting each of the sample music clips into the deep learning model may refer to inputting the audio features (such as MFCC features) obtained after processing each sample music clip by a predetermined audio processing method into the deep learning model. The deep learning model used in the training process can adopt the current mainstream music model, which is not limited here. The music recognition model obtained after the training is completed can accurately recognize the music feature information of an externally input piece of music (divided by beat or measure), for example, it can accurately perform style recognition and output rhythm vectors.

[0070] Step S300: searching for animation clips matching the music clips from a pre-established animation library according to the music feature information of the music clips.

[0071] For example, suppose that a music clip M is input into the music recognition model, and the model outputs the style information and rhythm information corresponding to the music clip M. Since each animation clip also has style information and rhythm information, and the style information has a higher level of semantics than the rhythm information, several animation clips with style information consistent with the music clip M can be first found as candidates, such as rock music is usually found in rock-style animation clips, and then more matching animation clips are further screened out according to the rhythm information of the music clip M. Among them, the L2 distance can be used to measure the feature distance between the feature vector corresponding to the rhythm information of the music clip M and the feature vector corresponding to the animation clip to screen the target animation clip that best matches the music clip M. For example, multiple animation clips with an L2 distance less than a certain threshold can be used as candidates, and then one of the target animation clips can be selected from the candidates, or the animation clip with the smallest L2 distance can be directly used as the target animation clip.

[0072] In a possible implementation of this embodiment, the animation library can be established at the same time as the sample music library is established. Figure 4 As shown, the animation library can be established through the following steps S310-S340, which is exemplarily described below.

[0073] Step S310, obtaining animations that match each sample music in the sample music library.

[0074] The animation may be a complete dance action scene, which may be obtained by using a conventional capture device or an optical capture device to capture the animation of a professional dancer wearing a recording device and dancing to corresponding sample music. Then, the captured entire dance action scene may be processed and saved.

[0075] Step S320, dividing each of the animations to obtain a plurality of animation segments, wherein each animation segment has the same duration as a sample music segment of the sample music corresponding to the animation.

[0076] Step S330, feature-labeling each of the animation clips to obtain the animation feature information corresponding to each animation clip, wherein the animation feature information includes the style information and rhythm information of the animation clip. The style information of the animation clip is the same as the style information and rhythm information of the corresponding sample music clip. At the same time, the style information of each animation clip obtained by segmenting the animation corresponding to the same sample music is consistent, but the rhythm information may be different. The segmented animation clip is the smallest unit for generating the animation, so that each music clip in the sample music library can correspond to an accurately matched animation clip. The length of the animation clip and the sample music clip can be less than 1 second (divided by beat) or 1 to 3 seconds (divided by measure).

[0077] Step S340, saving each animation clip marked with the animation feature information into a set database to obtain the animation library.

[0078] In this way, the samples contained in the sample music library finally established are different sample music clips and corresponding annotations (music feature tags), and the animation library is each animation clip and corresponding annotation (animation feature information). Among them, the music library is used to train the deep learning model to obtain the music recognition model, and the animation library is used to retrieve the recognition results (music feature information) of the music clips based on the music recognition model to obtain matching animation clips and then generate animations corresponding to the target music.

[0079] In a possible application scenario of the above step 300, the animation corresponding to the target music can be directly generated according to the animation clips matching each of the music clips. In this way, in an application scenario similar to online live broadcast, the anchor can obtain the animation corresponding to the played music while playing the music, realize the dance accompaniment to the music, and improve the live interactive effect with the user.

[0080] In another possible application scenario of the above step 300, the matching animation clip can be played while the target music is being played. For example, the target music can be the music currently being played. Before the music is played, the music clip to be played can be pre-animated and the animation clip matching the music clip can be found and played synchronously with the music clip to be played.

[0081] In the application scenario where music and corresponding animations are played synchronously, if the duration of the music clip and its matching animation clip are inconsistent, the problem of animation jump may occur. In order to avoid this problem, in the embodiment of the present application, when the music feature information of each music clip is obtained and the matching animation clip is searched in the animation library, the animation clip needs to be scaled accordingly. Specifically, in an optional implementation, the embodiment of the present application implements the search and matching process of the animation clips of each music clip and the scaling process of the animation clips through the following method, which can be included in step S300.

[0082] First, for any target music clip, multiple animation clips matching the music clip are searched from the animation library according to the style information of the target music clip;

[0083] Then, searching at least one candidate animation segment matching the target music segment from the multiple animation segments according to the rhythm information of the target music segment; generally speaking, multiple animation segments matching or similar to the target music segment can be searched according to the rhythm information as candidate animation segments;

[0084] Finally, the target animation segment that matches the target music segment is determined based on the motion information of the at least one candidate animation segment, wherein the motion information includes, but is not limited to, position information, speed information, rotation speed, and angular velocity information, etc., used to characterize the target object (such as the host, virtual character, etc.) in the animation segment. For example, as an example, the motion information of the last preset number of frames (such as the last 3 frames) of the animation segment that matches the previous music segment of the target music segment can be first obtained as an index, and then, based on the index, one of the multiple animations whose motion information of the previous preset number of frames (such as the previous 3 frames) is closest to the index is searched from the at least one candidate animation segment as the target animation segment that matches the target music segment, thereby increasing the diversity of animation generation.

[0085] For example, as an example, assuming that the target music segment is a music segment M, its motion information I(C -3～-1 ) as the index, the set of animation clips whose style information, rhythm information and music clip M match is There are n animations in total, and the first 3 frames of motion information of the i-th animation With I(C -3～-1 ) is the best match, then the final target animation segment is animation segment D i The matching degree between the animation clips can be measured by L2 loss, and the target animation clip can be determined according to the matching degree.

[0086] Furthermore, in order to avoid the problem of animation jumps and ensure that each music clip and its matching animation clip are consistent when finally played, in this embodiment, the target animation clip can be scaled using the B-spline interpolation method so that the target animation clip is consistent in duration with the target music clip and then played synchronously with the target music clip.

[0087] like Figure 5 As shown, the duration of the music clip M is t, and the corresponding animation clip D is matched. i The duration is t d , and with D i Corresponding sample music clip X i The duration is also t d , because the bpm of different music may be different, that is, the duration between the beats of different music is different. In this way, when the target music is random and not included in the sample music library, it may cause the music segment M and the animation segment D to be different. i The duration of the animation clip D is inconsistent. i The animation segment D is scaled to match the duration of the music segment M. Based on this, in this implementation, the B-spline interpolation method can be used to interpolate the animation segment D. i After scaling, it is played synchronously with the music clip M to be played to ensure the continuity of the animation played. In addition, this embodiment can ensure the first-order smoothness of the interpolation point, and the action at the beat point before and after scaling and at the beginning and end of the clip will not change, thus ensuring that the beat animation in the scaled action clip corresponds to the beat point of the music clip M to be played. The duration of the animation clip after interpolation D i ′ is the same length as the music clip M. When the music clip M is finished playing, the animation clip is also finished playing.

[0088] Step S400: generating an animation corresponding to the target music according to the animation clips matched by each of the music clips.

[0089] In this embodiment, after determining the animation clips that match each music clip, each animation clip can be spliced ​​or synthesized into a complete animation. In this embodiment, the generated animation can be used as the dance animation of the target music to realize the function of automatically matching the music, which can be used in the fields of dance animation production and network live broadcast. Taking network live broadcast as an example, while the anchor plays a piece of music, a dance animation corresponding to the music can be automatically generated and displayed in the live broadcast screen to enhance the interaction between the audience and the anchor. For example, at the beginning of the anchor providing live broadcast service, a music library and a high-quality dance action library paired with the music library can be prepared in advance. The recording of high-quality dance movements can be collected by a conventional capture device or a light capture device, and a professional dancer (or the anchor himself) wears a recording device to dance with the music, and the entire dance movement is processed and saved in the dance action library. In the process of making the music library, the music used can include music of various styles, such as rock, pop, classical, etc., which are stored in the music library after processing. Each piece of music is divided into segments of music fragments according to the beat or measure, and the position of the segmentation can be in the middle of the two nearest beat points. In a specific implementation, the music library can be used to train the deep learning model to obtain the music recognition model for identifying music feature information. The dance action library produced can be used as the animation library, and animation clips matching each music clip are searched according to the recognition results of the music feature information, and finally, dance animations matching the music played during the live broadcast are generated based on the searched animation clips, thereby realizing the interaction of the live broadcast.

[0090] like Figure 6 FIG. 1 is a schematic diagram of an animation generation system for implementing the above-mentioned animation generation method provided in an embodiment of the present application. In this embodiment, the animation generation system can be applied to Figure 7The computer device 100 shown. In detail, the computer device 100 may include one or more processors 110, a machine-readable storage medium 120, and an animation generation system 130. The processor 110 and the machine-readable storage medium 120 may be connected to each other via a system bus. The machine-readable storage medium 120 stores machine-executable instructions, and the processor 110 implements the animation generation method described above by reading and executing the machine-executable instructions in the machine-readable storage medium 120. In this embodiment, the computer device 100 may be a cloud server for executing the various functional modules included in the front end of the animation generation system, or a user client for executing the various functional modules included in the back end of the animation generation system, or a cloud server for executing the various functional modules included in the front end and back end of the animation generation system at the same time, or a combination of a cloud server and a user client for respectively executing the various functional modules included in the front end and back end of the animation generation system, and this embodiment does not limit this.

[0091] The machine-readable storage medium 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store a program, and the processor 110 executes the program after receiving an execution instruction.

[0092] The processor 110 may be an integrated circuit chip with signal processing capability. The processor may be, but is not limited to, a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0093] In this embodiment, the animation generation system 130 may include a segmentation module 131 , a recognition module 132 , a search module 133 , and a generation module 134 .

[0094] The segmentation module 131 is used to segment the target music into at least two music segments.

[0095] In this embodiment, the segmentation module 131 may be used to execute the above step S100. For more detailed contents of the segmentation module 131, reference may be made to the relevant contents of the above step S100, which will not be described again here.

[0096] The recognition module 132 is used to input each of the music clips into a pre-trained music recognition model in turn to perform music feature recognition to obtain music feature information of each of the music clips, wherein the music feature information includes style information and rhythm information of the music clip.

[0097] In this embodiment, the identification module 132 can be used to execute the above step S200. For more detailed content of the identification module 132, please refer to the relevant content of the above step S200, which will not be repeated here.

[0098] The search module 133 is used to search for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips.

[0099] In this embodiment, the search module 133 may be used to execute the above step S300. For more detailed contents of the search module 133, reference may be made to the relevant contents of the above step S300, which will not be described again here.

[0100] The generating module 134 is used to generate the animation corresponding to the target music according to the animation segments matching the music segments.

[0101] In this embodiment, the generating module 134 may be used to execute the above step S400. For more detailed contents of the generating module 134, reference may be made to the relevant contents of the above step S400, which will not be described again here.

[0102] In summary, the animation generation method, system and computer equipment provided by the embodiment of the present application, by using a large amount of music-animation data sets, do not adopt an end-to-end music-animation generation model, but only adopt a deep learning model (music recognition model) for the extraction of music features, and adopt a feature matching search algorithm for the processing of animation. During the search process, it can be modified according to the user's ideas to achieve the controllability of the entire animation matching and generation process, and the animations are all high-quality animations that are found and corrected from the action library, so the output animation is lossless. Therefore, the animation generation method in the present embodiment has good scalability, and the generated animation is also more natural and reasonable, which can enhance the user's experience. In addition, as the scale of the action library is gradually improved and becomes larger, the animation generation effect is correspondingly better.

[0103] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0105] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0106] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0107] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An animation generation method, characterized in that: The method comprises: Divide the target music into at least two music segments; Inputting each of the music clips into a pre-trained music recognition model in turn to perform music feature recognition, thereby obtaining music feature information of each of the music clips, wherein the music feature information includes style information and rhythm information of the music clip; Searching for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips; Generating an animation corresponding to the target music according to the animation segments matching the music segments; Searching for animation clips matching the music clips from a pre-established animation library according to the music feature information of the music clips, including: For any target music clip, searching the animation library for a plurality of animation clips matching the music clip according to the style information of the target music clip; the style information of the animation clips is consistent with the style information of the target music clip; searching at least one candidate animation segment matching the target music segment from the plurality of animation segments according to the rhythm information of the target music segment; Obtaining the motion information of the last preset number of frames of the animation clip matched by the previous music clip of the target music clip as an index; wherein the motion information includes position information, speed information, rotation speed and angular speed information for characterizing the target object in the candidate animation clip; According to the index, one of the candidate animation clips whose motion information of the previous preset number of frames is closest to the index is searched as the target animation clip matching the target music clip.

2. The animation generation method according to claim 1, characterized in that: The method further comprises the step of training a pre-selected deep learning model to obtain the music recognition model, the step comprising: Establishing a sample music library, wherein the sample music library may include sample music of different styles; Each piece of sample music in the sample music library is divided into a plurality of sample music segments, and corresponding music feature information is annotated for each sample music segment to obtain a music feature label for each sample music segment, wherein the music feature information corresponding to the sample music segment includes style information and rhythm information; The sample music library is used to train a pre-selected deep learning model to obtain a music recognition model for recognizing music feature information of music.

3. The animation generation method according to claim 2, characterized in that: The pre-selected deep learning model is trained using the sample music library to obtain a music recognition model for recognizing music feature information of music, including: Inputting each of the sample music clips into the deep learning model in turn, and using the deep learning model to identify and obtain music feature information of the sample music clips; Calculating the loss function value of the deep learning model based on the identified music feature information of the sample music clip and the music feature label of the sample music clip; The model parameters of the deep learning model are iteratively optimized according to the loss function value until the model training termination condition is met, and the trained deep learning model is used as the music recognition model.

4. The animation generation method according to claim 2 or 3, characterized in that: The method further comprises the step of pre-establishing the animation library, the step comprising: Obtaining animations that match each piece of sample music in the sample music library; Segmenting each of the animations to obtain a plurality of animation segments, wherein each animation segment has the same duration as a sample music segment of the sample music corresponding to the animation; Performing feature annotation on each of the animation clips to obtain animation feature information corresponding to each of the animation clips, wherein the animation feature information includes style information and rhythm information of the animation clip; Each animation clip marked with the animation feature information is saved in a set database to obtain the animation library.

5. The animation generation method according to claim 1, characterized in that: The method further comprises: The target animation clip is scaled by using a B-spline interpolation method so that the duration of the target animation clip is consistent with that of the target music clip.

6. An animation generation system, applied to a computer device, characterized in that: The animation generation system comprises: A segmentation module, used for segmenting the target music into at least two music segments; A recognition module, used to sequentially input each of the music clips into a pre-trained music recognition model to perform music feature recognition, and obtain music feature information of each of the music clips, wherein the music feature information includes style information and rhythm information of the music clip; A search module, used for searching for animation clips matching each of the music clips from a pre-established animation library according to the music feature information of each of the music clips; A generation module, used for generating animations corresponding to the target music according to the animation clips matching the music clips; The search module is also used to search for a plurality of animation clips matching any target music clip from the animation library according to the style information of the target music clip; the style information of the animation clip is consistent with the style information of the target music clip; Searching at least one candidate animation segment matching the target music segment from the plurality of animation segments according to the rhythm information of the target music segment; The motion information of the last preset number of frames of the animation clip that matches the previous music clip of the target music clip is obtained as an index; according to the index, one of the motion information of the previous preset number of frames is searched from the at least one candidate animation clip, which is closest to the index as the target animation clip that matches the target music clip; wherein the motion information includes position information, speed information, rotation speed and angular speed information for characterizing the target object in the candidate animation clip.

7. A computer device, characterized in that: The invention comprises a machine-readable storage medium and one or more processors, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the one or more processors, the animation generation method according to any one of claims 1 to 5 is implemented.

Images