Dance video generation method and device, equipment, storage medium and program product

By obtaining song and user feature information, and dynamically generating dance videos, the problem of high video similarity in the prior art is solved, and the effect of personalization and interest improvement is achieved.

CN118984403APending Publication Date: 2024-11-19MIGU CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410989985.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the prior art, when generating dance videos, the videos generated when the same song is repeatedly played have a high similarity, resulting in users losing their viewing interest and being unable to generate personalized dance videos based on user preferences or location information.

Method used

By obtaining song feature information and user behavior feature information, combining preset background image library and virtual character model, dance videos are dynamically generated, and different backgrounds and dance movements are generated according to the song's emotions and user preference changes.

Benefits of technology

It improves users' interest in watching dance videos, enhances the personalization and dynamicity of the videos, improves the generation efficiency, and meets users' needs for personalized videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984403A_ABST
    Figure CN118984403A_ABST
Patent Text Reader

Abstract

The invention relates to a dance video generation method and device, equipment, a storage medium and a program product. Comprising the steps of obtaining song feature information of a target song in response to a selection operation of a user for the target song, and determining emotion information of the target song according to the song feature information; acquiring behavior feature information of the user, and determining a target background image from a preset background image library according to the behavior feature information and the emotion information; and driving a preset virtual character to execute a corresponding dance action through the song feature information, and fusing the target background image and the dance action to generate a dance video. Visibly, according to the embodiment of the invention, the target background image can change along with the behavior characteristic information of the user, and different dance videos can be obtained through different target background images, so that different dance videos can be displayed for the user when the user plays the same song, and the watching interest of the user on the dance videos is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a method, device, equipment, storage medium and program product for generating a dance video. Background Art

[0002] With the development of Internet technology, dance videos containing virtual characters can be automatically generated when users play songs, thereby increasing users' viewing interest.

[0003] In the related art, a dance video corresponding to a song can be generated according to the rhythm of the song through a trained machine model, and the video background corresponding to the dance video is generally set in advance by professionals. Therefore, when a user repeatedly plays the same song, the generated dance videos have a high degree of similarity, and repeatedly watching dance videos with high similarity will cause the user to lose interest in watching. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a method, device, equipment, storage medium and program product for generating a dance video to solve the problems existing in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a dance video is provided, the method comprising: in response to a user's selection operation for a target song, obtaining song feature information of the target song, and determining emotional information of the target song based on the song feature information; obtaining the user's behavior feature information, and determining a target background image from a preset background image library based on the behavior feature information and the emotional information; driving a preset virtual character to perform corresponding dance movements through the song feature information, and fusing the target background image and the dance movements to generate a dance video.

[0006] According to a second aspect of an embodiment of the present disclosure, there is provided a device for generating a dance video, which is applied to the method for selecting a dance video as in the first aspect. The device comprises: an acquisition module, for acquiring song feature information of a target song in response to a user's playback operation on a target song, and determining emotional information of the target song based on the song feature information; a determination module, for acquiring user's behavior feature information, and determining a target background image from a preset background image library based on the behavior feature information and the emotional information; and a driving module, for driving a preset virtual character to perform corresponding dance movements through the song feature information, and fusing the target background image with the dance movements to generate a dance video.

[0007] According to a third aspect of the embodiments of the present disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned method for generating a dance video.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method for generating a dance video are implemented.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, and when the computer program / instructions are executed by a processor, the steps of the above-mentioned dance video generation method are implemented.

[0010] At least one of the above-mentioned technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects: in response to the user's selection operation on the target song, the song feature information of the target song is obtained, and the emotional information of the target song is determined based on the song feature information; the user's behavior feature information is obtained, and the target background image is determined from the preset background image library based on the behavior feature information and the emotional information; the preset virtual character is driven to perform corresponding dance movements through the song feature information, and the target background image and the dance movements are merged to generate a dance video.

[0011] Based on this, the disclosed embodiment can determine the emotional information of the target song according to the song characteristic information corresponding to the target song, and then determine the target background image from the preset background image library according to the emotional information and behavioral characteristic information. At the same time, the preset virtual character is driven by the song characteristic information to perform the corresponding dance movements, and the dance video corresponding to the target song is generated with the target background image as the background. It can be seen that the target background image of the disclosed embodiment will change with the user's behavioral characteristic information, and different dance videos can be obtained through different target background images, and then different dance videos can be displayed to the user when the user plays the same song, thereby enhancing the user's interest in watching the dance video. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A schematic diagram of a flow chart of a method for generating a dance video provided by an exemplary embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a flow chart of a song emotion determination model provided by an exemplary embodiment of the present disclosure;

[0015] Figure 3A schematic diagram of the structure of a skeleton node model provided by an exemplary embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of the structure of a dance movement generation model provided by an exemplary embodiment of the present disclosure;

[0017] Figure 5 A structural schematic diagram of a dance action corresponding to a skeleton node provided by an exemplary embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of a virtual character performing dance movements provided by an exemplary embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of a dance video containing a virtual character provided by an exemplary embodiment of the present disclosure;

[0020] Figure 8 A schematic flow chart of another method for generating a dance video provided by an exemplary embodiment of the present disclosure;

[0021] Fig. 9 A schematic structural diagram of a dance video generation device provided by an exemplary embodiment of the present disclosure;

[0022] Fig.10 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present disclosure;

[0023] Fig.11 A schematic diagram of the structure of a computer system provided for an exemplary embodiment of the present disclosure;

[0024] Fig.12 A schematic diagram of a computer program product provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] In the prior art, there are two solutions for determining the dance movements of virtual characters based on songs. The first solution is that professional choreographers design the dance movements of virtual characters based on the content of songs; the second solution is that dance movements are intelligently generated based on songs by means of machine learning. Specifically, the relationship between songs and dance movements in a large number of dance videos can be used to train a dance movement generation model, and then dance movements matching the song can be generated based on the corresponding song rhythm and style. Among them, neural networks are a branch of machine learning, which simulates the structure and function of biological brains and is composed of interconnected artificial neurons.

[0031] Although the dance moves generated by the first solution mentioned above have a high degree of match with the song, the first solution mainly relies on the choreographer's arrangement, which requires high labor costs and low efficiency in generating dance moves. Therefore, it is not suitable for C-end song software platforms and is not suitable for large-scale user groups. The second solution can meet the needs of intelligently generating dance scenes, but the dance moves generated for the same song are similar. It can be seen that neither of the above two solutions can generate user-exclusive personalized dance videos based on the user's preferences or the user's current location information. It should be understood that the C-end refers to a client for consumers and individual end users.

[0032] In view of the above problems, the embodiments of the present disclosure provide a method, device, equipment, storage medium and program product for generating a dance video, which can use machine learning related algorithms to train a dance action generation model through professional dance videos to obtain a dance action generation model. Based on this, in response to the user's selection of a song, the rhythm and melody of the song can be feature extracted to determine the audio information corresponding to the song; and the lyrics in the song can be segmented to obtain the lyrics information of the song, and then the emotional information of the song can be determined according to the lyrics information and the audio information, and finally the audio information is input into the pre-trained dance action generation model to generate the dance action corresponding to the song. At the same time, the background image can be determined from the preset image library according to the emotional information corresponding to the song, the user's current geographical location and the song style, and the appearance of the virtual character can be determined, and finally the virtual character, background image, dance action and song are synthesized to generate the final dance video, and then when the user listens to the song, a user-specific dance video can be generated, and the dance video can be displayed in the front-end interface of the terminal currently used by the user, thereby improving the user's listening pleasure and improving the generation efficiency of the dance video.

[0033] Figure 1 FIG. 1 is a flow chart of a method for generating a dance video provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, specifically including:

[0034] S101, input the target song into a pre-trained dance movement generation model to obtain dance movements corresponding to the target song.

[0035] In some embodiments, when a user opens a song playing software to listen to a song, in response to the user's operation of playing the song, a target song is determined, and then the audio signal of the target song can be analyzed through a convolutional neural network to obtain rhythm features and melody features, and the audio signal can be determined to correspond to the audio emotional information according to the rhythm features and melody features; at the same time, the Jieba word segmentation algorithm can be used to extract keywords from the lyrics corresponding to the target song, and the lyrics emotional information to be expressed by the lyrics can be determined according to the keywords. Among them, the Jieba word segmentation algorithm is a word segmentation algorithm based on probability statistics, which uses the idea of ​​dynamic programming for word segmentation. The Jieba word segmentation algorithm implements three word segmentation strategies: prefix-based word segmentation, maximum match word segmentation based on word frequency, and maximum probability word segmentation based on word meaning. Through the fusion of multiple word segmentation strategies, the Jieba word segmentation algorithm can segment Chinese text more accurately.

[0036] Specifically, firstly, the meaningless words in the target song lyrics need to be removed, where the meaningless words can be conjunctions and prepositions, etc. Then, the lyrics after removing the meaningless words can be subjected to keyword extraction operation through the Jieba word segmentation algorithm to obtain multiple keywords; finally, the multiple keywords can be matched with the emotion words in the preset emotion corpus respectively, and the emotion information of the lyrics corresponding to the multiple keywords can be determined according to the matching results. Among them, the preset emotion prediction library includes multiple preset emotion categories, and each preset emotion category includes multiple emotion words. Based on this, the calculation formula of the lyrics emotion information corresponding to the keyword t can be:

[0037]

[0038] Among them, A represents the number of keywords that belong to the target preset emotion category among multiple keywords, and the number of keywords that successfully match the target preset emotion category including multiple emotion words; B represents the number of keywords that do not belong to the target preset emotion category among multiple keywords, and do not successfully match the target preset emotion category, but successfully match other preset emotion categories including multiple emotion words; C represents the number of keywords that belong to the target preset emotion category among multiple keywords, but do not successfully match the target preset emotion category; D represents the number of keywords that do not belong to the target preset emotion category among multiple keywords, and do not successfully match other preset emotion categories including multiple emotion words; n represents the number of all emotion words in the preset emotion prediction library; FI represents the number of times keyword t appears in the current preset emotion category. Among them, the stronger the correlation between the keyword and the preset emotion category, the higher the chi-square statistic value x 2 The larger the x 2 When it is 0, it means that the keyword has nothing to do with the current preset emotion category.

[0039] In some embodiments, the emotional information of the lyrics corresponding to the target song can be determined based on the emotional information of the lyrics corresponding to multiple keywords. Specifically, the emotion corresponding to the emotional category containing the largest number of keywords can be used as the emotional information of the lyrics corresponding to the target song. For example, the preset emotional categories include sad category, happy category and depressed category. Based on this, when the number of keywords of a song is 10, among which 5 keywords belong to the sad category, 3 keywords belong to the happy category, and 2 keywords belong to the depressed category, since the number of keywords corresponding to the sad category is the largest, it can be determined that the emotional information of the lyrics corresponding to the target song is sad.

[0040] At the same time, the embodiment of the present disclosure can also convert keywords into word vectors through natural language processing technology (word2vec) to facilitate subsequent processing. Among them, word2vec is a word embedding method, which functions to convert words in natural language into dense vectors that can be understood by computers.

[0041] Specifically, when the audio signal of the target song is analyzed by a convolutional neural network to obtain the rhythm characteristics and melody characteristics, the audio signal can be pre-emphasized first. The pre-emphasis process is to make the frequency spectrum trend of the audio signal flat so as to perform spectrum analysis; then the pre-emphasized audio signal is framed according to the time sequence corresponding to the lyrics of the target song when it is played; then the audio signal after frame processing is windowed, and the purpose of windowing is to solve the Gibbs effect caused by the discontinuity of the start and end after framing; then each frame of the audio signal after the windowing process is fast Fourier transformed to obtain the spectrum of each frame signal, and then the spectrum of each frame signal is converted into Mel frequency scale through a triangular filter, and finally a discrete cosine transform is performed to obtain the Mel frequency cepstral coefficient (MFCC) feature vector of each frame of audio, and some other audio information containing emotional information is extracted, such as: zero crossing rate, spectrum centroid, spectrum bandwidth, spectrum attenuation, spectrum flux, chromaticity vector and chromaticity spectrum. Among them, the Mel frequency scale is a perceptual scale in which listeners judge that pitches of different frequencies are equal to each other, which represents the human ear's perception of equidistant pitch changes; Mel-Frequency Cepstral Coefficients (MFCC) is a feature extraction method commonly used in speech signal processing and audio analysis. It can convert audio signals into a set of feature vectors for subsequent classification, recognition and other tasks; and audio information includes low-level audio information (low level descriptors, LLDs). LLDs refer to some artificially designed low-level features, and LLDs can be used as frame features.

[0042] Figure 2 The following is a flow chart of a song emotion determination model provided by an exemplary embodiment of the present disclosure. Figure 2As shown in the figure, the song emotion determination model is constructed based on the fusion classification model of convolutional neural network (CNN) and long short-term memory neural network (LSTM). After the user determines the target song, the word vector and audio information corresponding to the target song are obtained according to the above method, and the word vector and audio information are used as the input of the song emotion determination model. Among them, the word vector and spectrogram features can play the role of feature extraction and feature selection through the convolution layer and pooling layer in the CNN network, and output a set of serialized feature vectors, which are input into the LSTM network as new features and output with the attention mechanism. Here, the spectrogram is the speech spectrum graph, which is generally obtained by processing the received time domain signal. Therefore, as long as there is a time domain signal of sufficient time length, it can be used; and the audio information LLDs can be combined into HSFs through statistical methods, and then combined with the word frequency weight vector after the weight is extracted by the chi-square test, and then the feature dimension reduction is performed by DNN. Finally, the feature vectors output by the two network structures are spliced ​​to obtain the fusion feature, and the fusion feature is input into the Softmax layer for classification processing to obtain the lyrics emotion and audio emotion, and then the two results are dimensional decision fused to obtain the emotional information of the target song. Among them, high level statistics functions (HSFs) refer to the features obtained by doing some statistics on the basis of LLDs, such as mean, maximum value, etc. HSFs can be used as discourse features.

[0043] Figure 3 FIG. 1 is a schematic diagram of a skeleton node model provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the disclosed embodiment can construct a bone node model corresponding to the virtual character by analyzing the bone structure of the virtual character. The bone node model includes the eyes, mouth, trunk and limbs of the virtual character. Since the process of the virtual character completing the dance action is the process of the bone node change, the bone node position and bone angle corresponding to different dance actions can be determined through the bone node model, thereby controlling the changes of each bone node in the bone node model to complete the dance action.

[0044] Specifically, the direction from one shoulder to the other shoulder of the virtual character in the initial posture can be the x direction, the y direction is the upward body vector, and the z direction is perpendicular to the plane formed by the x and y axes. Then, based on the Euler angle principle, the rotation angle in the motion capture data is converted into the bone node position data, where the rotation angle can be the roll angle, pitch angle and yaw angle. When converting the rotation angle into the bone node position data, the Euler angle is firstly matrix-converted. The specific formula is:

[0045]

[0046]

[0047] Among them, R represents the rotation matrix corresponding to the roll angle; P represents the rotation matrix corresponding to the pitch angle; Y represents the rotation matrix corresponding to the yaw angle; the rotation angle around the z-axis is r; the rotation angle around the x-axis is p; the rotation angle around the y-axis is q; according to the properties of the orthogonal matrix, the rotation matrix N(RPY) can be obtained based on the following formula: -1 :

[0048]

[0049] In some embodiments, the rotation matrix N(RPY) -1 The bone node position data corresponding to the desired bone node can be calculated by the relative orientation corresponding to the rotation matrix of the intermediate bone node between the desired bone node and the root bone node, and the bone node position data of the desired bone node in the bone node model can be obtained through the derotation and detranslation process, wherein the specific calculation formula of the derotation process is as follows:

[0050]

[0051] Where J0 represents the rotation matrix after derotation; (x0, y0, z0) represents the initial offset of the desired bone node N1, N i Represents the rotation matrix corresponding to all bone nodes between bone node N1 and root bone node Nr, i represents the bone node number, n represents the number corresponding to the last bone node, i = 1, 2, ..., n.

[0052] For the translation process, the position offsets of all intermediate bone nodes between the bone node and the root bone node can be calculated. The position offset of the intermediate bone node relative to its parent bone node is expressed as i Therefore, the position offset P of the desired bone node can be expressed as P = O1 + ... + O i +…+O r Among them, the parent bone node and the child bone node are concepts in the tree data structure. The parent bone node refers to the upper-level bone node of a bone node, and the child bone node refers to the lower-level bone node of a bone node.

[0053] In some embodiments, since the movement of the virtual character in space will cause the position of the bone node to change, that is, the change of the bone node position data, it is necessary to use the root bone node position data P N (x r ,y r ,z r ) calculates the position data of each bone node. Specifically, the change P0 of the bone node position data can be expressed as P0=P i(x,y,z)-P N (x r ,y r ,z r ), where P i (x, y, z) represents the bone node position data corresponding to the i-th bone node. Based on this, the coordinate changes of each bone node can be obtained. However, after obtaining the position changes of each bone node, it is also necessary to calculate the distance between each bone node, the bone angle data, the virtual character's orientation, spatial orientation, and the weight and smoothness of the force effect from the motion capture data. Characteristic parameters describing the biomechanics of the virtual character's movement, such as the distance between the bone nodes and the bone angle data, need to be considered from the perspective of limb structure. The distance between the bone nodes reflects the speed of the movement. The Euclidean distance can be used to calculate the distance h between the bone nodes. The calculation formula is:

[0054]

[0055] The bone angle data w represents the bending state between adjacent bones, and its calculation formula is:

[0056] w = arccos(h a ·h b / (||h a ||×||h b ||))

[0057] Among them, h a Indicates the distance between bone node a and bone node b; h b Indicates the distance of bone node b relative to bone node a, such as Figure 3 As shown, the bone pair is two connected bones, for example, the right hip and the left hip, and the bone pair angle data is the angle between the line between the right hip and the hip and the line between the left hip and the hip.

[0058] In some embodiments, a skeleton node model can be used to process dance movements in dance videos, that is, to determine the skeleton node position data and skeleton angle data corresponding to each dance movement, and then obtain the skeleton node data corresponding to each dance movement in the dance video, and construct training samples based on the skeleton node data, and then a multi-branch graph convolutional network can be used to construct a dance movement generation model, and the dance movement generation model can be trained based on the dance movement data.

[0059] Specifically, the bone node position data (s1, s2, ..., s n ), where s1 represents the position data of the bone node numbered 1 in the bone node model, s2 represents the position data of the bone node numbered 2 in the bone node model, and s nRepresents the position data of the bone node numbered n in the bone node model; the angle data of each bone pair w1, w2, ..., w n and the emotional information of the song e1,e2,…,e n As training parameters, w1 represents the angle data of the bone numbered 1 in the bone node model, w2 represents the angle data of the bone numbered 2 in the bone node model, and w n represents the angle data of the bone pair numbered n in the bone node model; e1 represents the emotional information of the first keyword of the target song, e2 represents the emotional information of the second keyword of the target song, and e n Represents the emotional information of the nth keyword of the target song.

[0060] Figure 4 FIG. 1 is a schematic diagram of a dance movement generation model provided by an exemplary embodiment of the present disclosure. Figure 4 As shown in the figure, in the separable graph convolution module (SSGC), the global spatiotemporal graph is introduced to obtain the global spatial relationship of the nodes, so that the dance movement generation model can learn rich dance movements and improve the accuracy of the dance movement generation model. Among them, the data in the processed training samples can be input into the multi-scale mixed time convolution module (MMTC), and the temporal features aggregated by different time skeleton nodes (T-pool) are fused through the dynamic fusion module (CAM) to adaptively obtain the time domain information of different dance movements.

[0061] At the same time, the disclosed embodiment also introduces the angle between the skeleton pairs with the same structure to obtain the spatiotemporal information of the motion trajectory of the skeleton nodes, and then splices them with the skeleton position branch and the emotion branch in the channel dimension to model the motion information of the skeleton nodes, and dynamically adjusts the high-level spatiotemporal information to encode the characteristics of different dance movements through the channel attention module (CAM). Then, the graph gated unit (G-GRU) is used to sequentially generate dance movements in the future. The spatial position of the nodes constrained by the static graph of the skeleton nodes can enhance the propagation state of the information in the multi-layer perceptron module (MLP), and the residual link is used to convert the prediction result y at the previous time point into the prediction result y at the previous time point. t-1 Fusion makes the model generate prediction results of bone node dance movements more stably.

[0062] The disclosed embodiment can also automatically adjust the weights of reconstruction loss and emotional loss according to the degree of completion of model training during the training of the dance movement generation model based on training samples. Among them, Wa represents the weight of the model reconstruction loss, Wb represents the weight of the emotional loss, Wa = data set to be trained / overall training data set, Wb = 1-(data set to be trained / trained data set). In the early stages of model training, Wa is relatively large, which can ensure that the generated dance movements can generate standard dance movements based on the bone node position data and the bone angle data. In the later stages of model training, as Wb increases, the model can be guided to generate more emotionally expressive dance movements, while making the song and the corresponding dance movements more integrated. The specific total loss L can be expressed as follows:

[0063] L=Wa*L_recon+Wb*L_emotion

[0064] Among them, L represents the total loss, L_recon represents the reconstruction loss, and L_emotion represents the emotion loss.

[0065] Figure 5 The following is a schematic diagram of a dance action structure corresponding to a skeleton node provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the target song can be input into the trained dance movement generation model, and the trained dance movement generation model generates the dance movements corresponding to the target song. Here, the dance movements are movements simulated by multiple bone nodes of the virtual character, and the dance movements corresponding to the target song are a series of coherent dance movements, and the multiple dance movements can be the same or different.

[0066] Figure 6 The following is a schematic diagram of a virtual character performing dance movements provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, virtual character images with different makeups can be pre-set, and then after the dance movements are determined, the dance movements and the corresponding virtual character images are input into the virtual human costume generation module to generate the dance image of the virtual character. Therefore, when a series of coherent dance movements and virtual character images are input into the virtual human costume generation module, a series of virtual character dance images can be obtained.

[0067] S102, obtaining user social behavior information, and determining a target background image according to the user social behavior information.

[0068] In some embodiments, the disclosed embodiments can classify the user's real-life preferences, that is, classify the social behavior information that the user usually generates on social networks, that is, classify the text, images, videos and other content sent by the user on social networks, mainly divided into three categories of real-life preferences. Among them, the first category is the preference for scenic spots, the second category is the preference for landmark buildings, and the third category is the preference for Internet celebrity check-in places; the specific classification method includes the following steps:

[0069] Step 1: Build a background image library of real-life images of different cities.

[0070] In some embodiments, the mainstream open source image datasets in the industry do not have the classification of domestic landmark buildings and Internet celebrity check-in places. Therefore, the disclosed embodiment retrieves the image data in the open source city image dataset (Mapillary Vistas), manually annotates the landmark buildings and Internet celebrity check-in places, retrieves the Chinese landscape database dataset, and uses the keywords of portrait, city, mountain, river, lake, waterfall, and building as classification keywords to perform a preliminary search, and then manually marks the retrieved image set to form a background image library of urban scenic spots. As shown in Table 1, taking Chengdu as an example, the real scene of Chengdu city can be divided into three categories of real scene preference materials:

[0071] Table 1. Classification results of real scene preference

[0072]

[0073] Step 2: Construct a training set based on the preset background image library composed of real-life city images, and train the MobileNet image classification model to obtain a trained image classification model. Based on this, the user's sharing behavior in social software or in a three-party social circle can be divided into four behaviors: publishing, collecting, sharing, and browsing. The images shared in the social circle are used as inputs to the image classification model to perform image classification predictions, determine the classification results of the images, and then obtain the user's real-life preferences. Among them, each time a user publishes an image in a social circle, it can be defined as a contribution to the classification preference.

[0074] Step 3: Calculate the user's three categories c respectively u Preference preference, where W p represents the weight corresponding to the publishing behavior, Wc represents the weight corresponding to the collection behavior, Ws represents the weight corresponding to the sharing behavior, and Wv represents the weight corresponding to the browsing behavior. The preference contribution brought by one behavior can be expressed as:

[0075]

[0076] Among them, t represents the monthly difference between the current time and the behavior time, t_0 represents a time decay cycle of 12 months, and the largest value is selected as the user's classification preference, j represents the image sequence number, j = 1, 2..., m, m represents the maximum sequence number; c u represents the preference for different types of real scenes, u=1,2,3, c1 represents the preference for scenic spots, c2 represents the preference for landmark buildings, c3 represents the preference for internet celebrity check-in spots, and W l are the weights corresponding to different behaviors, l = p, c, s, v. Based on this, after determining the preferences corresponding to different real-scene preference categories, the real-scene preference with the maximum preference value is selected as the target real-scene preference.

[0077] In some embodiments, the disclosed embodiments can also determine the target background image of the dance video by the user's geographical location when listening to the song, the emotion corresponding to the target song, and the target real scene preference. Among them, the geographical location can be obtained based on the real-time IP address information of the user currently listening to the song, and it is mainly necessary to obtain the user's current location information, that is, the city where the user is currently located. Among them, the IP address is a unique address of a network device on the Internet. Among them, the IP address is an Internet Protocol address used to identify devices on the network. Based on this, the user's current location information can be determined through the IP address information of the device currently used by the user.

[0078] Specifically, the disclosed embodiment mainly starts from two dimensions and divides the emotions of the song into four emotion categories: anger, happiness, relaxation and sadness. At the same time, the emotions corresponding to the background image are determined according to the three dimensions of color, light and season. Based on this, the emotions corresponding to the background image can be determined according to the emotions of the song.

[0079] For example, color can be a hue. Warm colors, such as red, orange, and yellow, can correspond to happy emotions and can be matched with relaxing and cheerful songs. Cold colors, such as blue, green, and purple, can correspond to sad emotions and can be matched with sad songs. The light dimension can include bright light, soft light, and gloomy light, where bright light can correspond to happy emotions, soft light can correspond to relaxing emotions, and gloomy light can correspond to sad emotions. The season can be spring, summer, autumn, and winter, where spring can correspond to energetic emotions, summer can correspond to happy emotions, spring can correspond to anxious emotions, and winter can correspond to sad emotions.

[0080] In practical applications, taking rock songs as an example, rock songs are mostly full of vitality, relaxing and cheerful songs. Therefore, for rock songs, a cool background can be selected, and the weather corresponding to the background can be spring. At the same time, the background light can be set to be brighter.

[0081] S103, generating a virtual human dance video according to the dance movements and the target background image.

[0082] In some embodiments, a user may select a virtual character from a preset virtual character library and dress up the virtual character according to preset skin props. The skin props may be traditional Chinese style skin props, fashionable skin props, and daily skin props. The skin props may be selected by the user or set by the server according to corresponding rules. When the user dresses up the virtual character, the server may determine the user's preferences based on the songs that the user has listened to in the past, and then recommend skin props.

[0083] Specifically, when the songs that the user has listened to in the past are mostly traditional Chinese style songs, the server can recommend traditional Chinese style skin props to the user; when the songs that the user has listened to in the past are mostly rock songs, the server can recommend fashion skin props to the user.

[0084] In actual applications, after the virtual character is determined, the virtual character, dance movements and target background image can be fused based on the deep learning texture mapping and graphics fusion method to obtain the dance video corresponding to the target song, and then the dance video can be played when the user plays the target song.

[0085] In some embodiments, when the user plays a target song, the user first inputs the target song into a song emotion determination model to determine the emotion of the target song. At the same time, the target song and the corresponding emotion are input into a dance movement generation model to determine the dance movement corresponding to the target song, and then a virtual character is determined. The target background image is determined by the user's geographic location when listening to the song, the emotion corresponding to the target song, and the target real scene preference. Finally, the virtual character, the target background image, and the dance movement are integrated to obtain a dance video corresponding to the target song. The dance video can be played synchronously when the user plays the target song for the user to watch. At the same time, the user can also set the permissions and expiration time of the dance video, and can also share the dance video on social platforms.

[0086] Figure 7 A schematic diagram of a dance video containing a virtual character provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, when the user opens the software to listen to songs during a trip to Chengdu, Sichuan, and the selected song is a traditional Chinese style song, the target song and the corresponding emotional information are input into the dance movement generation model to determine the dance movements corresponding to the target song. At the same time, the user's real-scene preferences are determined based on the user's social behavior information or the behavior in the third-party social circle. At the same time, the song emotions are matched to the real-scene picture of Qingcheng Mountain in the summer morning. Finally, the virtual character, the target background image and the dance movements are integrated to obtain the dance video corresponding to the target song.

[0087] The method for generating a dance video provided by the embodiment of the present disclosure may be executed by a terminal or by a chip applied to the terminal.

[0088] Exemplarily, the above-mentioned terminals may include one or more of mobile phones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technology, and may also include but not be limited to remote control devices, wearable devices, street lamps, home appliances and other smart terminals, which are not specifically limited in the embodiments of the present disclosure.

[0089] Figure 8 FIG. 1 is a flow chart of another method for generating a dance video provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, specifically including:

[0090] S801, in response to a user's selection operation on a target song, obtaining song feature information of the target song, and determining emotional information of the target song according to the song feature information.

[0091] In some embodiments, the song feature information corresponding to the target song may be the audio information and lyrics information of the target song, and the emotional information of the target song may be determined based on the lyrics information and the audio information, making the emotional judgment of the target song more accurate, wherein the audio information is the rhythm information and / or key information of the target song.

[0092] Specifically, assuming that the target song is a love song, the song is mainly played with classical instruments, the song has a slow rhythm, the sound of the keys is low, and one of the lyrics is "I haven't heard from you since we parted." At this time, the emotional information corresponding to the target song can be determined as sad based on the audio information, and the emotion expressed by the lyrics can be determined as sad after separation based on the words "parted" and "no" in the lyrics information. Based on this, the emotional information of the target song can be determined as sad based on the audio information and the lyrics information, that is, the target song is a sad love song.

[0093] S802, obtaining the user's behavior characteristic information, and determining a target background image from a preset background image library according to the behavior characteristic information and emotion information.

[0094] In some embodiments, the behavior feature information may be text, video or image information uploaded by the user in different social networks. Based on this, the user's preference information may be obtained based on the social behavior information, and then the target background image may be determined from the preset background image library in combination with the user's preference information and the emotional information corresponding to the target song. It can be seen that the disclosed embodiment may determine the target background image according to the user's preferences, thereby stimulating the user's interest in watching dance videos.

[0095] Specifically, the social behavior information of the user on the social network within one month can be obtained. When the social behavior information of the user is mostly about places of interest, multiple images of places of interest can be determined from the preset background image library. At the same time, when the emotion corresponding to the target song is sad, the image corresponding to the Wuhou Temple on a rainy day can be selected from the preset background images as the target background image. Among them, rainy days can render sad emotions, so for sad emotions, images of rainy days can be selected.

[0096] S803, driving a preset virtual character to perform corresponding dance movements through song feature information, and fusing the target background image with the dance movements to generate a dance video.

[0097] In some embodiments, the disclosed embodiments can drive a preset virtual character to perform corresponding dance movements with a target background image as the background according to the song feature information, thereby generating a user-specific dance video. Moreover, the dance video corresponding to the same song can change as the user's preference information changes. Based on this, different dance videos can stimulate the user's interest in watching dance videos.

[0098] It can be seen that the disclosed embodiment can determine the emotional information of the target song according to the song characteristic information corresponding to the target song, and then determine the target background image from the preset background image library according to the emotional information and behavioral characteristic information. At the same time, the preset virtual character is driven by the song characteristic information to perform corresponding dance movements, and the dance video corresponding to the target song is obtained with the target background image as the background. It can be seen that the target background image of the disclosed embodiment will change with the user's behavioral characteristic information, and different dance videos can be obtained through different target background images, and then different dance videos can be displayed to the user when the user plays the same song, thereby enhancing the user's interest in watching the dance video.

[0099] In some embodiments, the behavioral characteristic information includes the user's current location information and the user's social behavior information in the social network, and determining the target background image from a preset background image library based on the behavioral characteristic information and the emotional information includes: determining multiple candidate background images from the preset background image library based on the current location information, wherein multiple candidate background images match the current location information; determining the user's preference information based on the social behavior information, and determining the target background image from the multiple candidate background images based on the preference information and the emotional information.

[0100] Specifically, the disclosed embodiments can classify the user's real-scene preferences, that is, classify the social behavior information that the user usually generates on the social network, that is, classify the text, images, videos and other content sent by the user on the social network, mainly divided into three categories of real-scene preferences. Among them, the first category is the preference for scenic spots, the second category is the preference for landmark buildings, and the third category is the preference for Internet celebrity check-in places; the specific classification method includes the following steps:

[0101] Step 1: Build a background image library of real-life images of different cities.

[0102] In some embodiments, the mainstream open source image datasets in the industry do not have the classification of domestic landmark buildings and Internet celebrity check-in places. Therefore, the disclosed embodiment retrieves the image data in the open source city image dataset (Mapillary Vistas), manually annotates the landmark buildings and Internet celebrity check-in places, retrieves the Chinese landscape database dataset, and uses the keywords of portrait, city, mountain, river, lake, waterfall, and building as classification keywords to perform a preliminary search, and then manually marks the retrieved image set to form a background image library of urban scenic spots. As shown in Table 1, taking Chengdu as an example, the real scene of Chengdu city can be divided into three categories of real scene preference materials:

[0103] Table 1. Classification results of real scene preference

[0104]

[0105]

[0106] Step 2: Construct a training set based on the preset background image library composed of real-life city images, and train the MobileNet image classification model to obtain a trained image classification model. Based on this, the user's sharing behavior in social software or in a three-party social circle can be divided into four behaviors: publishing, collecting, sharing, and browsing. The images shared in the social circle are used as inputs to the image classification model to perform image classification predictions, determine the classification results of the images, and then obtain the user's real-life preferences. Among them, each time a user publishes an image in a social circle, it can be defined as a contribution to the classification preference.

[0107] Step 3: Calculate the user's three categories c respectively u Preference preference, where W p represents the weight corresponding to the publishing behavior, Wc represents the weight corresponding to the collection behavior, Ws represents the weight corresponding to the sharing behavior, and Wv represents the weight corresponding to the browsing behavior. The preference contribution brought by one behavior can be expressed as:

[0108]

[0109] Among them, t represents the monthly difference between the current time and the behavior time, t_0 represents a time decay cycle of 12 months, and the largest value is selected as the user's classification preference, j represents the image sequence number, j = 1, 2..., m, m represents the maximum sequence number; c u represents the preference for different types of real scenes, u=1,2,3, c1 represents the preference for scenic spots, c2 represents the preference for landmark buildings, c3 represents the preference for internet celebrity check-in spots, and W l are the weights corresponding to different behaviors, l = p, c, s, v. Based on this, after determining the preferences corresponding to different real-scene preference categories, the real-scene preference with the maximum preference value is selected as the target real-scene preference.

[0110] In some embodiments, the disclosed embodiments can also determine the target background image of the dance video by the user's geographical location when listening to the song, the emotion corresponding to the target song, and the target real scene preference. Among them, the geographical location can be obtained based on the real-time IP address information of the user currently listening to the song, and it is mainly necessary to obtain the user's current location information, that is, the city where the user is currently located. Among them, the IP address is a unique address of a network device on the Internet. Among them, the IP address is an Internet Protocol address used to identify devices on the network. Based on this, the user's current location information can be determined through the IP address information of the device currently used by the user.

[0111] Specifically, the disclosed embodiment mainly starts from two dimensions and divides the emotions of the song into four emotion categories: anger, happiness, relaxation and sadness. At the same time, the emotions corresponding to the background image are determined according to the three dimensions of color, light and season. Based on this, the emotions corresponding to the background image can be determined according to the emotions of the song.

[0112] For example, color can be a hue. Warm colors, such as red, orange, and yellow, can correspond to happy emotions and can be matched with relaxing and cheerful songs. Cold colors, such as blue, green, and purple, can correspond to sad emotions and can be matched with sad songs. The light dimension can include bright light, soft light, and gloomy light, where bright light can correspond to happy emotions, soft light can correspond to relaxing emotions, and gloomy light can correspond to sad emotions. The season can be spring, summer, autumn, and winter, where spring can correspond to energetic emotions, summer can correspond to happy emotions, spring can correspond to anxious emotions, and winter can correspond to sad emotions.

[0113] In practical applications, taking rock songs as an example, rock songs are mostly full of vitality, relaxing and cheerful songs. Therefore, for rock songs, a cool background can be selected, and the weather corresponding to the background can be spring. At the same time, the background light can be set to be brighter.

[0114] In some embodiments, when driving a preset virtual character to perform corresponding dance movements through song feature information, it includes: determining a skeleton node model corresponding to the preset virtual character; inputting the song feature information into a pre-trained dance movement generation model to obtain a target dance movement, and driving the skeleton node model to perform the target dance movement.

[0115] Specifically, the disclosed embodiment can construct a bone node model corresponding to the virtual character by analyzing the bone structure of the virtual character. The bone node model includes the eyes, mouth, trunk and limbs of the virtual character. Since the process of the virtual character completing the dance action is the process of the bone node change, the bone node position and bone angle corresponding to different dance actions can be determined through the bone node model, thereby controlling the changes of each bone node in the bone node model to complete the dance action.

[0116] like Figure 3 and Figure 4 As shown, the initial state of the virtual character's skeleton node model is as follows Figure 3 As shown, when it is necessary to perform a dance movement, the bone node position data and the bone angle data corresponding to each bone node of the bone node model after completing the dance movement can be determined. At the same time, the bone node position data and the bone angle data corresponding to each bone node in the initial state also need to be determined. Based on this, the bone node driving data can be determined according to the bone node position data and the bone angle data corresponding to each bone node in the initial state and when performing the dance movement. Finally, the bone node model corresponding to the preset virtual character can be driven according to the bone node driving data, and the target dance movement is performed with the target background image as the background to generate a dance video.

[0117] In some embodiments, song feature information is input into a pre-trained dance movement generation model to obtain target dance movements, including: constructing a dance movement database, wherein the dance movement database includes dance movements corresponding to different songs; constructing a sample set according to the dance movement database, and constructing training samples based on the sample set; training a preset model through the training samples to obtain a dance movement generation model.

[0118] like Figure 3 As shown, the disclosed embodiment can construct a bone node model corresponding to the virtual character by analyzing the bone structure of the virtual character. The bone node model includes the eyes, mouth, trunk and limbs of the virtual character. Since the process of the virtual character completing the dance action is the process of the bone node change, the bone node position and bone angle corresponding to different dance actions can be determined through the bone node model, thereby controlling the changes of each bone node in the bone node model to complete the dance action.

[0119] In practical applications, the direction from one shoulder to the other shoulder of the virtual character at the current playback moment can be the x direction, the y direction is the upward body vector, and the z direction is perpendicular to the plane formed by the x and y axes. Then, based on the Euler angle principle, the rotation angle in the motion capture data is converted into bone node position data, where the rotation angle can be the roll angle, pitch angle, and yaw angle. When converting the rotation angle into bone node position data, the Euler angle is first converted into a matrix. The specific formula is:

[0120]

[0121] Among them, R represents the rotation matrix corresponding to the roll angle; P represents the rotation matrix corresponding to the pitch angle; Y represents the rotation matrix corresponding to the yaw angle; the rotation angle around the z-axis is r; the rotation angle around the x-axis is p; the rotation angle around the y-axis is q; according to the properties of the orthogonal matrix, the rotation matrix N(RPY) can be obtained based on the following formula: -1 :

[0122]

[0123] In some embodiments, the rotation matrix N(RPY) -1 The position data of the desired bone node can be calculated by the relative orientation corresponding to the rotation matrix of the intermediate bone node between the desired bone node and the root bone node, and the position data of the desired bone node in the bone node model can be obtained through the derotation and detranslation process, wherein the specific calculation formula of the derotation process is as follows:

[0124]

[0125] Where J0 represents the rotation matrix after derotation; (x0, y0, z0) represents the initial offset of the desired bone node N1, N i Represents the rotation matrix corresponding to all bone nodes between bone node N1 and root bone node Nr, i represents the bone node number, n represents the number corresponding to the last bone node, i = 1, 2, ..., n.

[0126] For the de-translation process, the position offset of the intermediate bone node between the bone node and the root bone node can be calculated. The position offset of the intermediate bone node relative to its parent bone node is expressed as i Therefore, the position offset P of the desired bone node can be expressed as P = O1 + ... + O i +…+O r Among them, O1 represents the position offset of the first intermediate bone node relative to its parent bone node; O i Represents the position offset of the i-th intermediate bone node relative to its parent bone node; r Represents the position offset of the root bone node. It should be understood that the parent bone node and the child bone node are concepts in the tree data structure. The parent bone node refers to the upper-level bone node of a bone node, and the child bone node refers to the lower-level bone node of a bone node.

[0127] In some embodiments, since the movement of the virtual character in space will cause the position coordinates of the skeleton nodes, that is, the change of the skeleton node position data, it is necessary to use the root skeleton node position data P N (x r ,y r ,z r ) calculates the position data of each bone node. Specifically, the change P0 of the bone node position data can be expressed as P0=P i (x,y,z)-P N (x r ,y r ,z r ), where P i (x, y, z) represents the bone node position data corresponding to the i-th bone node. Based on this, the position change of each bone node, that is, the coordinate change, can be obtained. However, after obtaining the position change of each bone node, it is also necessary to calculate the distance between each bone node, the bone angle data, the virtual character's facial orientation, spatial orientation and the weight and smoothness of the force effect from the motion capture data to describe the characteristic parameters of the virtual character's motion biomechanics. Among them, when calculating the distance between bone nodes and the bone angle data, it is necessary to consider it from the perspective of limb structure, and the distance between bone nodes reflects the speed of movement. The distance h between bone nodes can be calculated using the Euclidean distance, and the calculation formula is:

[0128]

[0129] The bone angle data w represents the bending state between adjacent bones, and its calculation formula is:

[0130] w = arccos(h a ·h b / (||h a ||×||h b ||))

[0131] Among them, h a Indicates the distance between bone node a and bone node b; h b Indicates the distance of bone node b relative to bone node a, such as Figure 3 As shown, the bone pair is two connected bones, for example, the right hip and the left hip, and the bone pair angle data is the angle between the line between the right hip and the hip and the line between the left hip and the hip.

[0132] In some embodiments, the disclosed embodiments may also construct a dance movement database, wherein the dance movement database includes dance movements corresponding to different songs; construct a sample set based on the dance movement database; train a preset model through training samples to obtain a dance movement generation model, wherein the training samples are obtained based on the sample set.

[0133] Specifically, the dance movements in the existing dance video can be processed according to the skeleton node model of the virtual character to obtain the dance movement data corresponding to each dance movement in the dance video, and a sample set can be constructed according to the dance movement data. Then, a multi-branch graph convolutional network can be used to construct a dance movement generation model, and the dance movement generation model can be trained according to the dance movement data.

[0134] In some embodiments, constructing training samples based on a sample set in an embodiment of the present disclosure may include: obtaining song feature information corresponding to multiple songs in the sample set, and determining emotional information corresponding to the multiple songs respectively based on the corresponding song feature information; obtaining dance moves corresponding to multiple songs respectively from the sample set, and determining bone node data corresponding to the multiple dance moves respectively, wherein the bone node data includes bone node position data and bone angle data, and the bone angle data is the angle data between adjacent bones; constructing training samples based on the bone node position data corresponding to the multiple dance moves, the bone angle data, and the emotional information of the corresponding songs.

[0135] Specifically, the bone node position data (s1, s2, ..., s n ), where s1 represents the position data of the bone node numbered 1 in the bone node model, s2 represents the position data of the bone node numbered 2 in the bone node model, and sn Represents the position data of the bone node numbered n in the bone node model; the angle data of each bone pair w1, w2, ..., w n and the emotional information of the song e1,e2,…,e n As training parameters, w1 represents the angle data of the bone numbered 1 in the bone node model, w2 represents the angle data of the bone numbered 2 in the bone node model, and w n represents the angle data of the bone pair numbered n in the bone node model; e1 represents the emotional information of the first keyword of the target song, e2 represents the emotional information of the second keyword of the target song, and e n Represents the emotional information of the nth keyword of the target song.

[0136] Figure 4 FIG. 1 is a schematic diagram of a dance movement generation model provided by an exemplary embodiment of the present disclosure. Figure 4 As shown in the figure, in the separable graph convolution module (SSGC), a global spatiotemporal graph is introduced to obtain the global spatial relationship of nodes, so that the dance movement generation model can learn rich dance movements and improve the accuracy of the dance movement generation model. Among them, the data in the processed training samples can be input into the multi-scale mixed temporal convolution module (MMTC), and the temporal information of different dance movements can be adaptively obtained by dynamically fusing the temporal features aggregated by different time skeleton nodes.

[0137] At the same time, the disclosed embodiment also introduces the angle between the skeleton pairs with the same structure to obtain the spatiotemporal information of the motion trajectory of the skeleton nodes, and then splices them with the position branch and the emotion branch in the channel dimension to model the motion information of the skeleton nodes. The channel attention module (CAM) dynamically adjusts the high-level spatiotemporal information to encode the motion features corresponding to different dance movements. The graph gated unit (G-GRU) is then used to sequentially generate dance movements in the future. The spatial position of the node constrained by the static graph of the skeleton nodes can enhance the propagation state of the information in the MLP module. At the same time, the residual link is used to convert the prediction result y at the previous time point into the prediction result y at the previous time point. t-1 Fusion makes the model generate prediction results of bone node dance movements more stably.

[0138] The disclosed embodiment can also automatically adjust the weights of reconstruction loss and emotional loss according to the degree of completion of model training during the training of the dance movement generation model based on training samples. Among them, Wa represents the weight of the model reconstruction loss, Wb represents the weight of the emotional loss, Wa = data set to be trained / overall training data set, Wb = 1-(data set to be trained / trained data set). In the early stages of model training, Wa is relatively large, which can ensure that the generated dance movements can generate standard dance movements based on the bone node position data and the bone angle data. In the later stages of model training, as Wb increases, the model can be guided to generate more emotionally expressive dance movements, while making the song and the corresponding dance movements more integrated. The specific total loss L can be expressed as follows:

[0139] L=Wa*L_recon+Wb*L_emotion

[0140] Among them, L represents the total loss, L_recon represents the reconstruction loss, and L_emotion represents the emotion loss.

[0141] Figure 5 The following is a schematic diagram of a dance action structure corresponding to a skeleton node provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the target song can be input into the trained dance movement generation model, and the trained dance movement generation model generates the dance movements corresponding to the target song. Here, the dance movements are movements simulated by multiple bone nodes of the virtual character, and the dance movements corresponding to the target song are a series of coherent dance movements, and the multiple dance movements can be the same or different.

[0142] In some embodiments, the song feature information includes lyric information and audio information, and the emotional information of the target song is determined based on the song feature information, including determining the lyric emotional information based on the lyric information, and determining the audio emotional information based on the audio information; the emotional information of the target song is determined based on the lyric emotional information and the audio emotional information.

[0143] In some embodiments, when a user opens a song playing software to listen to a song, in response to the user's operation of playing the song, a target song is determined, and then the audio signal of the target song can be analyzed through a convolutional neural network to obtain rhythm features and melody features, and the audio signal can be determined to correspond to the audio emotional information according to the rhythm features and melody features; at the same time, the Jieba word segmentation algorithm can be used to extract keywords from the lyrics corresponding to the target song, and the lyrics emotional information to be expressed by the lyrics can be determined according to the keywords. Among them, the Jieba word segmentation algorithm is a word segmentation algorithm based on probability statistics, which uses the idea of ​​dynamic programming for word segmentation. The Jieba word segmentation algorithm implements three word segmentation strategies: prefix-based word segmentation, maximum match word segmentation based on word frequency, and maximum probability word segmentation based on word meaning. Through the fusion of multiple word segmentation strategies, the Jieba word segmentation algorithm can segment Chinese text more accurately.

[0144] Specifically, firstly, the meaningless words in the target song lyrics need to be removed, where the meaningless words can be conjunctions and prepositions, etc. Then, the lyrics after removing the meaningless words can be subjected to keyword extraction operation through the Jieba word segmentation algorithm to obtain multiple keywords; finally, the multiple keywords can be matched with the emotion words in the preset emotion corpus respectively, and the emotion information of the lyrics corresponding to the multiple keywords can be determined according to the matching results. Among them, the preset emotion prediction library includes multiple preset emotion categories, and each preset emotion category includes multiple emotion words. Based on this, the calculation formula of the lyrics emotion information corresponding to the keyword t can be:

[0145]

[0146] Among them, A represents the number of keywords that belong to the target preset emotion category among multiple keywords, and the number of keywords that successfully match the target preset emotion category including multiple emotion words; B represents the number of keywords that do not belong to the target preset emotion category among multiple keywords, and do not successfully match the target preset emotion category, but successfully match other preset emotion categories including multiple emotion words; C represents the number of keywords that belong to the target preset emotion category among multiple keywords, but do not successfully match the target preset emotion category; D represents the number of keywords that do not belong to the target preset emotion category among multiple keywords, and do not successfully match other preset emotion categories including multiple emotion words; n represents the number of all emotion words in the preset emotion prediction library; FI represents the number of times keyword t appears in the current preset emotion category. Among them, the stronger the correlation between the keyword and the preset emotion category, the higher the chi-square statistic value x 2 The larger the x 2 When it is 0, it means that the keyword has nothing to do with the current preset emotion category.

[0147] In some embodiments, the emotional information of the lyrics corresponding to the target song can be determined based on the emotional information of the lyrics corresponding to multiple keywords. Specifically, the emotion corresponding to the emotional category containing the largest number of keywords can be used as the emotional information of the lyrics corresponding to the target song. For example, the preset emotional categories include sad category, happy category and depressed category. Based on this, when the number of keywords of a song is 10, among which 5 keywords belong to the sad category, 3 keywords belong to the happy category, and 2 keywords belong to the depressed category, since the number of keywords corresponding to the sad category is the largest, it can be determined that the emotional information of the lyrics corresponding to the target song is sad.

[0148] At the same time, the embodiment of the present disclosure can also convert keywords into word vectors through natural language processing technology (word2vec) to facilitate subsequent processing. Among them, word2vec is a word embedding method, which functions to convert words in natural language into dense vectors that can be understood by computers.

[0149] Specifically, when the audio signal of the target song is analyzed by a convolutional neural network to obtain the rhythm characteristics and melody characteristics, the audio signal can be pre-emphasized first. The pre-emphasis process is to make the frequency spectrum trend of the audio signal flat so as to perform spectrum analysis; then the pre-emphasized audio signal is framed according to the time sequence corresponding to the lyrics of the target song when it is played; then the audio signal after frame processing is windowed, and the purpose of windowing is to solve the Gibbs effect caused by the discontinuity of the start and end after framing; then each frame of the audio signal after the windowing process is fast Fourier transformed to obtain the spectrum of each frame of the signal, and then the spectrum of each frame of the signal is converted into Mel frequency scale through a triangular filter, and finally the Mel frequency cepstrum coefficient (MFCC) of each frame of audio is obtained by discrete cosine transform, and some other audio information containing emotional information is extracted, such as: zero crossing rate, spectrum centroid, spectrum bandwidth, spectrum attenuation, spectrum flux, chromaticity vector and chromaticity spectrum. Among them, the Mel frequency scale is a perceptual scale in which listeners judge that pitches of different frequencies are equal to each other, which represents the human ear's perception of equidistant pitch changes; Mel-Frequency Cepstral Coefficients (MFCC) is a feature extraction method commonly used in speech signal processing and audio analysis. It can convert audio signals into a set of feature vectors for subsequent classification, recognition and other tasks; and audio information includes low-level audio information (low level descriptors, LLDs). LLDs refer to some artificially designed low-level features, and LLDs can be used as frame features.

[0150] Figure 2 The following is a flow chart of a song emotion determination model provided by an exemplary embodiment of the present disclosure. Figure 2 As shown in the figure, the song emotion determination model is constructed based on the fusion classification model of convolutional neural network (CNN) and long short-term memory neural network (LSTM). After the user determines the target song, the word vector and audio information corresponding to the target song are obtained according to the above method, and the word vector and audio information are used as the input of the song emotion determination model. Among them, the word vector and spectrogram features can play the role of feature extraction and feature selection through the convolution layer and pooling layer in the CNN network, and output a set of serialized feature vectors, which are input into the LSTM network as new features and output with the attention mechanism. Here, the spectrogram is the speech spectrum graph, which is generally obtained by processing the received time domain signal. Therefore, as long as there is a time domain signal of sufficient time length, it can be used; and the audio information LLDs can be combined into HSFs through statistical methods, and then combined with the word frequency weight vector after the weight is extracted by the chi-square test, and then the feature dimension reduction is performed by DNN. Finally, the feature vectors output by the two network structures are spliced ​​to obtain the fusion feature, and the fusion feature is input into the Softmax layer for classification processing to obtain the lyrics emotion and audio emotion, and then the two results are dimensional decision fused to obtain the emotional information of the target song. Among them, high level statistics functions (HSFs) refer to the features obtained by doing some statistics on the basis of LLDs, such as mean, maximum value, etc. HSFs can be used as discourse features.

[0151] In practical applications, it is assumed that the target song is a love song, and the song is played with classical instruments, the rhythm of the song is relatively slow, and one of the lyrics of the target song is "There is no news from you after we parted". In this case, the audio signal of the target song can be analyzed by a convolutional neural network to obtain the rhythm characteristics and melody characteristics of the target song, and a series of processing is performed based on the rhythm characteristics and melody characteristics to obtain the audio information containing emotional information in the target song; at the same time, the keywords "parted", "no", "you" and "message" in the lyrics are extracted, and the keywords are converted into word vectors through word2vec, and finally the keywords and the corresponding audio information are input into the song emotion determination model. Through the output result of the song emotion model, it is determined that the emotional information of the target song is sad information, that is, the target song is a sad love song. Among them, the audio emotion information corresponding to the target song can be obtained by processing the audio information using the song emotion determination model; the lyrics emotion information corresponding to the target song can be obtained by processing the word vector using the song emotion determination model, and the two results are then dimensional decision fused to obtain the emotional information of the target song.

[0152] Based on this, the disclosed embodiment can use machine learning related algorithms, and can use existing dance videos to train the dance movement generation model. In the training process, not only the audio information and lyrics information of the song are used to train the model, but also the emotional information of the song is used, and the weight of the emotional information is automatically increased according to the training process to guide the model to generate more emotionally expressive dance movements. At the same time, the disclosed embodiment can also classify the three preferences of scenic spots, landmark buildings, and Internet celebrity check-in places according to the user's publishing, browsing, sharing, and collection history in the social circle, map the emotional information of the song to the light, season, and color characteristics of the picture, add the real-time geographical location when listening to the song to search in the preset background image library, obtain the background image, match the virtual character's dress according to the background image, and merge it with the dance movement to generate the final video for the user to watch and share with the social circle when listening, thereby enhancing the user's fun when listening to the song and increasing the stickiness of the user and the song software.

[0153] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of the server. It is understandable that, in order to implement the above functions, the server includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0154] The embodiments of the present disclosure may divide the server into functional units according to the above method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one management module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0155] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a device for generating a dance video, which may be a server or a chip applied to a server. Fig. 9 The structure diagram of a dance video generation device provided by an exemplary embodiment of the present disclosure is as follows. Fig. 9 As shown, the dance video generation device 900 includes:

[0156] The acquisition module 901 is used to acquire the song characteristic information of the target song in response to the user's selection operation on the target song, and determine the emotional information of the target song according to the song characteristic information;

[0157] A determination module 902 is used to obtain the behavior characteristic information of the user, and determine a target background image from a preset background image library according to the behavior characteristic information and the emotion information;

[0158] The driving module 903 is used to drive the preset virtual character to perform corresponding dance movements according to the song feature information, and to merge the target background image with the dance movements to generate a dance video.

[0159] In an optional manner, the behavioral characteristic information includes the current location information of the user and the social behavior information of the user in the social network, and the target background image is determined from a preset background image library based on the behavioral characteristic information and the emotional information. The determination module 902 is also used to determine multiple candidate background images from the preset background image library based on the current location information, wherein multiple candidate background images match the current location information; determine the user's preference information based on the social behavior information, and determine the target background image from the multiple candidate background images based on the preference information and the emotional information.

[0160] In an optional manner, the preset virtual character is driven to perform corresponding dance movements through the song feature information, and the driving module 903 is also used to determine the skeleton node model corresponding to the preset virtual character; the song feature information is input into a pre-trained dance movement generation model to obtain the target dance movement, and the skeleton node model is driven to perform the target dance movement.

[0161] In an optional manner, the dance video generation device 900 also includes an acquisition module 904, which is used to input the song feature information into a pre-trained dance movement generation model to obtain target dance movements, including: constructing a dance movement database, wherein the dance movement database includes dance movements corresponding to different songs; constructing a sample set according to the dance movement database, and constructing training samples based on the sample set; training the preset model through the training samples to obtain the dance movement generation model.

[0162] In an optional manner, the training samples are constructed based on the sample set, and the dance video generation device 900 also includes a construction module 905, and the construction module 905 is used to obtain song feature information corresponding to multiple songs in the sample set, and determine the emotion information corresponding to the multiple songs according to the corresponding song feature information; obtain the dance movements corresponding to the multiple songs from the sample set respectively, and determine the bone node data corresponding to the multiple dance movements respectively, wherein the bone node data includes bone node position data and bone angle data, and the bone angle data is the angle data between adjacent bones; the training samples are constructed according to the bone node position data corresponding to the multiple dance movements, the bone angle data and the emotion information of the corresponding songs.

[0163] In an optional manner, the song characteristic information includes lyric information and audio information, and the emotional information of the target song is determined based on the song characteristic information. The acquisition module 901 is also used to determine the lyric emotional information based on the lyric information, and determine the audio emotional information based on the audio information; determine the emotional information of the target song based on the lyric emotional information and the audio emotional information.

[0164] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; a memory for storing instructions executable by at least one processor; wherein the at least one processor is used to execute instructions to implement the steps of the above method disclosed in the embodiment of the present disclosure.

[0165] Fig.10 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Fig.10 As shown, the electronic device 1000 includes at least one processor 1001 and a memory 1002 coupled to the processor 1001 , and the processor 1001 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0166] The processor 1001 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in the memory 1002, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1001 reads the information in the memory 1002 and completes the steps of the method in combination with its hardware.

[0167] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, for example, Fig.11 The computer system 1100 shown installs the programs constituting the software, and when the various programs are installed, the computer system can execute various functions, including the functions described above. Fig.11 A schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present disclosure.

[0168] Computer system 1100 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Fig.11As shown, the computer system 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the computer system 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0170] A plurality of components in the computer system 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the computer system 1100, and the input unit 1106 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1107 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 may include, but is not limited to, a disk, an optical disk. The communication unit 1109 allows the computer system 1100 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, for example, a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0171] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1102 and / or communication unit 1109. In some embodiments, the computing unit 1101 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0172] The embodiment of the present disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0173] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0175] Fig.12 Schematic diagram of a computer program product provided by an exemplary embodiment of the present disclosure. Fig.12 As shown, the computer program product 1200 includes a computer program 1201, wherein the computer program 1201 implements the above method disclosed in the embodiment of the present disclosure when executed by a processor.

[0176] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0179] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0180] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0181] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating a dance video, characterized in that: include: In response to a user's selection operation on a target song, song feature information of the target song is acquired, and emotional information of the target song is determined according to the song feature information; Acquire the behavior characteristic information of the user, and determine a target background image from a preset background image library according to the behavior characteristic information and the emotion information; The preset virtual character is driven to perform corresponding dance movements through the song feature information, and the target background image and the dance movements are merged to generate a dance video.

2. The method according to claim 1, characterized in that: The behavior characteristic information includes the current location information of the user and the social behavior information of the user in the social network, and determining the target background image from a preset background image library according to the behavior characteristic information and the emotion information includes: Determining a plurality of candidate background images from a preset background image library according to the current location information, wherein a plurality of the candidate background images match the current location information; The preference information of the user is determined according to the social behavior information, and the target background image is determined from the plurality of candidate background images according to the preference information and the emotion information.

3. The method according to claim 1, characterized in that The step of driving a preset virtual character to perform corresponding dance movements by using the song feature information includes: Determine the skeleton node model corresponding to the preset virtual character; The song feature information is input into a pre-trained dance movement generation model to obtain a target dance movement, and the skeleton node model is driven to perform the target dance movement.

4. The method according to claim 3, characterized in that: The step of inputting the song feature information into a pre-trained dance movement generation model to obtain a target dance movement comprises: Constructing a dance movement database, wherein the dance movement database includes dance movements corresponding to different songs; Constructing a sample set according to the dance movement database, and constructing a training sample based on the sample set; The preset model is trained through training samples to obtain the dance movement generation model.

5. The method according to claim 4, characterized in that The step of constructing a training sample based on the sample set includes: Acquire song feature information corresponding to a plurality of songs in the sample set, and determine emotion information corresponding to the plurality of songs respectively according to the corresponding song feature information; Acquire dance movements corresponding to a plurality of the songs from the sample set, and determine the bone node data corresponding to the plurality of the dance movements, wherein the bone node data includes bone node position data and bone angle data, and the bone angle data is angle data between adjacent bones; The training samples are constructed based on the skeletal node position data, skeletal angle data and emotional information of the corresponding songs corresponding to the multiple dance movements.

6. The method according to claim 1, characterized in that The song feature information includes lyrics information and audio information, and determining the emotional information of the target song according to the song feature information includes: Determine lyrics emotion information according to the lyrics information, and determine audio emotion information according to the audio information; The emotional information of the target song is determined according to the lyrics emotional information and the audio emotional information.

7. A dance video generation device, characterized in that: include: An acquisition module, configured to acquire song feature information of the target song in response to a user's selection operation on the target song, and determine the emotional information of the target song according to the song feature information; A determination module, used to obtain the behavior characteristic information of the user, and determine a target background image from a preset background image library according to the behavior characteristic information and the emotion information; The driving module is used to drive the preset virtual character to perform corresponding dance movements according to the song feature information, and to fuse the target background image with the dance movements to generate a dance video.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Citation Information

Cited By

  • SOR optimization system and method for tourism dance immersion experience

    CN121300637A

  • A sor optimization system and method for a travel dance immersive experience

    CN121300637B