Dancing motion generation method

Abstract

The invention discloses a dance motion generation method, and belongs to the field of dance orchestration. Rhythm extraction: analyzing the waveform of music by using an audio processing algorithm, and determining the rhythm, speed and accent position of the music; extracting rhythm features by using an autocorrelation function and a short-time Fourier transform method; the whole dance motion generation process does not need human intervention, and automation is achieved; the efficiency of dance creation is greatly improved, and manpower and time costs are saved. Meanwhile, subjectivity and uncertainty caused by human factors are avoided, the generated dance is more objective and stable, and proper elements can be selected from a large number of dance actions through the steps of action library construction and action selection. The action library covers actions of different types, emotional expressions and difficulty levels, and provides rich materials for dance creation. In this way, various dances can be generated, and the requirements and preferences of different users can be met.

Description

Technical Field

[0001] The present invention relates to the field of dance choreography, and particularly to a method for generating dance movements. Background Art

[0002] A dance generation method is a way to create dance works through specific technical means and processes.

[0003] In traditional dance generation methods, professional choreographers spend a lot of time on movement design and choreography. Especially for a short dance, it may take days or even weeks to complete. At the same time, motion capture and post-processing also require a lot of effort from professionals, and the whole process is inefficient.

[0004] In existing methods, once the choreography is completed, any changes to the music and movements require repeating the entire cumbersome process, including re-doing movement design, capture, and post-processing. This makes dance creation lack flexibility in the face of changes and difficult to quickly adapt to different needs;

[0005] Traditional methods usually rely on the personal experience and creativity of choreographers and do not make full use of a large amount of dance data and advanced machine learning algorithms. This results in relatively single dance movements, lacking diversity and innovation.

[0006] Therefore, a method for generating dance movements is needed to solve the problems mentioned above. Summary of the Invention

[0007] In view of the deficiencies of the prior art, the present invention provides a method for generating dance movements, which solves the problems raised in the above background art.

[0008] Technical Solution: To solve the above technical problems, according to one aspect of the present invention, more specifically, a method for generating dance movements is as follows:

[0009] S1. Music Analysis

[0010] Rhythm extraction: Using audio processing algorithms, analyze the waveform of the music to determine its beats, tempo, and accent positions; use autocorrelation functions and short-time Fourier transform methods to extract rhythm features;

[0011] Emotion analysis: Adopt machine learning algorithms to train an emotion classification model to classify the emotion of the music; consider the melody, harmony, and timbre features of the music, as well as the emotional tendency of the lyrics;

[0012] Structure analysis: Analyze the structure of the music, including the prelude, verse, chorus, and interlude parts; determine the structure of the music by detecting energy changes and melody repetition features in the music;

[0013] S2. Action Library Construction

[0014] Action Capture: Collect a large number of dance videos and use computer vision technology to extract the dance actions in them; Use motion capture software or deep learning algorithms to track and analyze the human actions in the dance videos;

[0015] Action Classification and Annotation: Classify and annotate the captured actions for subsequent action selection and combination; Classify according to the type of action, emotional expression, and difficulty level;

[0016] Action Storage and Management: Store the classified and annotated actions in a database for subsequent quick retrieval and use; Use relational databases or non-relational databases to store and manage the action data;

[0017] S3. Action Generation

[0018] Action Selection: Select appropriate action elements from the action library according to the results of music analysis; Use rule-based methods or machine learning algorithms for action selection;

[0019] Action Combination: Combine the selected action elements to generate a preliminary dance action sequence; Use random combination or rule-based combination methods;

[0020] Action Optimization: Optimize the preliminary generated dance action sequence to improve the quality and expressiveness of the actions; Use machine learning algorithms and optimization algorithms for action optimization;

[0021] S4. Dance Choreography

[0022] Paragraph Planning: Divide the optimized dance action sequence into different paragraphs according to the structure and emotional changes of the music; Use rule-based methods or machine learning algorithms for paragraph planning;

[0023] Space Utilization: Consider the spatial environment of the dance performance, such as the size, shape, and height of the stage, and design the movement and layout of the dance actions in space; Use computer graphics technology for space planning and simulation;

[0024] Dynamic Adjustment: Continuously perform dynamic adjustment during the dance choreography process to improve the quality and expressiveness of the dance; Use machine learning algorithms or optimization algorithms for dynamic adjustment;

[0025] S5. Output and Evaluation

[0026] Dance Output: Output the generated dance in the form of video, animation, or other forms for users to watch and evaluate; Use video editing software or animation production software to convert the dance action sequence into a visual dance work;

[0027] Dance evaluation: Automatically evaluate the generated dance and provide feedback on the quality and expressiveness of the dance; use machine learning algorithms or evaluation metrics for dance evaluation.

[0028] Furthermore, for the action capture in step S2, the steps are as follows:

[0029] 1. Action capture preparation

[0030] a. Determine the dance video data source: Collect a large number of dance videos of different styles and types, and conduct a preliminary screening of the collected videos to remove videos with blurred image quality, unclear or incomplete actions, so as to improve the accuracy of subsequent action extraction;

[0031] b. Select action extraction techniques: Use action capture software or deep learning algorithms in computer vision technology; adopt MotionBuilder action capture software to track actions by marking key body parts; the deep learning algorithm uses the AlphaPose human pose estimation model based on convolutional neural networks;

[0032] 2. Feature extraction steps

[0033] a. Human joint position extraction: For the case of using action capture software, set marker points in the dance video, using the head, neck, shoulders, elbows, wrists, hips, knees, and ankle joints as marker points, and track the position changes of the marker points in the video to obtain the coordinate information of the human joints;

[0034] Sort out and store the extracted joint position data for subsequent analysis; record the coordinate information of each joint in chronological order to form a time series data;

[0035] b. Motion trajectory analysis: According to the time series data of joint positions, calculate the displacement and speed of each joint at different time points; the displacement is obtained by calculating the difference between the joint coordinates of two adjacent time points, and the speed is the displacement divided by the time interval;

[0036] Analyze the motion trajectory of the whole body, represented by calculating the displacement and speed of the body center of gravity; at the same time, pay attention to the motion trajectory of specific body parts and analyze their motion direction, amplitude, and frequency characteristics;

[0037] c. Key frame extraction of actions: Key frames are representative frames in dance actions that can summarize the main features of the actions; determine key frames by calculating the change in joint positions or motion speeds between adjacent frames; when the change exceeds a certain threshold, the frame is considered a key frame; if the position of a certain joint changes significantly in several consecutive frames, indicating that an important change has occurred in the action, that frame is determined as a key frame;

[0038] The extracted key frames are used as representative samples of actions for subsequent action classification and annotation;

[0039] 3. Feature analysis steps

[0040] a. Action type classification: Classify the actions according to the extracted features; the action types include but are not limited to jumping, rotating, stretching, bending, swinging;

[0041] Use machine learning algorithms for action type classification. First, collect a batch of dance video data with labeled action types, extract features and use them as the training set; then, use the training set to train the classification model so that it can predict the action type according to the new action features; through the classification of a large number of actions, establish a database of action types to provide a basis for subsequent action selection and combination;

[0042] b. Emotional expression analysis: Analyze the emotional expression conveyed by the actions, such as cheerful, sad, exciting, soothing; the emotional expression of the actions is reflected by their speed, amplitude, strength, and body posture features;

[0043] Also use machine learning algorithms for emotional expression analysis; collect dance video data with labeled emotional expressions, extract features and use them as the training set; train the classification model so that it can predict the emotional expression according to the action features;

[0044] c. Difficulty level assessment: Assess the difficulty level of the actions, considering factors such as body flexibility, strength requirements, and coordination requirements;

[0045] Formulate an evaluation index system to determine the difficulty level according to the range of joint activities, muscle strength requirements, and action complexity factors; for each factor, set different levels, and comprehensively consider these factors to determine the overall difficulty level of the action.

[0046] The beneficial effects of a method for generating dance actions of the present invention are as follows:

[0047] (1). The entire process of generating dance actions of the present invention does not require human intervention, realizing automation; greatly improving the efficiency of dance creation, saving labor and time costs. At the same time, it avoids the subjectivity and uncertainty brought by human factors, making the generated dance more objective and stable.

[0048] (2). Through the action library construction and action selection steps of the present invention, it is possible to select appropriate elements from a large number of dance actions. The action library covers actions of different types, emotional expressions, and difficulty levels, providing rich materials for dance creation. In this way, diverse dances can be generated to meet the needs and preferences of different users.

[0049] (3) In the music analysis stage of the present invention, the rhythm, emotion, and structural features of the music are deeply extracted, and these features are fully considered during the movement generation and choreography process, making the generated dance movements highly matched with the music; enhancing the expressiveness and appeal of the dance, and being able to better convey the emotions and themes expressed by the music.

[0050] (4) The present invention automatically evaluates the generated dance through a dance evaluation model, and can timely feedback the quality and expressiveness of the dance. At the same time, using video editing software and animation production software can transform the dance movement sequence into high-quality video or animation works, more vividly showing the effect of the dance, and providing a better viewing experience for users.

[0051] (5) The present invention can adapt to different styles, different types of music, as well as different dance performance space environments. Whether it is lively pop music, sad classical music, or large-scale stage performances, small studio recordings, it can generate appropriate dance movements and choreography, with strong adaptability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described in detail below with reference to the drawings and specific implementation methods.

[0053] Figure 1 It is a schematic structural diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0054] The present invention will be described in detail below with reference to the drawings and embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0055] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0056] Refer to Figure 1 , a method for generating dance movements, characterized by the following steps:

[0057] S1. Music analysis

[0058] Rhythm extraction: Using audio processing algorithms, analyze the waveform of the music to determine its beats, tempo, and accent positions; use autocorrelation functions and short-time Fourier transform methods to extract rhythm features;

[0059] By calculating the autocorrelation function of the audio signal, find the periodic peaks to determine the beats of the music; calculate the tempo of the music according to the time interval between the peaks;

[0060] Sentiment analysis: Using machine learning algorithms to train a sentiment classification model to classify the sentiment of music; considering the melody, harmony, timbre features of music, as well as the sentiment tendency of lyrics;

[0061] Using a deep learning model to analyze the spectrogram of music through a convolutional neural network (CNN) to determine whether the sentiment of the music is cheerful, sad, exciting or soothing;

[0062] Structure analysis: Analyze the structure of music, including the intro, verse, chorus, and interlude parts; determine the structure of music by detecting the energy changes and melody repetition features in the music;

[0063] And by calculating the short-time energy of the music signal, find the energy peaks in the music, which are important structural parts of the music; at the same time, determine the positions of the verse and chorus by analyzing the repetition patterns of the melody;

[0064] S2. Action library construction

[0065] Action collection: Collect a large number of dance videos and use computer vision technology to extract the dance actions in them; use motion capture software or deep learning algorithms to track and analyze the human actions in the dance videos;

[0066] Use the OpenPose human pose estimation software to extract the body joint positions and movement trajectories of the dancers from the dance videos as the basic data for the action library;

[0067] Action classification and annotation: Classify and annotate the collected actions for subsequent action selection and combination; classify according to the type of action (such as jumping, spinning, stretching), emotional expression (such as cheerful, sad, exciting), and difficulty level;

[0068] Use machine learning algorithms to automatically classify and annotate the actions; train a classifier to classify the actions into different categories according to the characteristics of the actions (such as joint angle changes, speed, acceleration);

[0069] Action storage and management: Store the classified and annotated actions in a database for subsequent quick retrieval and use; use a relational database or a non-relational database to store and manage the action data;

[0070] Use the MySQL relational database to store information such as the name, type, emotional expression, difficulty level, and action description of the actions; establish an indexing and search mechanism to facilitate the quick search for specific types of actions;

[0071] S3. Action generation

[0072] Action Selection: Select appropriate action elements from the action library based on the results of music analysis; Use rule-based methods or machine learning algorithms for action selection;

[0073] For lively music, select actions with lively emotional expressions and fast rhythms; For sad music, select actions with sad emotional expressions and slow rhythms; At the same time, according to the structure of the music, select corresponding action combinations, such as selecting simple actions in the prelude part and complex actions in the chorus part;

[0074] Action Combination: Combine the selected action elements to generate a preliminary dance action sequence; Use random combination or rule-based combination methods;

[0075] Use genetic algorithm optimization algorithms to optimize the action combination; Transform the action combination problem into an optimization problem, and through continuous crossover, mutation, and selection, generate action sequences with high fitness; The fitness function is designed based on factors such as the matching degree between actions and music and the naturalness of transitions between actions;

[0076] Action Optimization: Optimize the preliminarily generated dance action sequence to improve the quality and expressiveness of the actions; Use machine learning algorithms and optimization algorithms for action optimization;

[0077] Use reinforcement learning algorithms to train an agent, and through interaction with the environment, continuously adjust the action sequence to improve the matching degree between actions and music and the expressiveness of the actions; The agent obtains rewards or punishments based on the execution effects of the actions, thereby guiding it to learn the optimal action sequence;

[0078] S4. Dance Choreography

[0079] Paragraph Planning: Divide the optimized dance action sequence into different paragraphs according to the structure and emotional changes of the music; Use rule-based methods or machine learning algorithms for paragraph planning;

[0080] According to the prelude, verse, chorus, and interlude parts of the music, divide the dance action sequence into corresponding paragraphs; Each paragraph has a different theme and style to better express the content of the music;

[0081] Space Utilization:

[0082] Consider the spatial environment of the dance performance, such as the size, shape, and height of the stage, and design the movement and layout of the dance actions in space; Use computer graphics technology for space planning and simulation;

[0083] Use 3D modeling software to create a virtual model of the stage, then map the dance movement sequence onto the virtual stage for spatial planning and simulation; adjust the range and direction of the dance movements according to the size and shape of the stage; adjust the vertical height and jumping height of the dance movements according to the height of the stage.

[0084] Dynamic adjustment: Continuously perform dynamic adjustment during the choreography process to improve the quality and expressiveness of the dance; use machine learning algorithms or optimization algorithms for dynamic adjustment.

[0085] Use the particle swarm optimization algorithm to optimize and adjust the parameters of the dance movements; according to the objective function of the dance, such as the matching degree between the movements and the music, and the expressiveness of the movements, continuously adjust the parameters of the dance movements, such as the amplitude, speed, and direction of the movements, to improve the quality and expressiveness of the dance.

[0086] S5. Output and evaluation

[0087] Dance output: Output the generated dance in the form of video, animation, or other forms for users to watch and evaluate; use video editing software or animation production software to convert the dance movement sequence into a visual dance work.

[0088] Use video editing software to synthesize the dance movement sequence with music, add special effects and subtitles to generate a high-quality dance video work; also use Blender animation production software to convert the dance movement sequence into a 3D animation work to more vividly display the effect of the dance.

[0089] Dance evaluation: Automatically evaluate the generated dance and feedback the quality and expressiveness of the dance; use machine learning algorithms or evaluation metrics for dance evaluation.

[0090] Use a deep learning algorithm to train a dance evaluation model to automatically evaluate the generated dance; the evaluation model learns the relationship between the characteristics and quality of the dance, such as the fluency of the movements, the matching degree with the music, and the expressiveness; according to the output of the evaluation model, feedback the quality and expressiveness of the dance to provide a reference for subsequent dance generation.

[0091] Preferably, for the action collection in step S2, the steps are as follows:

[0092] 1. Action collection preparation

[0093] a. Determine the dance video data source: Collect a large number of dance videos of different styles and types, and conduct a preliminary screening on the collected videos to remove videos with blurred picture quality, unclear or incomplete movements to improve the accuracy of subsequent action extraction.

[0094] b. Selection of action extraction technology: Use motion capture software or deep learning algorithms in computer vision technology; Adopt MotionBuilder motion capture software to track actions by marking key body parts; The deep learning algorithm uses the AlphaPose human pose estimation model based on convolutional neural network (CNN).

[0095] 2. Feature extraction steps

[0096] a. Extraction of human joint positions: For the case of using motion capture software, set marker points in the dance video. Use the head, neck, shoulders, elbows, wrists, hips, knees, and ankle joints as marker points to track the position changes of the marker points in the video and obtain the coordinate information of human joints.

[0097] Organize and store the extracted joint position data for subsequent analysis; Record the coordinate information of each joint in chronological order to form a time series data.

[0098] b. Motion trajectory analysis: According to the time series data of joint positions, calculate the displacement and speed of each joint at different time points; The displacement is obtained by calculating the difference between the joint coordinates of two adjacent time points, and the speed is the displacement divided by the time interval.

[0099] Analyze the motion trajectory of the whole body, which is represented by calculating the displacement and speed of the body center of gravity; At the same time, pay attention to the motion trajectories of specific body parts and analyze their motion direction, amplitude, and frequency characteristics.

[0100] c. Extraction of key frames of actions: Key frames are representative frames in dance actions that can summarize the main features of the actions; Determine the key frames by calculating the changes in joint positions or motion speeds between adjacent frames; When the change exceeds a certain threshold, the frame is considered a key frame; If the position of a certain joint changes significantly in several consecutive frames, it indicates that an important change has occurred in the action, and this frame is determined as a key frame.

[0101] The extracted key frames are used as representative samples of the actions for subsequent action classification and annotation.

[0102] 3. Feature analysis steps

[0103] a. Action type classification: Classify the actions according to the extracted features; Action types include but are not limited to jumping, rotating, stretching, bending, and swinging.

[0104] For action type classification using machine learning algorithms, first, collect a batch of dance video data with labeled action types, extract features, and use them as the training set. Then, train a classification model using the training set so that it can predict the action type based on new action features. By classifying a large number of actions, establish a database of action types to provide a basis for subsequent action selection and combination.

[0105] b. Emotional expression analysis: Analyze the emotional expression conveyed by the actions, such as cheerful, sad, exciting, and soothing. The emotional expression of the actions is reflected by their speed, amplitude, strength, and body posture features.

[0106] Similarly, use machine learning algorithms for emotional expression analysis. Collect dance video data with labeled emotional expressions, extract features, and use them as the training set. Train a classification model so that it can predict the emotional expression based on the action features.

[0107] c. Difficulty level assessment: Assess the difficulty level of the actions, considering factors such as body flexibility, strength requirements, and coordination requirements.

[0108] Formulate an evaluation index system to determine the difficulty level based on factors such as the range of joint movement, muscle strength requirements, and action complexity factors. For each factor, set different levels and comprehensively consider these factors to determine the overall difficulty level of the action.

[0109] The above-described embodiments only represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A method for generating dance movements, characterized in that, The following steps: S1. Music analysis Rhythm extraction: Using audio processing algorithms, analyze the waveform of the music to determine its beat, tempo, and accent positions; Use autocorrelation functions and short-time Fourier transform methods to extract rhythm features; Emotion analysis: Adopt machine learning algorithms to train an emotion classification model to classify the emotion of the music; Consider the melody, harmony, and timbre features of the music, as well as the emotional tendency of the lyrics; Structure analysis: Analyze the structure of the music, including the prelude, verse, chorus, and interlude parts; Determine the structure of the music by detecting energy changes and melody repetition features in the music; S2. Action library construction Action collection: Collect a large number of dance videos and use computer vision technology to extract the dance actions in them; Use motion capture software or deep learning algorithms to track and analyze the human actions in the dance videos; Action classification and annotation: Classify and annotate the collected actions for subsequent action selection and combination; Classify according to the type, emotional expression, and difficulty level of the actions; Action storage and management: Store the classified and annotated actions in a database for subsequent quick retrieval and use; Use relational databases or non-relational databases to store and manage the action data; S3. Action generation Action selection: According to the results of music analysis, select appropriate action elements from the action library; Use rule-based methods or machine learning algorithms for action selection; Action combination: Combine the selected action elements to generate a preliminary dance action sequence; Use random combination or rule-based combination methods; Action optimization: Optimize the preliminary generated dance action sequence to improve the quality and expressiveness of the actions; Use machine learning algorithms and optimization algorithms for action optimization; S4. Dance choreography Paragraph planning: According to the structure and emotional changes of the music, divide the optimized dance action sequence into different paragraphs; Use rule-based methods or machine learning algorithms for paragraph planning; Space utilization: Consider the spatial environment of the dance performance, such as the size, shape, and height of the stage, and design the movement and layout of the dance actions in space; Use computer graphics technology for space planning and simulation; Dynamic adjustment: During the dance choreography process, continuously perform dynamic adjustment to improve the quality and expressiveness of the dance; Use machine learning algorithms or optimization algorithms for dynamic adjustment; S5. Output and evaluation Dance output: Output the generated dance in the form of video, animation, or other forms for users to watch and evaluate; Use video editing software or animation production software to convert the dance action sequence into a visual dance work; Dance evaluation: Automatically evaluate the generated dance and provide feedback on the quality and expressiveness of the dance; Use machine learning algorithms or evaluation metrics for dance evaluation.

2. The method for generating a dance movement according to claim 1, wherein: For the action collection in step S2, the steps are as follows:

1. Action collection preparation a. Determine the dance video data source: Collect a large number of dance videos of different styles and types, and conduct a preliminary screening of the collected videos to remove videos with blurred image quality, unclear or incomplete actions to improve the accuracy of subsequent action extraction; b. Selection of action extraction technology: Use motion capture software or deep learning algorithms in computer vision technology; adopt MotionBuilder motion capture software to track actions by marking key body parts; the deep learning algorithm uses the AlphaPose human pose estimation model based on convolutional neural networks; 2. Feature extraction steps a. Extraction of human joint positions: For the case of using motion capture software, set marker points in the dance video, using the head, neck, shoulders, elbows, wrists, hips, knees, and ankle joints as marker points, and track the position changes of the marker points in the video to obtain the coordinate information of human joints; Organize and store the extracted joint position data for subsequent analysis; record the coordinate information of each joint in chronological order to form a time series data; b. Analysis of motion trajectories: According to the time series data of joint positions, calculate the displacement and speed of each joint at different time points; the displacement is obtained by calculating the difference between the joint coordinates of two adjacent time points, and the speed is the displacement divided by the time interval; Analyze the motion trajectory of the entire body, represented by calculating the displacement and speed of the body's center of gravity; at the same time, pay attention to the motion trajectories of specific body parts and analyze their motion direction, amplitude, and frequency characteristics; c. Extraction of key frames of actions: Key frames are representative frames in dance actions that can summarize the main features of the actions; determine key frames by calculating the changes in joint positions or motion speeds between adjacent frames; when the change exceeds a certain threshold, consider this frame as a key frame; if the position of a certain joint changes significantly in several consecutive frames, indicating that an important change has occurred in the action, this frame is determined as a key frame; The extracted key frames are used as representative samples of actions for subsequent action classification and annotation; 3. Feature analysis steps a. Classification of action types: Classify the actions according to the extracted features; action types include but are not limited to jumping, rotating, stretching, bending, and swinging; Use machine learning algorithms for action type classification. First, collect a batch of dance video data with labeled action types, extract features and use them as the training set; then, use the training set to train the classification model so that it can predict the action type according to the new action features; through the classification of a large number of actions, establish a database of action types to provide a basis for subsequent action selection and combination; b. Analysis of emotional expression: Analyze the emotional expression conveyed by the actions, such as cheerful, sad, exciting, and soothing; the emotional expression of the actions is reflected by their speed, amplitude, strength, and body posture characteristics; Also use machine learning algorithms for emotional expression analysis; collect dance video data with labeled emotional expressions, extract features and use them as the training set; Train the classification model so that it can predict the emotional expression according to the action features; c. Evaluation of difficulty level: Evaluate the difficulty level of the actions, considering factors such as body flexibility, strength requirements, and coordination requirements; Formulate an evaluation index system, and determine the difficulty level according to the range of motion of joints, the strength requirements of muscles, and the complexity factors of movements; for each factor, set different levels, and comprehensively consider these factors to determine the overall difficulty level of the movement.

Images