A robot dance automatic generation method and system based on music feature analysis

Abstract

The application provides a robot dance automatic generation method and system based on music feature analysis, and relates to the technical field of robot control. First, a music feature spectrum covering rhythm, energy and other dimensions is constructed to convert abstract music into structured information understandable by machines, and then a dance semantic label sequence bound to a time axis is generated to provide accurate basis for action selection. By combining with a forward-looking window to predict future music features, the current and future candidate actions are filtered through a multi-objective model to realize the dual matching of action and music micro-rhythm and macro-structure. The finally generated control instruction sequence directly drives the physical robot, which breaks the dependence on video materials, realizes the automatic generation of dance with creativity and artistic expression, and improves the generation efficiency and adaptive flexibility.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method and system for automatically generating robot dances based on music feature analysis. Background Technology

[0002] With the rapid development of robotics technology, service and entertainment robots are increasingly being used in the performing arts. Robot dance performances, as an important form of showcasing the fusion of technology and art, present a core challenge: how to enable robots to automatically generate high-quality dance sequence sequences based on music.

[0003] Currently, most mainstream robot dance motion generation relies on pre-recorded human dance videos or motion capture data. The common approach is to first acquire high-quality human dance videos, then use computer vision algorithms to extract key points or 3D pose sequences, then map the human poses onto the target robot model using motion retargeting technology, and finally conduct long-term reinforcement learning training in a physical simulation environment to generate stable and executable robot joint control commands. There are also some video-free solutions, such as real-time matching methods based on a basic dance motion library. These rely on a predefined, fixed "table of basic dance motion connections" and determine the next motion by querying pre-stored "motion pair" probabilities.

[0004] However, existing methods rely on video material libraries or preset rule libraries, which are costly to acquire and limited by the style and content of the source videos. While video-free solutions eliminate video dependence, their splicing logic based on local probabilities lacks a global understanding of the deep structure of the music. The resulting dances are often mechanical stacks of movements, lacking artistic expression and a sense of transition. In summary, existing methods lack a dynamic response to the deep emotions and overall structure of music, making it difficult to flexibly create complete dance choreography with artistic expression. Summary of the Invention

[0005] This application provides a method and system for automatically generating robot dance based on music feature analysis, which is used to generate high-quality robot dance movement sequences with logical transitions, natural and smooth movements, and the ability to deeply interpret the deep emotions of the music.

[0006] In a first aspect, this application provides a method for automatically generating robot dance based on music feature analysis. The method includes: acquiring audio data associated with target music and constructing a multi-dimensional music feature map, which includes the music's rhythm, energy, spectral features, macroscopic music structure, and emotional tags; generating a dance semantic tag sequence on a timeline based on the multi-dimensional music feature map; accessing a pre-set parametric dance action meta-language library, which stores multiple atomic dance actions; acquiring the current dance semantic tag sequence for the current time period and the future dance semantic tag sequence within a pre-set lookahead window, and then retrieving current candidate actions and future candidate actions from the parametric dance action meta-language library; determining a target dance action from the current candidate actions and a future target dance action from the future candidate actions using a multi-objective real-time determination model; generating a robot joint control command sequence containing the target dance action and the future target dance action and sending it to the robot to generate the dance.

[0007] By employing the aforementioned technical solution, a music feature atlas encompassing multiple dimensions such as rhythm and energy is first constructed, transforming abstract music into structured information that machines can understand. Then, a time-bound sequence of dance semantic tags is generated, providing a precise basis for movement selection. Combining this with a look-ahead window to predict future music features, a multi-objective model filters current and future candidate movements, achieving a dual fit between movement and the music's micro-rhythm and macro-structure. The final generated control command sequence directly drives the physical robot, eliminating reliance on video footage and enabling automatic dance generation that combines creativity and artistic expression, improving generation efficiency and adaptability.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the atomic dance movements in the parameterized dance movement meta-language library are organized in the form of a knowledge graph, each atomic dance movement including machine-understandable metadata, which includes BPM range, energy level, style vector, joint control keyframe sequence, and transition compatibility vector with adjacent movements.

[0009] By adopting the above technical solutions, the associative characteristics of the knowledge graph enable the system to efficiently match the connection relationships between actions. The transition compatibility vector directly quantifies the naturalness of action connections, avoiding mechanical stacking. At the same time, the joint control keyframe sequence provides accurate data support for robot execution, ensuring that the actions not only meet stylistic requirements but also have executability, significantly improving the smoothness and execution stability of dance sequences.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, after obtaining the audio data associated with the target music, the step of constructing a multi-dimensional music feature map specifically includes: extracting multiple acoustic feature vector sequences from the audio data, wherein the acoustic feature vector sequences include at least beat timing, energy profile, and spectral distribution features; performing self-similarity matrix analysis based on the acoustic feature vector sequences to determine the macroscopic structural segments of the music; determining the audio information corresponding to the acoustic feature vector sequences within the macroscopic structural segments and the text information extracted from the lyrics text corresponding to the target music; inputting the audio information and the text information into a transformer-based cross-modal attention model to generate the music sentiment label, wherein the cross-modal attention model is used to capture the semantic conflict features between the audio information and the text information to identify semantically reversed sentiment.

[0011] By employing the aforementioned technical solutions, multiple acoustic feature vector sequences are extracted, providing a comprehensive data foundation for music analysis. Self-similarity matrix analysis accurately identifies macroscopic structural segments, enabling synchronization between dance and music segments. A cross-modal attention model integrates audio and lyric text information, accurately capturing semantic conflicts between the two and effectively identifying semantically reversed emotions. This technical process ensures that the music feature map contains both surface rhythmic energy features and deep emotional connotations, providing rich semantic support for subsequent choreography. This allows the dance to accurately express the core emotions of the music, enhancing its artistic appeal.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the steps of determining a target dance movement from the current candidate movements and a future target dance movement from the future candidate movements using a multi-objective real-time determination model specifically include: calculating and determining a first comprehensive matching score for each current candidate movement using the multi-objective determination model, and determining the top n movements with the highest scores as initial target dance movements; calculating and determining a second comprehensive matching score for each future target dance movement using the multi-objective determination model, and determining the top n movements with the highest scores as initial future target dance movements; determining a transition connection score from each initial target dance movement to the initial future target dance movement, the transition connection score being used to characterize the naturalness and smoothness of the transition from the current candidate movement to the future candidate movement; and determining the target dance movement and the future target dance movement based on the set of movement sequences with the highest transition connection scores.

[0013] By employing the aforementioned technical solution, the multi-objective model first selects high-scoring initial and future target movements to ensure the high quality of individual movements. Then, by calculating transition scores, the naturalness of the transitions between movements is quantified, and the optimal sequence is selected from multiple movement combinations. This process ensures both the matching degree of individual movements with the music and the overall coherence of the movement sequence, avoiding the problem of local optima leading to global imbalance. The resulting dance sequence is both in line with the characteristics of the music and flows smoothly and naturally, possessing a complete artistic choreography logic.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the step of calculating and determining a first comprehensive matching score for each current candidate action using a multi-objective determination model further includes: determining the beat fit degree based on the matching degree between the preset BPM range corresponding to the current candidate action and the current BPM of the music; determining the energy alignment degree based on the matching degree between the energy level required by the current candidate action and the current energy level of the music; determining the timing synchronization accuracy based on the alignment error between the keyframe of the current candidate action and the music timing timestamp; calculating the connection naturalness between the current candidate action and the determined previous action based on the predefined transition compatibility vector in the parameterized dance action meta-language library, and determining the transition smoothness based on the connection naturalness; determining the style consistency degree based on the matching degree between the style vector of the current candidate action and the overall style and music emotional label of the current dance segment; and determining the first comprehensive matching score by weighted summation based on at least one of the objective evaluations of beat fit, energy alignment, timing synchronization accuracy, transition smoothness, and style consistency.

[0015] By employing the aforementioned technical solution, candidate movements are quantitatively evaluated from multiple core dimensions, including rhythm, energy, and timing. Each dimension corresponds to a key characteristic of the music or a core requirement of the movement. A weighted summation is used to calculate a comprehensive matching score, achieving a holistic and objective assessment of the movements and avoiding the bias caused by single-dimensional evaluation. This evaluation method transforms subjective choreography logic into calculable quantitative indicators, ensuring that the selected movements achieve optimal performance in terms of rhythmic accuracy, energy adaptability, and stylistic consistency, thereby enhancing the harmony between dance and music.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: when the music emotion label is semantic inversion emotion, adjusting the weight coefficient of the multi-objective real-time determination model to increase the style consistency, so as to preferentially select dance movements that contrast with the surface emotion of the music in style and intensity to express semantic inversion meaning.

[0017] By employing the aforementioned technical solutions, dance transcends the limitations of superficial rhythmic matching, delving into the deeper semantic connotations of music. Through contrasting movement styles and intensity, it precisely conveys semantically reversed emotions. This technology enables dance to accurately express complex emotions, preventing emotional expression from becoming superficial and further enhancing the artistic depth and expressiveness of dance.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, before generating a sequence of robot joint control commands containing the target dance movement and the future target dance movement and sending it to the robot to generate a dance, the method further includes: choreographing a sequence of multiple target dance movements and the future target dance movement to construct an initial dance sequence to be executed; performing a multi-dimensional quantitative evaluation of the initial dance sequence to be executed using a preset dance evaluation function to obtain multi-dimensional evaluation index results; adjusting the weight parameters and choreography rules of the multi-objective real-time determination model based on the multi-dimensional evaluation index results; and using the adjusted multi-objective real-time determination model to correct the initial dance sequence to be executed to obtain the final dance sequence to be executed.

[0019] By employing the aforementioned technical solution, an initial dance sequence is first constructed, and then a multi-dimensional quantitative evaluation is performed using a pre-defined evaluation function to comprehensively detect deficiencies in rhythm, transitions, and emotional expression. Based on the evaluation results, the model weights and choreography rules are dynamically adjusted to make targeted corrections to the initial sequence, forming a closed loop of "generation-evaluation-optimization." This process continuously iterates to improve dance quality, effectively avoiding problems caused by defects in choreography rules or deviations in model parameters, ensuring that the final output dance sequence reaches optimal standards in all dimensions.

[0020] In conjunction with some embodiments of the first aspect, in some embodiments, the step of generating a sequence of robot joint control commands containing the target dance movement and the future target dance movement and sending it to the robot to generate the dance specifically includes: optimizing the trajectory of the initial dance sequence to be executed so that the initial dance sequence to be executed conforms to the dynamic constraints of the target robot; decomposing the optimized initial dance sequence to be executed into the robot's multi-task end trajectory using a zero-space mapping whole-body control method; calculating the corresponding expected joint torque based on the multi-task end trajectory; using the joint torque as a feedforward quantity, and combining it with the proportional-derivative feedback control quantity based on the real-time joint position error and velocity error, to generate the final control command sequence sent to the robot joint actuator.

[0021] By employing the above technical solutions, trajectory optimization ensures that the dance sequence conforms to robot dynamics constraints, avoiding exceeding physical execution limits. Zero-space mapping technology enables the rational allocation of trajectories across multiple task ends, ensuring coordinated and consistent joint movements. Combining desired joint torque feedforward and proportional-derivative feedback control effectively compensates for model errors and external disturbances, improving the accuracy of control commands. This technical process transforms abstract dance sequences into control commands that the robot can stably execute, ensuring that dance movements accurately reproduce the choreography while possessing high stability and safety.

[0022] In a second aspect, this application provides an automatic robot dance generation system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors call the computer instructions to cause the automatic robot dance generation system to perform any of the methods in the first aspect.

[0023] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a robot dance automatic generation system, cause the robot dance automatic generation system to perform any of the methods in the first aspect.

[0024] Fourthly, this application provides a computer program product, including a computer program that, when run on a robot dance automatic generation system, causes the robot dance automatic generation system to perform any of the methods in the first aspect.

[0025] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0026] 1. By employing core technologies such as constructing multi-dimensional music feature maps, generating time-axis-bound dance semantic label sequences, and combining forward-looking window prediction with multi-target real-time determination model to select actions, the technology effectively solves the technical problems of existing technologies that rely on video materials, lack deep structural and emotional dynamic responses to music, and generate mechanical dances with insufficient artistic expression. This enables the automatic generation of robot dances that combine micro-rhythmic fit, macro-structural coherence, and artistic creativity without the need for video materials, significantly improving the efficiency and flexibility of dance generation.

[0027] 2. By employing a multi-objective determination model to select high-scoring initial movements, calculating the transition and connection scores to quantify the naturalness of movement connections, and determining target movements according to the optimal sequence combination, the technical problems of existing technologies, such as partial matching of dance movements but stiff overall connections and lack of complete artistic choreography logic, are effectively solved. This achieves a dance sequence that ensures both a high degree of fit between individual movements and music and a smooth and natural overall connection effect, forming a complete dance choreography effect that conforms to the laws of artistic expression.

[0028] 3. By employing a multi-dimensional quantitative evaluation method for candidate movements, considering factors such as rhythm, energy, timing, transitions, and style, and calculating a comprehensive matching score through weighted summation, this technology effectively solves the technical problems of existing technologies, such as the single dimension of movement evaluation, the difficulty in quantifying subjective choreography logic, and the one-sidedness of movement selection. This enables a comprehensive and objective evaluation of candidate movements, ensuring that the selected movements achieve optimal performance in terms of rhythmic accuracy, energy adaptability, and stylistic consistency, significantly improving the overall integration of dance and music. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating an automatic robot dance generation method based on music feature analysis in an embodiment of this application.

[0030] Figure 2 This is a schematic diagram of music energy analysis in music feature parsing in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of music beat adjustment in music feature analysis in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of BPM change analysis based on music feature parsing in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the dance sequence quality assessment and generation results in an embodiment of this application;

[0034] Figure 6 This is another flowchart illustrating the automatic generation method for robot dance based on music feature analysis in this application embodiment;

[0035] Figure 7 This is a schematic diagram of the physical device structure of an automatic robot dance generation system in the embodiments of this application. Detailed Implementation

[0036] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0037] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0038] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating an automatic robot dance generation method based on music feature analysis in an embodiment of this application.

[0039] S101. After obtaining the audio data associated with the target music, construct a multi-dimensional music feature map, which includes the music's rhythm, energy, spectral features, macroscopic music structure, and music emotional tags.

[0040] This step is executed after the system receives the user's dance generation request and specifies the target music, but before any dance movements are generated. The execution scenario is when the system is in the music analysis and feature extraction stage, needing to extract sufficiently rich features from the original audio to provide a foundation for subsequent dance semantic tag generation and movement matching.

[0041] The system first acquires the audio data associated with the target music through an audio input interface. This audio data can come from local files, network streaming media, or real-time user recordings. The system then performs frame-by-frame processing on the audio data, dividing the continuous audio signal into multiple short time frames. For each frame, acoustic features are extracted, including beat timing, energy profile, and spectral distribution characteristics, forming multiple acoustic feature vector sequences. The beat timing is obtained through a beat tracking algorithm to determine the rhythmic position of the music; the energy profile is obtained by calculating the short-time energy or loudness of each frame to reflect the dynamic changes in the music; and the spectral distribution characteristics are extracted using methods such as Fourier transform to describe the timbre and frequency components of the music.

[0042] Next, the system constructs a self-similarity matrix based on the acoustic feature vector sequence, where each element represents the feature similarity between two different time points. The system analyzes the self-similarity matrix, identifying macroscopic structural sections of the music, such as verses, choruses, and interludes, by searching for repeating structures and significant changes within the matrix.

[0043] Then, the system determines the corresponding audio information within each macroscopic structural segment. This information can be the statistical features of the acoustic feature vector sequence within that segment or encoded high-dimensional vectors. Simultaneously, if the target music contains lyrics, the system performs word segmentation, vectorization, and other processing on the lyrics to extract textual information, including semantic vectors of words and emotional inclinations.

[0044] The system inputs audio and text information into a transformer-based cross-modal attention model. This model comprises an audio encoder, a text encoder, and a cross-modal attention layer, capable of encoding audio and text separately and capturing the correlation between them through an attention mechanism. Specifically, the model can identify situations where the audio sentiment and the text meaning are inconsistent, i.e., semantic conflict features. For example, when the audio has high energy and a cheerful melody, but the lyrics express sadness or negativity, the model can capture this conflict and generate a corresponding semantically reversed sentiment label.

[0045] Finally, the model outputs music emotion tags, which can be discrete emotion categories such as "cheerful," "sad," or "tense," or continuous emotion vectors. For segments with semantically inverted emotions, the model generates corresponding tags so that subsequent steps can consider such complex emotional expressions when generating dance semantics.

[0046] To facilitate understanding, an example will be given below, using "Example Song" as an example:

[0047] Structure recognition: Through chroma map self-similarity matrix analysis, the following are automatically marked: 0:00-0:15 (intro), 0:15-0:45 (verse A), 0:45-1:15 (chorus B), 1:15-1:35 (bridge), etc.

[0048] Multidimensional feature alignment: At the beginning of the chorus B section (0:45), the spectral record is as follows: {Time: 45.0s, BPM: 128, beat intensity: high frequency strong beat, energy level: 0.85 (high), section label: "chorus_B", semantic label: ["climax", "strong rhythm", "high energy burst"]}.

[0049] In this way, the system can simultaneously grasp the overall structure of the music and the fine-grained features at each point in time, providing accurate basic data for subsequent emotion analysis and dance movement generation.

[0050] This step allows the system to extract deep features from both audio and lyrics modalities, and utilize a cross-modal attention model to capture the semantic connections and conflicts between them. This results in more accurate and comprehensive music emotion labels, particularly in recognizing semantically reversed emotions. This step addresses the problem of inaccurate emotion recognition caused by existing methods relying solely on single modality or shallow features. It enables the system to generate dance movements that match the true emotions of the music in subsequent steps, enhancing the artistic expressiveness and emotional depth of the dance.

[0051] S102. Based on the multi-dimensional music feature map, generate a dance semantic label sequence on the time axis;

[0052] The system takes a multi-dimensional music feature map as input and divides the music into blocks along the timeline, such as by beat, measure, or musical section, resulting in several consecutive time segments. For each time segment, the system generates corresponding dance semantic tags based on the segment's rhythm, energy, spectral features, macrostructure, and emotional labels, using rule-based reasoning or machine learning models. For example, in a fast-paced, high-energy chorus, the system might generate semantic tags such as "jumping," "rapid gestures," and "intense body movements"; in a slow-paced, low-energy verse, the system might generate semantic tags such as "gentle gestures," "slow turns," and "lyrical postures."

[0053] The system can utilize a pre-trained mapping model to map multi-dimensional music feature vectors into a probability distribution of dance semantic labels, and then determine the dance semantic label for that time segment based on a preset threshold or the maximum probability principle. For transitional regions between adjacent time segments, the system can also generate transitional dance semantic labels to ensure the continuity and fluency of the semantic sequence. The system concatenates the dance semantic labels of all time segments in chronological order to form a complete dance semantic label sequence, which corresponds one-to-one with the target music on the timeline, providing a semantic description of the action requirements for subsequent action retrieval.

[0054] S103. Access the preset parametric dance motion meta-library, which stores multiple atomic dance motions.

[0055] The system accesses a pre-built parametric dance motion metadata library via an internal interface. This library can be stored on local storage or retrieved from a remote server over a network. Each atomic dance motion in the library is stored in a structured manner, including metadata such as motion identifier, motion name, motion type, duration range, applicable BPM range, energy level, style vector, joint control keyframe sequence, and transition compatibility vector. When accessing the motion library, the system can first load the library into memory and create an index for subsequent fast retrieval based on semantic tags and features. The system can also preprocess the motion library, such as standardizing the joint sequences of the motions to adapt them to the joint structures and workspaces of different robot models. By accessing the parametric dance motion metadata library, the system obtains the basic motion units required to construct dance sequences, providing a resource foundation for subsequent selection of appropriate motions based on music features and semantic tags.

[0056] To facilitate understanding, the following is a specific example of a parameterized dance movement meta-database: This meta-database pre-stores hundreds of reusable atomic dance movements. Taking the atomic movement "Power Jump" as an example, each movement contains three types of structured information: metadata, core data, and related data. Its complete description is as follows:

[0057] Metadata: {BPM range: [115, 145], Energy level: High, Style vector: {Hip Hop: 0.8, Power: 0.9, Explosiveness: 0.9}, Duration: 2.2 seconds};

[0058] Core data: Joint space trajectory keyframes (12 joints, 50 keyframes, stored as a lightweight JSON file);

[0059] Related data: Transition compatibility vector (0.7 compatibility with "Spin" action, 0.2 compatibility with "Slow Wave Hand"), used to quantify the naturalness and smoothness of the transition between this action and other actions.

[0060] The aforementioned metadata is used to quickly filter actions that match music features. The core data is the core basis for driving the robot to perform actions, while the associated data provides a quantitative reference for subsequent evaluation of the quality of transitions between actions. The three types of data work together to ensure both the efficiency of action filtering and the accuracy and smoothness of action execution.

[0061] S104. After obtaining the current dance semantic label sequence and the future dance semantic label sequence within the preset look-ahead window, retrieve the current candidate action and the future candidate action from the parameterized dance action meta-language library.

[0062] The “preset look-ahead window” refers to the length of a future period that the system looks ahead when making an action selection. It is used to represent the range of future contexts that the system considers when making a decision on the current action. It can also be several preset action windows in the future, which is not limited here.

[0063] This step is executed when the system has generated a complete sequence of dance semantic tags and entered the stage of generating dance movements segment by segment, repeating the process cyclically for each current time period. The execution scenario involves the system selecting suitable candidate movements from the movement library based on current and future semantic needs, providing a selection space for subsequent movement decisions.

[0064] The system first determines the current time period on the timeline, which can be a measure, a beat, or a fixed-length time segment. The system extracts the portion corresponding to the current time period from the complete dance semantic tag sequence to obtain the current dance semantic tag sequence. Simultaneously, based on the preset lookahead window length, the system extracts a future time interval following the current time period and extracts the corresponding portion from the dance semantic tag sequence to obtain the future dance semantic tag sequence. The system uses the current and future dance semantic tag sequences as search criteria to perform a search in a parameterized dance movement meta-lexicon. The search process can be based on semantic matching, feature vector similarity calculation, or rule matching. For example, the style vector and semantic tags of the movement are compared with the current and future semantic tags, selecting movements with higher similarity as candidate movements. The system can retrieve a set of current candidate movements for the current time period and a set of future candidate movements for the future time period. Each set of candidate movements can contain multiple movements to maintain a certain selection space. The system can also perform preliminary filtering of candidate movements based on attributes such as the applicable BPM range and energy level, eliminating movements that are clearly unsuitable for the current music characteristics.

[0065] This step enables the system to perform action retrieval by combining current and future semantic needs. It considers not only the matching at the current moment but also the connection and development of future actions, thereby avoiding action selection that is locally optimal but inconsistent overall. This step solves the problem in existing methods that rely solely on local probability to splice actions and lack global planning, resulting in a lack of coherence and artistic structure in dance, and improves the overall quality of dance action sequences.

[0066] S105. Determine the target dance action from the current candidate action and the future target dance action from the future candidate action using a multi-objective real-time determination model;

[0067] This step is executed after the system has retrieved the current and future candidate actions, but before generating joint control commands. The execution scenario is when the system is in the action decision-making phase, needing to select from multiple candidate actions to simultaneously satisfy multiple objectives such as music matching, action continuity, and stylistic consistency.

[0068] The specific steps will be described in detail in subsequent sections S201-S204, and will not be repeated here.

[0069] This step enables the system to globally optimize and select from multiple candidate movements, ensuring not only the match between the current movement and the music but also considering the connection of future movements and the overall coherence of the dance. This results in a more natural, fluid, and artistically expressive sequence of dance movements. This step addresses the problems of lacking global planning in movement selection and mechanically stacking dance movements in existing methods, thus improving the overall quality and artistic effect of the dance.

[0070] S106. Generate a sequence of robot joint control instructions containing the target dance movement and the future target dance movement, and send it to the robot to generate the dance.

[0071] This step is executed after the system has determined the target dance moves and future target dance moves, and when the robot needs to actually perform the dance moves. The execution scenario is when the system is in the action execution phase, and it needs to transform the abstract action description into low-level control instructions that the robot can execute, and control the robot to complete the dance performance.

[0072] The system first obtains the raw joint trajectory data of the movement based on the joint control keyframe sequences corresponding to the target and future target dance movements in the parametric dance movement meta-language library. This data typically includes the target angles or positions of each joint at different time points. The system performs time alignment and interpolation on the raw joint trajectories to ensure they match the beat and tempo of the current music and guarantee smooth transitions. If necessary, the system can also perform dynamic optimization on the joint trajectories, such as adjusting the speed and acceleration of the movements, to meet the robot's physical constraints, such as joint limits, torque limitations, and collision avoidance. The system converts the processed joint trajectories into a sequence of joint control commands recognizable by the robot controller. Each command contains information such as the target joint position, speed, or torque at the corresponding time point. The system sends the joint control command sequence to the robot's main control unit or joint actuator via a communication interface (such as serial port, Ethernet, or wireless communication). Upon receiving the commands, the robot executes each joint control command sequentially according to the time sequence, driving the movement of each joint to complete the continuous execution of the target and future target dance movements, ultimately presenting a dance performance that matches the target music. During the dance performance, the system can also adjust the instructions in real time as needed to adapt to the robot's actual state or changes in the external environment.

[0073] In some embodiments, when a certain number of target dance movements and future target dance movements have been identified, and it is necessary to optimize the overall composition of these movements, the system can improve the overall quality and artistic expression of the dance through overall evaluation and feedback adjustments.

[0074] Specifically, the system first arranges multiple target dance movements determined in the current and historical time periods, along with future target dance movements determined by the look-ahead mechanism, in chronological order to form a continuous sequence of movements. During the choreography process, the system performs preliminary processing on the connection points between adjacent movements, such as adjusting the duration of the movements to match the music beat, or inserting simple transitional movements to ensure basic continuity, thereby constructing the initial dance sequence to be executed.

[0075] Subsequently, the system invokes a pre-defined dance evaluation function to perform a multi-dimensional quantitative evaluation of the initial dance sequence to be executed. The evaluation function analyzes the quality of the dance sequence from multiple perspectives: in terms of rhythm, it assesses the alignment of movements with the musical beat and accents; in terms of fluidity, it analyzes the smoothness of joint angle changes and the continuity of movement trajectories between movements; in terms of stylistic consistency, it checks whether the overall movement style of the sequence aligns with the emotional tone and macrostructure of the music; in terms of movement richness, it counts the number of different movement types and the frequency of movement changes in the sequence to avoid overly monotonous movements; and in terms of energy matching, it assesses whether the energy changes in the dance sequence match the energy profile of the music. The evaluation function calculates the values ​​for each of these dimensions to obtain multi-dimensional evaluation index results.

[0076] Based on the results of multi-dimensional evaluation indicators, the system adjusts parameters and rules accordingly. For example, assuming the initial dance sequence to be executed covers all macro-structural segments of the target music, including the intro, verse, chorus, and bridge, the core output of the evaluation function is: "Music Structure Coverage" 98% (meets the high score standard of "Structural Integrity"), "Average Deviation of Bridge Energy" 18% (above the preset threshold of 10%, which is the main deduction item), and a total score of 88 points for the dance sequence. System analysis shows that the deductions stem from insufficient matching between the dance movement energy and music energy in the bridge segment—although the energy of the music bridge segment is lower than that of the chorus, it has slight fluctuations, while the energy of the bridge segment movements in the initial sequence is too flat and does not match this change. Based on this, the system automatically triggers a weight adjustment mechanism: the weight (W2) of the "energy alignment" corresponding to the "bridge" segment is temporarily increased by 5% in the rules of the current choreography task (for example, from the original weight of 0.3 to 0.315), so that when the subsequent movement is corrected, the model will focus more on matching the energy changes of the bridge segment; at the same time, the system records the adjustment strategy of "the bridge segment needs to increase the energy alignment weight by 5%" in the experience base. When processing music with similar structures in the future, this experience can be directly reused without repeated trial and error.

[0077] Finally, the system uses the adjusted multi-objective real-time determination model to optimize and correct the initial dance sequence to be executed, resulting in an optimized final dance sequence to be executed. This sequence achieves a high level in terms of rhythm, fluency, stylistic unity, and richness of movements, and can better match the structure and emotion of the music, presenting a more artistically expressive dance effect.

[0078] Through this step, the system achieves closed-loop optimization of dance sequences, continuously improving the overall quality of the dance through a process of "choreography-evaluation-feedback-correction." This step solves the problem of unstable dance segment quality caused by relying solely on local motion selection and lacking overall optimization in existing methods, thereby enabling the automatic generation of robot dances that can adaptively adjust and have high overall quality.

[0079] In the above embodiments, by comprehensively extracting multi-dimensional deep features of music to construct a feature map, accurate analysis of music rhythm, emotion, and macro structure is achieved. Then, the music features are mapped to dance semantics. Combined with a pre-set parametric action meta-language library, the cost of action acquisition is reduced. The look-ahead window mechanism takes into account the global planning of current and future actions. Finally, the multi-objective model is used to achieve comprehensive optimization of action matching, connection, and style. Therefore, it can get rid of the dependence on video material library and preset rule library, break through the limitations of local action splicing, and effectively solve the problems of insufficient dynamic response to deep emotions and overall structure of music and lack of artistic expression and coherence in the generated dance. Thus, it achieves a deep fit between dance movements and music in terms of rhythm, emotion, and structure, and automatically generates robot dance choreography that is both smooth and artistically appealing.

[0080] Please refer to the following: Figure 2 This is a schematic diagram of music energy analysis in music feature parsing in an embodiment of this application.

[0081] exist Figure 2 The image above is a visualization of the music energy analysis results, fully presenting the energy-related features obtained after multi-dimensional feature analysis of the target music:

[0082] Musical Energy Distribution: The bar chart above uses the time axis (0 seconds to 45 seconds) as the horizontal axis and the energy value (0.00 to 0.35) as the vertical axis, marking low-energy, high-energy, and medium-energy sections with green, red, and orange colors, respectively. It can be seen that the music has a gentle low-energy start from 0-10 seconds, then enters a core section dominated by high energy interspersed with medium energy after 10 seconds, reaching an energy peak at 30-35 seconds, and then falling back to the medium-energy range after 40 seconds, visually reflecting the dynamic change in musical energy over time.

[0083] Distribution of high-energy time points: The chart below uses time as the horizontal axis and energy value as the vertical axis, marking all high-energy time points with red bars, showing the density of high-energy events in time, and providing accurate time reference for the design of subsequent dance moves.

[0084] Key quantitative indicators: Four core quantitative results are further presented below the chart:

[0085] The rhythmic stability is 51.4%, indicating that the rhythmic fluctuation of the music is moderate, and attention should be paid to the accuracy of the timing synchronization in subsequent motion generation.

[0086] The average musical energy value is 0.206, which is above average.

[0087] The music style was identified as rock, providing a clear style guide for subsequent action style matching;

[0088] The number of high-energy points is 1163, reflecting the density of high-energy events in the music, which can be used to guide the design of the proportion of high-energy dance movements.

[0089] This diagram allows the system to fully grasp the energy dynamics, rhythmic characteristics, and stylistic attributes of the target music, providing crucial feature data for generating dance movement sequences that closely match the music.

[0090] Please refer to the following: Figure 3 This is a schematic diagram of music beat adjustment in music feature analysis in an embodiment of this application.

[0091] exist Figure 3 The chart presents a visualization of the adjusted music beat analysis. It uses the time axis (0 to 175 seconds) as the horizontal axis and the beat intensity (0.0 to 1.0) as the vertical axis, marking the distribution of strong, medium, and weak beats with red, orange, and green colors, respectively. The chart clearly shows that medium (orange) and weak (green) beats are more densely distributed in segments such as 0-25 seconds, 50-100 seconds, and 125-150 seconds. Strong beats (red) only show two significant peaks at approximately 80 and 100 seconds, intuitively reflecting the changes in the strength and temporal distribution of the music beat.

[0092] This visualization provides a precise timing reference for the design of timing points in subsequent dance movements. It can guide the system to arrange stronger movements at strong timing points and match moderate or gentle movements at medium and weak timing points, thereby improving the synchronization accuracy between dance and music rhythm.

[0093] Please refer to the following: Figure 4 This is a schematic diagram of BPM change analysis based on music feature analysis in an embodiment of this application.

[0094] exist Figure 4 The chart in the image visualizes the BPM variation analysis, with the time axis (0 to 250 seconds) on the horizontal axis and the BPM value (125 to 155) on the vertical axis. The blue curve represents real-time BPM changes, and the red dashed line represents the average BPM (approximately 129.8). As can be seen from the chart, the music exhibits two distinct BPM peaks at approximately 50 and 150 seconds, reaching a high of nearly 155. During the remaining time, the BPM fluctuates slightly around the average, clearly demonstrating the dynamic tempo changes of the target music throughout its duration.

[0095] This analysis helps the system dynamically adjust the speed and rhythm of subsequent movements, matching faster movement rates during peak BPM periods and maintaining a constant speed during stable periods, thereby improving the adaptability of dance to music speed changes and avoiding a disconnect between movement and rhythm due to BPM fluctuations.

[0096] Please refer to the following: Figure 5 This is a schematic diagram of the dance sequence quality assessment and generation results in an embodiment of this application.

[0097] exist Figure 5 In the middle, this interface displays the results of the system's multi-dimensional quantitative evaluation of the generated dance sequence and the detailed choreography:

[0098] Dance sequence quality assessment: The top gives an overall score of 74.2 points, along with the following tips: "The dance sequence is good, it is recommended to preview it before use" and "The music rhythm stability is low, which may affect the synchronization effect." These tips directly reflect the overall quality and potential problems of the current dance sequence.

[0099] The generated dance sequence clearly displays the choreography of three segments along the timeline (0.0s-45.0s):

[0100] 0.0s-15.0s: Perform the "Body Rhythm" movement, labeled as medium energy, BPM 85-115, and groove style;

[0101] 15.0s-30.0s: Perform a "smooth turn" action, labeled as medium energy, BPM 90-120, urban style;

[0102] 30.0s-45.0s: Perform the "gesture wave" action, labeled as low energy, BPM 80-110, wave style.

[0103] Key quantitative indicators: The bottom statistics show the key indicators: the total dance duration is 45.0s, containing 3 dance moves, with 3 different styles, and the music coverage reaches 100.0%, indicating that the movement sequence has completely covered the entire duration of the target music.

[0104] This diagram enables the system to achieve interpretability assessment and visualization of dance sequences, providing a clear basis for subsequent adjustments to the weights of the multi-objective model and corrections to dance movements based on the assessment results.

[0105] Following the above embodiments, the method provided in this embodiment will now be described in more detail. Please refer to [link / reference]. Figure 6 This is another flowchart illustrating the automatic generation method of robot dance based on music feature analysis in this application.

[0106] S201. Calculate and determine the first comprehensive matching score for each current candidate action using a multi-objective determination model, and determine the top n actions with the highest scores as the initial target dance actions.

[0107] Among them, the "multi-objective determination model" refers to an algorithm model that can simultaneously take into account multiple optimization objectives (such as beat matching, energy alignment, style consistency, etc.) and comprehensively evaluate and make decisions on candidate actions, so as to achieve multi-dimensional optimal action selection.

[0108] This step is executed after the system has retrieved the current candidate movements, but before the initial screening of future target dance movements and optimization of movement combinations. The execution scenario is when the system is in the initial screening stage of movement decision-making, requiring the selection of high-quality movements from a large number of current candidate movements to narrow down the subsequent optimization scope, balancing decision accuracy and real-time performance.

[0109] The system first calls a multi-objective determination model, inputting the metadata of each current candidate action (including BPM range, energy level, style vector, transition compatibility vector, etc.) along with the multi-dimensional music features (rhythm, energy, emotional tags, etc.) corresponding to the current time period and the determined information of the previous action into the model. The model calculates the beat fit (the degree of matching between the action's BPM range and the current music BPM), energy alignment (the degree of matching between the action's energy level and the current music energy), timing synchronization accuracy (the alignment error between the action's keyframes and the music's timing timestamps), transition smoothness (the naturalness of the connection with the previous action), and style consistency (the degree of matching between the action's style and the current music's emotional and section styles) for each current candidate action according to preset weights. Then, the scores of each dimension are weighted and summed to obtain the first comprehensive matching score for each current candidate action.

[0110] To more clearly illustrate the decision-making process of the multi-objective determination model, the following example is given in a specific scenario: Suppose that the system has completed the multi-dimensional feature analysis of the target music "Example Song" and identified the features of the starting point of section B of the chorus (0:45) as follows: {Time: 45.0s, BPM: 128, beat intensity: high frequency strong beat, energy level: 0.85 (high), section label: "chorus_B", semantic label: ["climax", "strong rhythm", "high energy burst"]}, and the previous determined action is "body rhythm" (medium energy, rhythm style).

[0111] Candidate retrieval: Based on the semantic tags of "high energy", "BPM~128" and "hiphop style", the system retrieves current candidate movements that meet the conditions such as "power jump" and "fast spin" from the knowledge graph of the parametric dance movement meta-language database;

[0112] Multi-objective scoring: Taking "strength jump" as an example, the model calculates the first comprehensive matching score according to preset dimensions and weights:

[0113] Beat fit (weight W1=0.3): The BPM range of this move [115, 145] perfectly matches the current music BPM of 128, with a score of 1.0;

[0114] Energy Alignment (Weight W2=0.3): The motion energy level is "High", which is highly matched with the current music energy level of 0.85, with a score of 0.9;

[0115] Synchronization accuracy (weight W3=0.2): The action start keyframe is precisely aligned with the strong timing at 0:45 seconds, with an alignment error close to 0, and a score of 0.95;

[0116] Transition smoothness (weight W4=0.15): Based on the pre-stored transition compatibility vector in the meta-language database, the compatibility between "power jump" and the previous action "body rhythm" is 0.4, with a score of 0.4;

[0117] Style consistency (weight W5=0.05): The "power" and "explosiveness" dimensions in the action style vector match the semantic label of "climax" in the B section of the chorus, with a score of 0.1;

[0118] Weighted total score: 0.3×1.0+0.3×0.9+0.2×0.95+0.15×0.4+0.05×0.1=0.855;

[0119] Forward-looking decision-making: By setting a forward-looking window, such as predicting that the next 15 seconds (0:45-0:60) will still be a high-energy chorus section, the system analyzes that the ending posture of the "power jump" (limb extension, energy maintained at a high level) is more conducive to connecting with subsequent high-energy movements such as "rapid rotation", which can form a coherent climax dance segment. Therefore, it is given priority to be included in the high-score candidates.

[0120] After calculation, the system sorts all current candidate actions by their first comprehensive matching scores in descending order and selects the top n actions as the initial target dance actions. The value of n needs to balance decision efficiency and optimization space. If n is too large, it will increase the computational load of subsequent action combination evaluations, affecting real-time performance; if n is too small, it may miss the optimal action, reducing dance quality. Typically, it can be dynamically adjusted based on the size of the action library and the real-time requirements of robot dance generation.

[0121] Through this step, the system achieves preliminary and accurate screening of current candidate movements, eliminating movements that do not match the current music features and semantic requirements. This not only narrows down the scope of subsequent movement combination optimization and ensures real-time decision-making, but also preserves the selection space for high-quality movements, laying the foundation for generating high-quality dance movement sequences in the future.

[0122] In some embodiments, the system evaluates actions before the multi-objective real-time determination model performs the action evaluation, and after determining that the current music emotion label is semantically inverted. The system needs to process music segments with complex emotional expressions, and by adjusting the action selection strategy, it uses the contrast of dance movements to enhance the deep semantic expression of the music, avoiding the dance from mechanically following the surface rhythm of the music and ignoring the true emotion of the lyrics.

[0123] Specifically, in a conventional action decision-making process, multi-objective real-time determination models typically consider or prioritize beat fit and energy alignment to ensure that dance movements remain synchronized with the rhythm and intensity of the music. However, when the system detects that the current music sentiment label is semantically inverted, continuing to use conventional weights may result in dance movements that only match the surface-level cheerfulness or sadness of the music, failing to express the opposite emotion implied in the lyrics, leading to a lack of artistic expression.

[0124] Therefore, the system triggers the adjustment mechanism in this step. First, the system identifies the current state of the music's emotional label and confirms the existence of semantic inversion (for example, audio features show high energy and fast tempo, implying "cheerful," but lyric text analysis shows the emotion as "sad"). Subsequently, the system automatically adjusts the weight configuration of the multi-objective real-time determination model, significantly increasing the weight coefficient of the "style consistency" dimension. This means that when calculating the comprehensive matching score of actions, the degree of matching between action style and deep semantic emotion (such as sadness) will dominate, exceeding the matching requirements for surface rhythm or energy.

[0125] Under the adjusted weighting system, when retrieving and evaluating candidate actions, the model will prioritize those actions that embody deep emotions in style and intensity. For example, for a segment with "upbeat melody + sad lyrics," the system will reduce its preference for "high-energy" actions such as large jumps and rapid waving, and instead prioritize slow, graceful, low-hanging postures or lighter, more subtle "sad style" actions. This selection creates a clear visual contrast between the dance movements and the upbeat melody of the music. Through this contrast, the robot's dance performance is no longer merely a simple reflection of the music, but rather, through the tension of body language, it externalizes the complex emotional conflicts in the music that are implied rather than explicitly stated. This mode of expression guides the audience to focus on the deeper meaning of the music, enhancing the narrative and artistic appeal of the dance.

[0126] Through this step, the system achieves a refined understanding and dynamic response to musical emotions, solving the problems of emotional expression deviation and superficial dance movements that are easily generated when existing methods process semantically inverted music. This enables the creation of robot dance choreography that can accurately convey the deep emotions of music and has high artistic expression.

[0127] S202. Calculate and determine the second comprehensive matching score for each future target dance movement using a multi-objective determination model, and determine the top n movements with the highest scores as the initial future target dance movements.

[0128] The system adopts the multi-objective determination model from S201, maintaining the evaluation dimensions and weighting system unchanged. Only the evaluation criteria are adjusted to include future music features (including BPM, energy change trends, emotional tags, macro-structural segment attributes, etc.) and future dance semantic tag sequences within a preset look-ahead window. For each future candidate movement, the model calculates its beat fit, energy alignment, and timing accuracy with the future music features, as well as its style consistency with the future segment style (transition smoothness is not calculated at this stage because the preceding movements are not yet determined; this will be considered in subsequent combination evaluations). A second comprehensive matching score for each future candidate movement is then obtained through weighted summation.

[0129] Subsequently, the system sorts all future candidate movements in descending order of their second comprehensive matching scores, and selects the top n movements as the initial future target dance movements. Here, the value of n is consistent with S201 to ensure that the number of initial target dance movements matches the number of initial future target dance movements. This facilitates the subsequent formation of one-to-one, one-to-many, or many-to-many movement combinations for transitional evaluation, avoiding incomplete combination evaluation due to differences in the number of movements.

[0130] Through this step, the system simultaneously completes the initial screening of future candidate movements, forming a corresponding candidate set with the initial target dance movements, providing a high-quality and quantitatively matched movement foundation for subsequent evaluation of the smoothness of the transition between the two.

[0131] S203. Determine the transition score from each initial target dance movement to the initial future target dance movement. The transition score is used to characterize the naturalness and smoothness of the transition from the current candidate movement to the future candidate movement.

[0132] This step is performed when the system is in the motion combination optimization stage. It is necessary to evaluate the connection quality between the initial target dance motion and the initial future target dance motion of all possible combinations, so as to provide a basis for selecting the optimal motion combination.

[0133] The system first constructs a full combination matrix of initial target dance movements and initial future target dance movements. Each element in the matrix corresponds to a combination of "initial target dance movement - initial future target dance movement". For each combination, the system extracts metadata of the two movements from the parametric dance movement metadata library. Focusing on the joint control keyframe sequence and transition compatibility vector of the movement, the system calculates the transition score from three dimensions: First, posture compatibility, which calculates the difference between the joint angles at the end of the initial target dance movement keyframe and the start of the initial future target dance movement keyframe. The smaller the difference, the more natural the posture transition, and the higher the corresponding score. Second, motion continuity, which analyzes the motion trajectory, speed, and acceleration change trends of the two movements. If the motion trend of the former movement is consistent with that of the latter movement (e.g., both are rotations, both are translations), and the speed change is smooth without abrupt changes, the score is higher. Third, style consistency, which combines the style vectors of the two movements to evaluate whether the movement style is consistent with the emotional label of the music in the current and future time periods. If the style is consistent and matches the musical emotion, additional points can be awarded.

[0134] The system pre-defines the weights of each evaluation dimension and performs a weighted sum of the scores from the three dimensions to obtain the transition score for each action combination. Simultaneously, if the transition compatibility vectors for two actions are pre-stored in the action meta-language library, the system can directly use the values ​​from these vectors as the base score, and then fine-tune them based on posture compatibility and motion continuity to further improve scoring accuracy and efficiency. For example, if the preset value of the transition compatibility vector for two actions is 0.8 (out of 1.0), and the differences in posture joint angles are small and the movement trends are consistent, the transition score can be adjusted to 0.85; if the posture differences are large, it can be adjusted to 0.7.

[0135] This step quantifies the connection quality between the initial target dance movements and the initial future target dance movements, providing a core evaluation basis for subsequent selection of the optimal movement combination. This step solves the problem of existing methods that rely solely on local probability to splice movements and lack quantitative evaluation of the smoothness of transitions between movements, resulting in stiff and disjointed dances. It ensures the generation of natural and fluid dance movement sequences.

[0136] S204. Determine the target dance movement and the future target dance movement based on the sequence of movements with the highest transition score.

[0137] The system first sorts the transition scores of all calculated combinations of "initial target dance movement - initial future target dance movement" and selects the combination with the highest transition score. If multiple combinations have the same transition score (i.e. tied for first place), the system further compares the sum of the first and second comprehensive matching scores of each combination and selects the combination with the highest total score as the optimal combination. If the total scores are still the same, the system prioritizes the combination with a higher degree of fit between the style vector and the current and future music emotional tags to ensure the comprehensive adaptability of the combination.

[0138] After determining the optimal action combination, the system identifies the initial target dance action in the combination as the target dance action for the current time period, and the initial future target dance action in the combination as the future target dance action for future time periods. At the same time, the system stores the action combination in the determined action sequence library, updates the action decision state, and provides context for action retrieval, filtering, and combination optimization in the next time period (i.e., the "previous action" in the next time period is the target dance action determined in this instance).

[0139] Furthermore, if the preset look-ahead window contains multiple future time periods, the system can repeatedly execute steps S201-S204 to determine the future target dance movements for each time period, ensuring the coherence and smoothness of the entire dance sequence. For example, if the look-ahead window covers two future time periods, the future target dance movements for the first future time period are determined first. Then, based on these movements, candidate movements for the second future time period are retrieved, and the process of filtering and combining these movements is repeated until the movements for all time periods within the look-ahead window are determined.

[0140] Through this step, the system achieves the global optimal combination selection of current and future actions, which not only ensures the fit between individual actions and the corresponding time period's music features and semantic requirements, but also ensures the smooth transition between actions, avoiding the problem of local optimization but overall incoordination.

[0141] In this embodiment, by screening the initial target action step by step and introducing a look-ahead mechanism to evaluate the transition quality between actions, the action decision-making process is transformed from local optimum to global optimum. Therefore, while ensuring a high degree of matching between the action and music features, the natural fluency and overall coherence of the action sequence are significantly improved. This effectively solves the problems of lack of global planning in action selection, mechanical stacking of dance movements, and stiff transitions in existing methods, thereby realizing the automatic generation of robot dance movements that have both musical compatibility and artistic expression.

[0142] In some embodiments, the generation dimensions of the multi-dimensional music feature map can be expanded to include music phrase recognition functionality, which includes repeated phrase detection and symmetrical phrase detection.

[0143] Repeated phrase detection: By calculating the autocorrelation function of the MFCC feature of the music, the repeated phrases in the music that meet the preset threshold of feature similarity are located and marked with a repeating identifier R(t), where R(t)=1 indicates that the current time period is a repeated phrase and R(t)=0 indicates that the current time period is a non-repeated phrase;

[0144] Symmetrical phrase detection: Analyze the mirror symmetry of musical rhythm sequences, identify symmetrical segments with reversed beat patterns, and mark them with a symmetry identifier S(t), where S(t)=1 indicates that the current segment is a symmetrical phrase, and S(t)=0 indicates that the current segment is an asymmetrical phrase.

[0145] Specifically, the system extracts basic features such as beat and energy while simultaneously executing a musical phrase recognition algorithm. For repeated phrase detection, the system calculates the autocorrelation function of the music's MFCC features—MFCC features accurately represent the melody and timbre characteristics of music. By analyzing the similarity of MFCC features within different time windows, when the similarity exceeds a preset threshold (e.g., 0.85), it is determined to be a repeated phrase, and R(t)=1 is marked. For symmetrical phrase detection, the system decomposes the rhythmic sequence of the music (e.g., drumbeat distribution, beat strength patterns), and by comparing the mirror consistency of the rhythmic sequences before and after (e.g., the rhythmic patterns of the first 8 measures and the last 8 measures are reversed), it identifies symmetrical segments and marks them S(t=1). Both types of marking are precisely bound to the timeline and integrated into the multi-dimensional music feature map.

[0146] Through this step, the system breaks through the limitations of traditional methods that rely solely on superficial features such as rhythm and energy, and delves deeper into the sentence structure logic of music.

[0147] Based on the above embodiments, in some embodiments, programmable choreography rule injection and execution are also included. The choreography rules include a repetitive musical phrase movement matching rule R1 and a symmetrical musical phrase movement matching rule R2, and the rule priority is preset to R1 being higher than R2.

[0148] Rule R1 (Repeated musical phrase action matching): If the repetition identifier R(t) = 1 in the current time period, extract the dance action sequence corresponding to the original musical phrase from the historical action sequence, directly reuse the action sequence as the target dance action in the current time period, and skip the regular calculation of the first comprehensive matching score and the second comprehensive matching score.

[0149] Rule R2 (Symmetrical Phrase Action Matching): If the symmetry identifier S(t) = 1 in the current time period, a new symmetric matching score Csym is added to the multi-objective real-time determination model. Csym = S(t) × Sym(A), where Sym(A) is the symmetry parameter of the current candidate action A (Sym(A) ∈ [0, 1], 0 represents complete asymmetry, and 1 represents complete symmetry). The symmetric matching score Csym is included in the comprehensive matching score calculation. A new weight W6 is added to weight Csym. The calculation formula of the first comprehensive matching score is updated as follows: First comprehensive matching score = W1 × beat fit + W2 × energy alignment + W3 × timing accuracy + W4 × transition smoothness + W5 × style consistency + W6 × Csym, where W1 + W2 + W3 + W4 + W5 + W6 = 1.

[0150] Specifically, the system pre-sets rule R1 to have higher priority than R2. It first detects the repetition identifier R(t) for the current time period: if R(t) = 1, it means the current musical phrase repeats a historical musical phrase. The system directly extracts the corresponding movement combination from the established historical movement sequence, without needing to repeatedly calculate the comprehensive matching score, ensuring that repeated musical phrases correspond to repeated movements, conforming to the audience's aesthetic perception habits. If R(t) = 0, it then detects the symmetry identifier S(t): if S(t) = 1, the system calls the symmetry parameter Sym(A) of each current candidate movement (this parameter is pre-stored in the movement metadata of the parameterized dance movement metadata library) to calculate the symmetry matching score Csym. Subsequently, the comprehensive matching score calculation formula is updated, adding a weight W6 (which can be set to 0.1-0.15), incorporating Csym into the weighted summation, so that highly symmetrical movements receive higher comprehensive scores and are selected first.

[0151] This step transforms the "repetition to repetition, symmetry to symmetry" logic of professional choreography into executable algorithmic rules, solving the problems of existing methods that lack aesthetic logic and mechanically pile up movements. Rule R1 ensures that the repetitive structure of the dance and music echoes each other, enhancing memorability; Rule R2 improves the visual coordination of the dance by matching symmetrical movements with symmetrical musical phrases. The combination of these two rules endows the generated dance with human-like choreographic intelligence, achieving semantic alignment between music and dance and significantly enhancing artistic expression.

[0152] Based on the above embodiments, in some embodiments, users can customize the configuration according to choreography needs. The priority of the programmable choreography rules is configurable. Rule R1 can be set as a hard constraint (forcing action reuse only when the condition of repeated musical phrases is met), and rule R2 can be set as a soft reward (optimizing action selection only through bonus points). Furthermore, the choreography rules support modular expansion, allowing the addition of action matching rules corresponding to musical phrases. When adding a new rule (such as "compare musical phrases corresponding to comparative actions"), only the corresponding music identifier detection method and scoring calculation logic need to be defined; there is no need to modify the core architecture of the multi-objective real-time determination model (such as encoders, attention mechanisms, etc.), allowing for seamless integration. Through this step, the system solves the problems of existing choreography rules being fixed and rigid, unable to adapt to diverse needs, and having high expansion costs. The configurability of the rules allows the system to adapt to different user preferences, improving its flexibility.

[0153] The automatic robot dance generation system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 7 This is a schematic diagram of the physical device structure of an automatic robot dance generation system in this application embodiment.

[0154] It should be noted that, Figure 7 The structure of the robot dance automatic generation system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0155] like Figure 7 As shown, the robot dance automatic generation system includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 702 or programs loaded from storage section 708 into Random Access Memory (RAM) 703, such as performing the methods described in the above embodiments. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0156] The following components are connected to I / O interface 705: input section 706 including audio input devices, push-button switches, etc.; output section 707 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 708 including a hard disk, etc.; and communication section 709 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 709 performs communication processing via a network such as the Internet. Drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 710 as needed so that computer programs read from them can be installed into storage section 708 as needed.

[0157] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the various functions defined in the present invention.

[0158] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0160] Specifically, the robot dance automatic generation system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the robot dance automatic generation method based on music feature analysis provided in the above embodiment.

[0161] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the robot dance automatic generation system described in the above embodiments; or it may exist independently and not assembled into the robot dance automatic generation system. The storage medium carries one or more computer programs, which, when executed by a processor of the robot dance automatic generation system, cause the robot dance automatic generation system to implement the robot dance automatic generation method based on music feature analysis provided in the above embodiments.

[0162] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0163] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for automatically generating robot dance based on music feature analysis, characterized in that, The method includes: After acquiring the audio data associated with the target music, a multi-dimensional music feature map is constructed. The multi-dimensional music feature map includes the music's rhythm, energy, spectral features, macroscopic music structure, and music emotional tags. Based on the multi-dimensional music feature map, a dance semantic tag sequence is generated on the time axis; Access a pre-defined parametric dance motion meta-language library, which stores multiple atomic dance motions; After obtaining the current dance semantic tag sequence for the current time period and the future dance semantic tag sequence within the preset look-ahead window, the current candidate action and the future candidate action are retrieved from the parameterized dance action meta-language library. The multi-objective real-time determination model determines the target dance action from the current candidate actions and the future target dance action from the future candidate actions. Generate a sequence of robot joint control commands that includes the target dance movement and the future target dance movement, and send it to the robot to generate the dance.

2. The method according to claim 1, characterized in that, The atomic dance movements in the parametric dance movement metadata library are organized in the form of a knowledge graph. Each atomic dance movement includes machine-understandable metadata, which includes BPM range, energy level, style vector, joint control keyframe sequence, and transition compatibility vector with adjacent movements.

3. The method according to claim 1, characterized in that, After obtaining the audio data associated with the target music, the steps for constructing a multi-dimensional music feature map include: Extract multiple acoustic feature vector sequences from the audio data, wherein the acoustic feature vector sequences include at least beat timing, energy profile, and spectral distribution features; Self-similarity matrix analysis is performed based on the acoustic feature vector sequence to determine the macroscopic structural segments of the music; Determine the audio information corresponding to the acoustic feature vector sequence within the macroscopic structural paragraph and the text information extracted from the lyrics text corresponding to the target music; The audio information and the text information are input into a transformer-based cross-modal attention model to generate the music sentiment label. The cross-modal attention model is used to capture semantic conflict features between audio information and text information in order to identify semantically reversed sentiment.

4. The method according to claim 3, characterized in that, The steps of determining the target dance move from the current candidate moves and determining the future target dance move from the future candidate moves using a multi-objective real-time determination model specifically include: A multi-objective determination model is used to calculate and determine a first comprehensive matching score for each current candidate action, and the top n actions with the highest scores are determined as the initial target dance actions; A second comprehensive matching score is calculated for each of the future target dance movements using a multi-objective determination model, and the top n movements with the highest scores are determined as the initial future target dance movements. Determine the transition link score from each initial target dance move to an initial future target dance move, the transition link score being used to characterize the naturalness and smoothness of the transition from the current candidate move to the future candidate move; The target dance move and the future target dance move are determined based on the sequence of movements with the highest transition score.

5. The method according to claim 4, characterized in that, The step of calculating and determining a first comprehensive matching score for each current candidate action using a multi-objective determination model further includes: The beat fit is determined based on the matching degree between the preset BPM range corresponding to the current candidate action and the current BPM of the music. The energy alignment is determined based on the degree of matching between the energy level required by the current candidate action and the current energy level of the music. The timing synchronization accuracy is determined based on the alignment error between the keyframe of the current candidate action and the music timing timestamp. Based on the predefined transition compatibility vector in the parametric dance motion meta-language library, the naturalness of the connection between the current candidate motion and the determined previous motion is calculated, and the smoothness of the transition is determined according to the naturalness of the connection. The style consistency is determined based on the matching degree between the style vector of the current candidate action and the overall style and musical emotion label of the current dance segment; The first comprehensive matching score is determined by weighted summation based on at least one of the following objectives: beat fit, energy alignment, timing synchronization accuracy, transition smoothness, and style consistency.

6. The method according to claim 5, characterized in that, The method further includes: When the music emotion label is semantic reversal emotion, the weight coefficient of the multi-objective real-time determination model is adjusted to increase the style consistency, so as to give priority to dance movements that contrast with the surface emotion of the music in style and intensity to express the meaning of semantic reversal.

7. The method according to claim 6, characterized in that, Before the step of generating a sequence of robot joint control commands containing the target dance movement and the future target dance movement and sending it to the robot to generate the dance, the method further includes: Multiple target dance moves are combined with the future target dance moves to create an initial dance sequence to be executed. The initial dance sequence to be executed is quantitatively evaluated in multiple dimensions using a preset dance evaluation function to obtain multi-dimensional evaluation index results. Based on the results of the multi-dimensional evaluation indicators, adjust the weight parameters and choreography rules of the multi-objective real-time determination model; The initial dance sequence to be executed is corrected using the adjusted multi-objective real-time determination model to obtain the final dance sequence to be executed.

8. The method according to claim 7, characterized in that, The step of generating a sequence of robot joint control commands containing the target dance movement and the future target dance movement, and sending it to the robot to generate the dance, specifically includes: The initial dance sequence to be executed is optimized to ensure that it conforms to the dynamic constraints of the target robot. The optimized initial dance sequence to be executed is decomposed into the robot's multi-task trajectory by using the zero-space mapping whole-body control method; Calculate the corresponding expected joint torque based on the multi-task trajectory; The joint torque is used as a feedforward quantity, and combined with the proportional-derivative feedback control quantity based on the real-time joint position error and velocity error, a final control command sequence is generated and sent to the robot joint actuator.

9. An automatic robot dance generation system, characterized in that, The robot dance automatic generation system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the robot dance automatic generation system to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the robot dance automatic generation system, the robot dance automatic generation system performs the method as described in any one of claims 1-8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is run on the robot dance automatic generation system, the robot dance automatic generation system performs the method as described in any one of claims 1-8.

Images