Group dance generation method based on music driving

Abstract

The invention discloses a group dance generation method based on music driving, and aims to generate a coordinated multi-dancer dance choreography in combination with music beats and a three-dimensional action sequence. The relative position relation between dancers is effectively kept through the dancer positioning embedding technology, it is ensured that the distance between the dancers is within a reasonable range through a distance consistency loss function, an exchange mode embedding mechanism is designed to mark the position changing track of the dancers, a step method adapter module is constructed to optimize and adjust original actions, and the foot sliding phenomenon is remarkably reduced; a long sequence diffusion sampling strategy is provided, position information is injected into noise input to suppress attitude sudden change, and meanwhile, a state space sequence layer is introduced to enhance the selective processing capacity of a model on a long sequence; and finally, reconstructing coordinated group dance by using the generated three-dimensional action sequence. Experiments show that TCDiff + + shows leading performance in a long-time scene, and a high-quality and coherent group dance sequence can be generated.

Description

Technical Field

[0001] The present invention relates to the fields of computer animation and artificial intelligence, and in particular to a group dance generation method based on music driving. Background Art

[0002] Dance, one of the most expressive art forms, has profoundly influenced culture, film, and academia. The choreography process requires a deep integration of body movement and musical rhythm, achieving a harmonious unity of artistic expression through the synchronization of rhythm and structure. Early music-driven choreography focused primarily on solo dance. With the growing demand for immersive experiences, research has gradually shifted its focus to group dance. Despite initial exploration, existing group dance generation methods still face three key challenges: multi-dancer collisions, foot slippage of individual dancers, and abrupt positional shifts between dancers during long-term generation. For example, in the AIOZ-GDance dataset, over 80% of the movements are similar. This leads to ambiguous dancer identities, making it difficult for the model to distinguish individual dancers, leading to collisions. Foot slippage occurs when a dancer's feet appear to glide across the ground while their upper body maintains normal motion. This problem often stems from the difficulty of accurately modeling the relationship between global trajectory and local body rotation. Existing models have demonstrated the ability to generate group dance movements lasting a few seconds, while a full group dance performance synchronized to music typically lasts several minutes or, for musicals, even hours. This disparity highlights a critical gap in current capabilities and underscores the urgent need for methods capable of generating prolonged group dances. Summary of the Invention

[0003] The present invention aims to provide a music-driven group dance generation method that uses a diffusion-based generative model to generate coordinated dance movements for multiple dancers. Unlike existing two-stage approaches that separate dancer trajectory prediction and movement generation, this method employs an end-to-end architecture to simultaneously generate dancer trajectories and dance movements, thereby improving the overall coherence of movement and position. This method can maintain a harmonious group dance formation while ensuring the natural and smooth flow of individual movements based on the input music.

[0004] The technical solution for achieving the purpose of the present invention is: a music-driven group dance generation method, comprising the following steps:

[0005] Step 1: Input audio conditions and obtain the rhythm and style features of L frames of target music signals. Use the music encoder to obtain the corresponding audio features, and use the group dance generation model to generate the corresponding motion sequence.

[0006] Step 2: Encode the position of each dancer based on their initial relative position, and use dance localization embedding to enhance the spatial representation capability of the input features;

[0007] Step 3: Diffusion initialization, add noise to the group dance action sequence and represent it as random noise x T , the audio feature M, the dancer's spatial exchange pattern S and the time step T are injected into the noise sequence using a conditional generation method as the initial conditions of the diffusion process to jointly guide the movement generation;

[0008] Step 4: The generation process is represented as the reverse diffusion process D(·), by estimating The group dance decoder uses the fusion projection module to perform high-dimensional mapping of features to reduce dancer ambiguity, and strengthens feature extraction through the timing selection mechanism of the sequence decoder to output the original motion. The long-sequence group dance sampling strategy uses the generated results as conditions to autoregressively expand short sequences, solving the position consistency problem when generating long sequences of group dance.

[0009] Step 5: Apply the footwork adapter to the generated dance sequence to adjust the foot joint motion of each dancer according to the inter-frame displacement of each dancer to ensure that the dancer's foot motion is consistent with the whole body motion and avoid sliding artifacts. Indicates that FA(·) is the adaptation network, is the speed sequence;

[0010] Step 6: From Take the upper body from Remove the lower body and perform feature splicing Generate the final action sequence.

[0011] Furthermore, in step 1, the corresponding audio features are obtained through the music encoder where m i Represents the audio feature vector of the input music sequence at the i-th frame; the corresponding motion sequence is generated by the group dance generation model in There are C dancers in total, x( i ), c represents the c-th dancer pose sequence generated at the i-th frame; the audio feature is the feature set M∈R derived from the Librosa toolkit 35 The feature set includes: audio envelope signals, 20 MFCCs (Mel-Frequency Cepstral Coefficients) that characterize spectral details, 12 chromaticity features for pitch analysis, and binary markers for beat detection and peak marking. These features are critical for achieving precise synchronization of movements with audio rhythm.

[0012] Furthermore, the dancer motion modeling in step 1 adopts the 24-joint SMPL human body model framework, where each joint pose d∈R 24×6＝144The model also contains a binary identifier f∈R that represents the contact state between the heel and the toe. 4 , and the 3D root bone position p∈R 3 , together constitute a complete motion descriptor Compared with traditional 3D key point methods, this rotation representation-based approach significantly improves the smoothness of action sequences.

[0013] Furthermore, the dancer localization embedding in step 2 optimizes the feature representation by injecting the dancer’s spatial position information to alleviate the collision problem. Extract the x-axis coordinate of each dancer Use the sorting operator σ(·) to sort in ascending order of coordinate values: satisfy Generate an ordered sequence x sorted Apply the diffusion forward process to inject Gaussian noise and get the noisy input Finally, add it to the dance positioning embedding DPE to get

[0014] Furthermore, the fusion projection in step 4 enhances the dancers’ differentiation through the feature dimension-up strategy. The core principle is that high-dimensional feature space makes it easier to achieve data separability. Input motion sequence x∈R with C' channels, L frames, and hidden layer dimension d' C'×L×d' , concatenate features along the channel dimension and reconstruct as x'∈R L×(C'×d') , mapping the reconstructed vectors into a high-dimensional latent space using a multi-layer perceptron to capture the unique characteristics of each dancer. Compared to traditional single-dancer MLP projection, this method captures group-level differences in a higher-dimensional space, effectively alleviating ambiguity at the individual level.

[0015] Furthermore, the sequence decoder in step 4 enhances the model's selective attention to long-term temporal information through a structured state-space model architecture, thereby optimizing the feature screening of sequence data. The motion sequence x is first refined locally by the self-attention mechanism, and then the SSM architecture is used to implement temporal feature screening. The conditional information is then fused through the cross-attention mechanism, and the time step information T, the music condition M, and the dancer's spatial exchange pattern S are concatenated and injected into the feature linear modulation layer (FiLM). The exchange pattern S is an embedding derived from a digital sequence, representing the left and right position order of the dancer in the final frame. The position information comes from the sorted sequence x. sorted The left and right order of the final positions in . The indices of these orders are mapped through a linear transformation to generate the final exchange pattern S. The exchange pattern adds position information, which helps to reduce the spatial uncertainty caused by the exchange of dancers during the generation process.

[0016] Furthermore, the long sequence group dance sampling strategy in step 4 solves the position consistency problem when generating long sequences of group dance by segmented overlapping generation and recursive fusion mechanism. Its core lies in the use of autoregressive expansion strategy. For generating a 150-frame sequence, first generate a short segment with 75 frames of overlap. Apply noise to get noisy input The second half of the generated segment (frames 75-150) was then mixed with random noise. Splicing to construct new input By constraining the deterministic part of the input noise (the first 75 frames are generated content), randomness is reduced. Compared with the method of directly inputting the full sequence of random noise, this strategy provides cleaner position constraints in the generation stage, effectively alleviating the long-term generation position offset problem caused by the exchange of dancer positions across sampling periods.

[0017] Furthermore, the footwork adapter in step 5 optimizes the foot trajectory by specifically adjusting the dancer's lower body movement, directly using the original movement generated by the group dance decoder As a conditional source. First, extract the root bone position coordinates p from the original motion data. r , the frame-by-frame speed sequence is calculated by calculating the difference between adjacent frames This speed condition is input into the adaptation network composed of the linear layer and the ConcatSquashLinear layer to correct the step and generate the adapted motion On this basis, a motion synthesis strategy is adopted to retain the original motion Expressive upper body movements, integrated with adaptive exercise Optimized lower limb gait data, through feature splicing operation The resulting final movement sequence effectively eliminates foot sliding while maintaining the natural expressiveness of dance movements.

[0018] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned music-driven group dance generation method is implemented.

[0019] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the above-mentioned music-driven group dance generation method.

[0020] Compared with the existing technology, the present invention has the following significant advantages: (1) The present invention is an end-to-end group dance generation model that outperforms the existing technology in generating coherent position-body movements and handling long group dance generation; this more common design also provides ideas for future end-to-end models. (2) In order to reduce the collision of multiple dancers, the present invention proposes a dancer positioning embedding method to better maintain relative positions and modify the distance consistency loss so that it can be directly applied to the end-to-end model to constrain the distance between dancers within a reasonable range; a simple but effective FP module is used to effectively alleviate the problem of dancer identity ambiguity. (3) To address the problem of single dancer foot sliding, the present invention newly incorporates transposition pattern information to reduce spatial uncertainty and modifies the step adapter to directly enhance the consistency of foot movements and overall movements in the end-to-end model generation results. (4) To facilitate the generation of long dances, the end-to-end architecture of the present invention removes the dance trajectory navigator (DTN) module because this module often causes movement fluctuations due to spatial uncertainty during long generation processes. In addition, a long-group diffusion sampling strategy is proposed to reduce sudden position changes during long-duration generation by injecting position information into the noisy input. The present invention also incorporates the Sequence Selection Module (SSM) architecture into the model, improving its sequence selection capabilities in long-duration generation scenarios.

[0021] The present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the process of the present invention.

[0023] Figure 2 Figure 2 is a diagram of the problem solved by the present invention.

[0024] Figure 3 This is the TCDiff++ framework diagram of the present invention.

[0025] Figure 4 This is a detailed diagram of the FP module of the present invention.

[0026] Figure 5 A schematic diagram of a long-term strategy is generated for the present invention.

[0027] Figure 6 This is a comparison chart between the present invention and other group dance generation methods.

[0028] Figure 7 Generate result graphs for different numbers of people in the present invention.

[0029] Figure 8 This is a comparison of the trajectory of the present invention with other group dance generation methods in long-term generation.

[0030] Figure 9 Conduct user research on the present invention.

[0031] Figure 10 This is the LGDS module ablation experiment diagram based on long-term generation of the present invention.

[0032] Figure 11 This is a diagram of the FP module ablation experiment on long-term generation of the present invention. DETAILED DESCRIPTION

[0033] This paper discloses a music-driven group dance generation method, aiming to combine music beats with three-dimensional motion sequences to generate coordinated multi-dancer choreography. Currently, many existing methods still face three core issues in group dance generation: collisions between multiple dancers, foot slippage of individual dancers, and positional abrupt changes during long sequence generation. To address these challenges, this paper proposes an end-to-end music-driven group dance generation model, TCDiff++. Specifically, to mitigate the multi-dancer collision issue, this paper employs dancer positioning embedding technology to effectively maintain the relative positional relationships between dancers and ensures that the distance between dancers remains within a reasonable range using a distance consistency loss function. To address the issue of foot slippage of individual dancers, this paper designs an exchange pattern embedding mechanism to indicate the dancers' positional transitions and constructs a footwork adapter module to optimize and adjust the original movements, significantly reducing foot slippage. For long-sequence group dance generation, this paper proposes a long-sequence diffusion sampling strategy that injects position information into the noise input to suppress posture abrupt changes. It also introduces a state-space sequence layer to enhance the model's selective processing capabilities for long sequences. Ultimately, a coordinated group dance is reconstructed using the generated three-dimensional motion sequence. Experiments show that TCDiff++ demonstrates leading performance in long-duration scenes, especially in the complex choreography example shown in the accompanying figure, where it can generate high-quality and coherent group dance sequences. The accompanying figure demonstrates the stunning performance of TCDiff++.

[0034] like Figure 1 As shown in Figure 1, the end-to-end music-driven group dance generation model TCDiff++ proposed in this paper consists of a group dance decoder (GDD) and a footwork adapter (FA). The group dance decoder (GDD) first generates an original action sequence without trajectory overlap based on the music input. Then, the gait adapter (FA) uses the position information of the original action to further adjust the foot movement and generate an adapted action with footsteps. These footsteps can alleviate the problem of foot sliding. Finally, we integrate the footsteps into the original movements to get a harmonious dance sequence with solid steps and no dancers colliding. The specific steps are as follows:

[0035] Step 1: Input audio conditions, obtain the rhythm and style features of L frames of target music signals, obtain the corresponding audio features through the music encoder; and generate the corresponding motion sequence through the group dance generation model;

[0036] Step 2: Encode the position of each dancer based on their initial relative position, and use dance localization embedding to enhance the spatial representation capability of the input features;

[0037] Step 3: Diffusion initialization, add noise to the group dance action sequence and represent it as random noise x T , the audio feature M, the dancer's spatial exchange pattern S and the time step T are injected into the noise sequence using a conditional generation method as the initial conditions of the diffusion process to jointly guide the movement generation;

[0038] Step 4: The generation process can be expressed as a reverse diffusion process D(·), by estimating The group dance decoder first performs high-dimensional mapping of features through the fusion projection module to reduce dancer ambiguity, and then strengthens feature extraction through the temporal selection mechanism of the sequence decoder, and finally outputs the original motion The long-sequence group dance sampling strategy uses the generated results as conditions to autoregressively expand short sequences, solving the position consistency problem when generating long sequences of group dance.

[0039] Step 5: Apply the footwork adapter to the generated dance sequence to adjust the foot joint motion of each dancer according to the inter-frame displacement of each dancer to ensure that the dancer's foot motion is consistent with the whole body motion and avoid sliding artifacts. Indicates that FA(·) is the adaptation network, is the speed sequence;

[0040] Step 6: From Take the upper body from Remove the lower body and perform feature splicing Generate the final action sequence.

[0041] Preferably, in step 1, the audio features derived directly from the Librosa toolkit are used. These features are crucial for achieving precise synchronization of movements with the audio rhythm. The dancer's movements are modeled using the 24-joint SMPL human body model framework. Compared to traditional 3D keypoint methods, this rotation-based representation significantly improves the smoothness of the movement sequence.

[0042] Preferably, Dance Positioning Embedding is introduced in step 2 to strengthen the constraints on the dancers' global positions by encoding their relative spatial positions in the formation (for example, encoding the dancers in order from left to right on the stage). This embedding vector serves as an additional input feature of the model, enabling the generation process to recognize and maintain the established relative arrangement relationship between dancers, avoiding disordered interweaving of dancer trajectories.

[0043] Preferably, the generative model trained in step 4 includes a fusion projection module, which processes the high-dimensional feature space and introduces a fusion projection module (FP), which separates and projects the features of different dancers, further alleviating the problem of confusion of dancer features and ensuring that the movement features of each virtual dancer are independent and identifiable.

[0044] Preferably, in step 4, at the beginning of each dance sequence generation, prior information reflecting the dancers' initial positions and formations is injected into the random noise input to the diffusion process. This allows the model to retain the dancers' relative positions during the denoising and restoration of the action sequence, avoiding sudden and drastic position shifts. Furthermore, this strategy imposes smoothing constraints on the amplitude of dancer position changes during the diffusion sampling iterations, mitigating trajectory oscillations and thus achieving a smooth transition between positions and movements during long-term group dance generation.

[0045] Preferably, step 5 uses the displacement information between consecutive frames of the dancer to adjust the movement of the foot joints: when it is detected that the dancer's global movement amplitude is small, the constraints on the foot position of the corresponding frame are automatically strengthened to avoid unnecessary displacement; when the dancer needs to change position, the corresponding foot lifting and stepping movements are generated according to the movement direction and distance. By introducing this mechanism, the dancer's foot movements in the generated dance are consistent with their whole body displacement, which significantly reduces the phenomenon of foot slipping. At the same time, the relative forward kinematic constraint loss (RFK loss) is combined in the model training process to strengthen the connection between the dancer's torso (root node) movement and the leg and foot movement, so that the foot landing and lifting are more in line with the real dance rules.

[0046] Through the synergistic effect of the aforementioned modules, this invention achieves the automatic generation of music-driven group dances, achieving technological breakthroughs in avoiding dancer collisions, reducing foot slippage, and maintaining stability over long sequences. Experimental tests have shown that compared to existing methods, the group dances generated by this invention significantly improve both the quality of individual movements and the effectiveness of group coordination, enabling the continuous production of highly synchronized, expressive, and harmonious dance performances over longer musical excerpts.

[0047] Table 1 Quantitative comparison with different methods.

[0048]

[0049] Table 2 Quantitative comparison of different methods with different number of people.

[0050]

[0051]

[0052] Table 3 Quantitative comparison with different methods under long-term generation.

[0053]

[0054] Table 4. Ablation experiments.

[0055]

[0056] The present invention uses multi-dancer and single dancer evaluation indicators to evaluate the model. For multi-dancer evaluation, the following are used:

[0057] Group Motion Realism (GMR): Feature similarity is measured by Frechet Inception Distance (FID).

[0058] Group Motion Correlation (GMC): Evaluates movement coherence by generating cross-correlations between dancers.

[0059] Trajectory Intersection Frequency (TIF): Evaluates the frequency of collisions between dancers.

[0060] For single dancer assessment, use:

[0061] Frechet Inception Distance (FID): quantifies the feature similarity between a single dance move and the real dance move.

[0062] Generation Diversity (Div): Evaluates the diversity of dance movements through the dynamic characteristics of the movements.

[0063] Motion-Music Consistency (MMC): evaluates how well the generated dance movements are synchronized with the music rhythm.

[0064] Physical Foot Contact score (PFC): measures the physical plausibility of gait by considering the correlation between center of mass and foot velocity.

[0065] Tables 1 and 2 compare the present invention with other methods. The single-dancer model EDGE performs poorly in multi-dancer scenarios, exhibiting severe foot sliding and frequent multi-dancer collisions (high TIF and PFC) due to dancer ambiguity. Similarly, GCD over-focuses on the interaction between dancers and ignores the modeling of coordinate differences between dancers, which makes it susceptible to dancer ambiguity and leads to severe foot sliding (high PFC). CoDancers reduces ambiguity (low TIF) but compromises the correlation between dancers (low GMC) and formation integrity, resulting in uncoordinated group formations. In addition, CoDancers ignores the relationship between steps and movements, limiting its ability to combine step movements with position changes, thereby hindering the generation of accurate steps (low PFC). TCDiff reduces ambiguity, improves coordination, and enhances formation quality by separating dancer coordinates and movements into two stages. However, this two-stage generation process separates position trajectories from body modeling, resulting in inconsistencies between movements and positions, which compromises individual fidelity (FID) and leads to unsatisfactory results. In contrast, our TCDiff++ adopts an end-to-end architecture to mitigate incoherence in the generation process and leverages its internal module design to address dancer ambiguity. This results in more consistent positioning and body movements, leading to a sustained advantage in group dance metrics and excellent performance in Div and FID for individual dances.

[0066] Table 3 extends the test to 720 frames to evaluate long-term performance. It can be observed that all models exhibit varying degrees of performance degradation during long-term generation. This includes issues such as frame freezing (resulting in reduced MMC) and mutation swapping (resulting in decreased PFC and TIF performance). Due to the dancer swapping phenomenon, simple long diffusion sampling (EDGE, GCD) leads to severe mutation swapping during long-term generation, as the loss of spatial information disrupts position consistency between epochs. Among them, EDGE maintains more stable motion quality by focusing primarily on individual dancers. However, this comes at the expense of ignoring global features, resulting in more severe dancer collisions (higher TIF) and poor group dance performance. Both CoDancers and TCDiff employ an autoregressive approach in the first stage, with TCDiff generating trajectory coordinates and CoDancers synthesizing complete dance movements. However, CoDancers focuses solely on individual-level information and ignores group-level features, resulting in decreased performance on group dance metrics. TCDiff estimates dancer positions without explicitly modeling movements, making it susceptible to errors due to motion uncertainty. These errors accumulate over time, leading to disjointed motion and inconsistent positions, limiting its applicability for long-term generation. In contrast, our proposed TCDiff++ employs an end-to-end design to improve the consistency between position and body motion, achieving the best long-term performance. It also integrates past results to maintain position consistency and mitigate the impact of sudden swaps.

[0067] Table 4 evaluates the effectiveness of the modules designed in the present invention in long-term scenarios (720 frames). The performance of different simplified models is arranged from worst to best to provide an intuitive order of performance improvement. The results show that the model performs best when all modules are applied simultaneously. Overall, all modules help improve the model's performance on group indicators, thus demonstrating their effectiveness in enhancing the harmony of group dance. This improvement is attributed to the fact that they alleviate the collisions and foot sliding of multiple dancers to varying degrees, making the dance movements more realistic. It is worth noting that the proposed fusion projection (FP) and long group diffusion sampling (LGDS) significantly reduce mutation exchanges. Among them, the simple and effective FP module contributes the most to improving the performance of the end-to-end model, effectively alleviating the adverse effects of dancer ambiguity. At the same time, the LGDS strategy improves the performance of the model in expanding group dance action sequences within the diffusion framework in a training-free manner, effectively reducing the occurrence of sudden exchanges. In terms of mitigating foot sliding, the Footwork Adaptor (FA) and Swap Mode (SM) modules play an important role. They enhance the model's ability to capture sequence dependencies and incorporate additional spatial information. In terms of mitigating multi-dancer collisions, the Dancer Positioning Loss (DPE) and the distance-consistency loss (L D ) are particularly effective because they amplify the characteristic differences between dancers and introduce spatial constraints.

[0068] Specifically, after implementing FP, the model showed significant improvements in both multi-dancer and single-dancer metrics. GMR decreased from 30.75 to 14.67, indicating improved group formation quality; GMC increased from 79.14 to 81.64, reflecting increased connectivity between dancers. Furthermore, TIF decreased from 0.23 to 0.15, indicating reduced dancer collisions and improved formation organization. With FP, the model reduced random mutation exchanges and unrealistic formations, resulting in more structured and coherent dances. This improvement resulted in a slight decrease in Div from 17.48 to 16.19, while FID decreased significantly from 43.69 to 20.37, enhancing the realism of the generated dances. Finally, PFC decreased from 1.89 to 1.53, improving the physical plausibility of the steps. These results confirm that FP plays a crucial role in improving the structural coherence and visual quality of dance generation.

[0069] The FA of the present invention significantly improves the PFC value of the model and even improves the model's performance in group dance indicators and FID. It is worth noting that adding FA reduces PFC from 2.34 to 1.53, enhancing the physical credibility of the steps. After using FA, GMR dropped from 25.37 to 14.67, indicating that the quality of group formation has improved; while GMC increased from 80.75 to 81.64, enhancing the coordination of the dancers. For single dancer evaluation, FA significantly reduced FID from 33.27 to 20.37, enhancing the similarity with the real dance sequence by ensuring more realistic steps. In addition, Div increased from 12.75 to 16.19, indicating richer movement variations. This is because FA captures the connection between position changes and steps, resulting in more realistic dance steps.

[0070] For group dance evaluation, the SD module reduces GMR from 25.75 to 14.67, indicating a significant improvement in the realism of the group formation. GMC increases slightly from 80.72 to 81.64, indicating improved dancer coordination. TIF also decreases from 0.17 to 0.15, reflecting a reduction in dancer collisions. For individual dance evaluation, FID increases from 28.75 to 20.37, indicating greater consistency with real dance sequences. Div increases from 14.24 to 16.19, indicating a wider variety of motions in the full model. Our SD leverages the aging properties of the SSM architecture to enhance the selectivity of sequential data, making it easier to generate expressive motions and less susceptible to noise in long-duration scenes. While the increased diversity (Div) introduces some redundant motions that are slightly inconsistent with the music rhythm (lower MMC), it significantly improves the model's visual performance, as reflected in the improved group dance metrics.

[0071] After implementing the DPE module, GMR increased from 20.97 to 14.67, indicating a significant improvement in the realism of group formation. GMC remained stable, increasing slightly from 80.91 to 81.64, reflecting improved coordination between dancers. TIF decreased from 0.18 to 0.15, indicating reduced dancer collisions. For single dance evaluation, FID increased from 40.75 to 20.37, reflecting a higher consistency between the generated dance and the real sequence. Div increased from 10.52 to 16.19, indicating a richer range of movement variations in the full model. MMC remained unchanged at 0.23, indicating synchronization with the music rhythm. PFC decreased slightly from 1.58 to 1.53, indicating a slight improvement in the legibility of the steps. DPE effectively mitigated multi-dancer collisions (reduced TIF) by incorporating left-right spatial information of the dancers, which helped maintain their relative positions and improved the overall model performance (improved group dance metrics and FID).

[0072] Compared to the full model, using SM improves GMR from 16.71 to 14.67, indicating improved realism in group formation. TIF decreases slightly from 0.17 to 0.15, indicating fewer dancer collisions. For the individual dance evaluation, FID increases from 25.97 to 20.37, indicating better consistency with the real dance sequence. Div decreases slightly from 15.75 to 16.19, reflecting a slight improvement in movement diversity. PFC decreases slightly from 1.75 to 1.53, reflecting a slight improvement in footwork plausibility. These improvements can be attributed to the role SM plays in reducing spatial uncertainty caused by swapping movements. By providing the dancers' starting and ending positions, SM helps alleviate uncertainty during spatial changes (such as swapping), resulting in improved group coherence, more realistic footwork, and improved overall performance.

[0073] LGDS mainly alleviates sudden swaps during long stretches. Although this short-lived phenomenon has little impact on quantitative indicators due to its short duration, it can significantly reduce visual quality. Therefore, although LGDS does not improve as much as FP in terms of indicators, it plays a vital role in enhancing visual perception. LGDS significantly reduces the occurrence of sudden swaps in long-term generation (reducing TIF) and can be applied without training in the inference phase, significantly improving model performance. This is because LGDS enforces consistency in the input phase, thereby reducing randomness and ensuring clearer position information during the generation process, thereby minimizing sudden swaps.

[0074] By emphasizing appropriate spatial relationships, LD helps maintain optimal distances between dancers. This makes group dance steps more natural and coherent. As a result, the model achieves better overall performance in terms of group dance metrics.

[0075] Attachment Figure 2 We demonstrate three key issues addressed by our method: multi-dancer collisions, single-dancer sliding, and long group dance generation where the framed characters suddenly swap positions. Our approach eliminates these issues and delivers superior visual aesthetics.

[0076] Attachment Figure 3 The end-to-end TCDiff++ framework presented in this paper consists of two key components: Group Dance Decoder (GDD) and Footwork Adapter (FA). GDD first generates a raw action sequence without trajectory overlap according to the given music. Then, FA uses the position information of the original action to refine the steps and generate an adaptive action The movement improves the footwork to reduce foot slippage. Finally, the adaptive footwork is integrated into the original movement to generate a harmonious dance sequence with stable footwork and less collisions between dancers. Compared with the previous two-stage version, TCDiff++ only requires one training stage and shows better footwork-movement coherence.

[0077] Attachment Figure 4 We demonstrate that our Fusion Projection (FP) module addresses the problem of dancer ambiguity. Imbalanced feature representations can cause positions to be misinterpreted as similar, leading to identical predictions. The FP module increases the input dimensionality to enhance dancer differentiation, preserve positional differences, and reduce collisions.

[0078] Attachment Figure 5 We present our Long Group Diffusion Sampling (LGDS) method, which first generates partially overlapping fragments and then merges them to form a complete sequence. Unlike simple sampling, LGDS enforces consistency at the input stage rather than the sampling stage. This approach reduces randomness and ensures clearer positional information during generation, thereby reducing mutational exchange.

[0079] Attachment Figure 6 A visual comparison of the present invention and other methods is shown. The single dancer model EDGE is affected by the dancer blur phenomenon, resulting in dancer collisions (highlighted by boxes). This shows that there are difficulties in directly applying the single dancer model to the multi-dancer generation scenario. The group dance model GCD uses a global attention mechanism to capture dancer interactions, however, it ignores position differences, resulting in multi-dancer collisions. Due to incomplete group information, CoDancers will generate unreasonable initial positions, resulting in serious overlap. TCDiff solves the trajectory overlap problem caused by dancer blur through a two-stage generation method. However, this two-stage process separates trajectory modeling from body modeling, resulting in poor performance. This separation causes the dancer's coordinates to gradually converge, ultimately leading to collisions. In contrast, the present invention TCDiff++ adopts an end-to-end generation method, using the design of internal modules to enhance the distinction between spatial features, thereby solving the dancer ambiguity problem and producing more coherent posture-body movements.

[0080] Attachment Figure 7 Results generated by our method using different numbers of dancers are presented. Our method is able to capture group-level interactions while maintaining the quality of individual movements. It not only generates harmonious choreography (e.g., with five dancers), but also generates diverse interactions such as exchanges (with three dancers) and varied movements (with two and four dancers).

[0081] Attachment Figure 8This paper presents a comparison of the position trajectory visualization of our method with other methods under long-term generation conditions (720 frames). To better illustrate the dancer formation, we visualize not only the dancers' pose skeletons but also their position trajectories, thereby more comprehensively demonstrating the dynamic formation transitions. Due to the dancer swapping phenomenon, simple long diffusion sampling methods (EDGE, GCD) can produce severe mutation swapping during long-term generation. This is because the lack of spatial information during the sampling phase leads to inconsistent generation of dancer positions in different epochs, resulting in mutation swapping when the generated segments are merged. Since CoDancers only generates one dancer at a time, it lacks information about the dancers' positions, such as their positions. Therefore, due to incomplete group information, CoDancers generates unreasonable initial positions, resulting in significant overlap. Furthermore, dancers tend to perform nearly identical movements with minimal interaction, losing the collaborative dynamics that distinguish group dance from solo performances. TCDiff generates dancer positions without considering their movements, making it prone to introducing incorrect position estimates due to uncertainty. These errors can significantly accumulate over long periods of group dance generation, causing the dancers' coordinates to gradually converge and eventually collide, rendering TCDiff unsuitable for extended sequences. In contrast, our end-to-end approach considers previously generated results during the extension process, improving positional consistency across time periods. This effectively reduces the occurrence of crossover.

[0082] Attachment Figure 9 The user study of the present invention based on four criteria (realism of movement, correlation between music and movement, beauty of formation, and harmony of dancers) is presented. The present invention has gained more favor from users and demonstrated superior performance in aesthetic appeal. The single-dancer model EDGE performed poorly in multi-person scenarios, with severe foot sliding and frequent multi-dancer collisions (high TIF and PFC) due to dancer ambiguity. Similarly, GCD over-focused on the interaction between dancers and neglected to model the coordinate differences between dancers, which made it susceptible to dancer ambiguity, resulting in severe foot sliding (high PFC). CoDancers reduced ambiguity (low TIF) but compromised the correlation between dancers (low GMC) and formation integrity, resulting in uncoordinated group formations. The results show that the visual effects generated by the present invention are most popular with users.

[0083] Attachment Figure 10 The ablation effect of the Long Group Diffusion Sampling (LGDS) module under long-term generation conditions is demonstrated. LGDS effectively reduces the abrupt switching of dancer roles during long-term generation and enhances spatial consistency.

[0084] Attachment Figure 11The ablation effect of the Fusion Projection (FP) module under long-term generation conditions is demonstrated. Skeletal motion visualization shows that the FP module effectively alleviates the overlap and mutation exchange phenomena.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A music-driven group dance generation method, characterized in that: The steps include: Step 1: Input audio conditions and obtain the rhythm and style features of L frames of target music signals. Use the music encoder to obtain the corresponding audio features, and use the group dance generation model to generate the corresponding motion sequence. Step 2: Encode the position of each dancer based on their initial relative position, and use dance localization embedding to enhance the spatial representation capability of the input features; Step 3: Diffusion initialization, add noise to the group dance action sequence and represent it as random noise x T , the audio feature M, the dancer's spatial exchange pattern S and the time step T are injected into the noise sequence using a conditional generation method as the initial conditions of the diffusion process to jointly guide the movement generation; Step 4: The generation process is represented as the reverse diffusion process D(·), by estimating The group dance decoder uses the fusion projection module to perform high-dimensional mapping of features to reduce dancer ambiguity, and strengthens feature extraction through the timing selection mechanism of the sequence decoder to output the original motion. The long-sequence group dance sampling strategy uses the generated results as conditions to autoregressively expand short sequences, solving the position consistency problem when generating long sequences of group dance. Step 5: Apply the footwork adapter to the generated dance sequence to adjust the foot joint motion of each dancer according to the inter-frame displacement of each dancer to ensure that the dancer's foot motion is consistent with the whole body motion and avoid sliding artifacts. Indicates that FA(·) is the adaptation network, is the speed sequence; Step 6: From Take the upper body from Remove the lower body and perform feature splicing Generate the final action sequence.

2. The method for generating group dance based on music driving according to claim 1, characterized in that: In step 1, the corresponding audio features are obtained through the music encoder where m i Represents the audio feature vector of the input music sequence at the i-th frame; the corresponding motion sequence is generated by the group dance generation model in There are C dancers in total, x (i),c represents the c-th dancer pose sequence generated at the i-th frame; Audio The feature is the feature set M∈R derived from the Librosa toolkit 35 The feature set includes: audio envelope signal, 20 MFCCs representing spectral details, 12 chromatic features for pitch analysis, and binary identifiers for beat detection and peak marking.

3. The method for generating group dance based on music driving according to claim 1, characterized in that: The motion sequence generated in step 1 adopts the 24-joint SMPL human body model framework, where each joint pose d∈R 24×6＝144 The model also contains binary identifiers f∈R representing the contact state of the heel and toe with the ground. 4 , and the 3D root bone position p∈R 3 , together constitute a complete motion descriptor 4. The method for generating group dance based on music driving according to claim 1, characterized in that: The dance localization embedding in step 2 optimizes the feature representation by injecting the dancer's spatial position information, and the real motion sequence of C dancers Extract the x-axis coordinate of each dancer Use the sorting operator σ(·) to sort in ascending order of coordinate values: satisfy Generate an ordered sequence x sorted Apply the diffusion forward process to inject Gaussian noise and get the noisy input Finally, add it to the dance positioning embedding DPE to get 5. The method for generating group dance based on music driving according to claim 1, characterized in that: The fusion projection in step 4 enhances the dancer's discrimination by the feature dimension-raising strategy; the input motion sequence x∈R contains C' channels, L frames, and hidden layer dimension d' C'×L×d' , concatenate features along the channel dimension and reconstruct as x'∈R L×(C'×d') ,The unique features of each dancer are obtained by mapping them to a high dimensional latent space and reorganizing the vectors.

6. The method for generating group dance based on music driving according to claim 1, characterized in that: The sequence decoder in step 4 enhances the model's selective attention to long-term temporal information through a structured state-space model architecture, optimizing the feature selection of sequence data. The motion sequence x is first refined locally by the self-attention mechanism, followed by temporal feature selection using the SSM architecture. The conditional information is then fused through the cross-attention mechanism, and the time step information T, audio condition M, and dancer spatial exchange pattern S are concatenated and injected into the feature linear modulation layer. The exchange pattern S is an embedding derived from a digital sequence, representing the left and right position order of the dancer in the final frame. The position information comes from the sorted sequence x. sorted The left and right order of the final positions in ; the indices of these orders are mapped through a linear transformation to produce the final exchange pattern S.

7. The method for generating group dance based on music driving according to claim 1, characterized in that: The long sequence group dance sampling strategy in step 4 solves the position consistency problem when generating long sequences of group dance through segmented overlapping generation and recursive fusion mechanism; for generating 150-frame sequences, first generate a short segment with 75 frames of overlap Apply noise to get noisy input The second half of the generated segment is then mixed with random noise Splicing to construct new input Reduce randomness by constraining the deterministic part of the input noise.

8. The method for generating group dance based on music driving according to claim 1, characterized in that: The footwork adapter in step 5 optimizes the foot trajectory by adjusting the dancer's lower body movement, directly using the original movement generated by the group dance decoder As a conditional source, where f r ∈R 4 Indicates the contact status between the heel and toe and the ground, p r ∈R 3 Indicates the root bone position, d r ∈R 24×6＝144 6D rotation is used to represent the posture of each joint; first, the root bone position coordinate p is extracted from the original motion data r , the velocity sequence between the i-th frame and the i-1-th frame is calculated by the adjacent frame difference where v i Reflects the dancer's position change in frame i relative to frame i-1; this speed condition Input the adaptation network FA(·) composed of linear layer and ConcatSquashLinear layer to perform step correction and generate the adapted motion Use motion synthesis strategy to preserve original motion Expressive upper body movements Fusion Adaptive Sports Optimized lower limb gait movements Through feature splicing operation Generate the final action sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the music-driven group dance generation method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the instructions are executed by a processor, the music-driven group dance generation method according to any one of claims 1 to 8 is implemented.

Images