Human-machine collaborative dance motion optimization method and system based on digital twinning

Abstract

This application provides a method and system for optimizing human-machine collaborative dance motion based on digital twins. The method includes: collecting data from a performance scene and constructing a unified digital twin corresponding to the human, machine, scene, and time; performing semantic analysis on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships to generate prediction results of human-machine collaboration relationships; generating candidate robot motion trajectories and performing rolling optimization to obtain the target motion trajectory; comparing the deviation between the physical performance feedback and the simulation output of the unified digital twin, and updating the timing parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin based on the deviation results; and generating and outputting the real-time collaborative dance motion control results of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory. This application can improve the adaptability of human-machine collaboration, enhance the dynamic optimization capability of motion trajectories, and improve the accuracy of virtual-real linkage correction.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and system for optimizing human-computer collaborative dance motion based on digital twins. Background Technology

[0002] With the development of humanoid robots, digital twins, and human-robot collaborative control technologies, the demand for robots to participate in dance performances, interactive displays, and other application scenarios is gradually increasing. Existing research shows that digital twins can achieve real-time simulation, monitoring, and control through virtual mapping of physical systems; simultaneously, robot collaborative safety standards have evolved from safety requirements focused on individual robots to system-level safety constraints for the entire collaborative application. Recent public reports also indicate that humanoid robots have begun to enter dance and entertainment performance scenarios.

[0003] In existing technologies, dance performance robots mostly employ pre-choreographed trajectory execution, human motion capture redirection, or partial online tracking control. While these methods can reproduce basic movements, they are prone to problems such as unstable movement transitions, mismatched collaboration relationships, and conservative or insufficient safety controls when real dancers temporarily change speed, switch positions, shift rhythms, or collaborate at close range. Furthermore, existing solutions typically lack linkage and update mechanisms for performance sequence, collaboration relationships, and virtual-real discrepancies, making it difficult to ensure performance continuity while simultaneously ensuring human-robot collaboration safety and movement consistency. They also lack smooth recovery capabilities based on choreography semantics in abnormal states, thus failing to meet the application requirements of highly dynamic human-robot collaborative dance performance scenarios. Summary of the Invention

[0004] In view of this, embodiments of this application provide a human-computer collaborative dance motion optimization method and system based on digital twins to solve the problems of poor human-computer collaboration adaptability, insufficient dynamic optimization of motion trajectory, and difficulty in linking and correcting virtual-real deviations in the prior art.

[0005] A first aspect of this application provides a human-machine collaborative dance motion optimization method based on digital twins, comprising: collecting human motion data, robot state data, environmental perception data, and performance timing data in a performance scene to construct a unified digital twin corresponding to the human, robot, scene, and time; performing associative semantic analysis on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships based on the unified digital twin to generate a prediction result of the human-machine collaborative relationship corresponding to a future target continuous performance window; generating candidate robot motion trajectories in the unified digital twin based on the prediction result of the human-machine collaborative relationship, and performing rolling optimization based on motion reachability constraints, collaborative relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory; comparing the deviation between the physical performance feedback and the simulation output of the unified digital twin, and updating the timing parameters, motion parameters, risk thresholds, and constraint boundaries corresponding to the unified digital twin based on the deviation result; and generating and outputting the real-time collaborative dance motion control result of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory.

[0006] A second aspect of this application provides a human-machine collaborative dance motion optimization system based on digital twins, comprising: a data acquisition module for acquiring human motion data, robot state data, environmental perception data, and performance timing data in a performance scene, and constructing a unified digital twin corresponding to the human, robot, scene, and time; a parsing module for performing associative semantic parsing on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships based on the unified digital twin, and generating a prediction result of the human-machine collaborative relationship corresponding to a future target continuous performance window; a generation module for generating candidate robot motion trajectories in the unified digital twin based on the prediction result of the human-machine collaborative relationship, and performing rolling optimization based on motion reachability constraints, collaborative relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory; an update module for comparing the deviation between the physical performance feedback and the simulation output of the unified digital twin, and updating the timing parameters, motion parameters, risk thresholds, and constraint boundaries corresponding to the unified digital twin based on the deviation result; and an output module for generating and outputting the real-time collaborative dance motion control result of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory.

[0007] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By collecting human motion data, robot state data, environmental perception data, and performance timing data from the performance scene, a unified digital twin corresponding to human, robot, scene, and time is constructed. Based on the unified digital twin, semantic analysis is performed on the human motion sequence, performance timing, preset choreography relationships, and human-robot relative pose relationships to generate prediction results of human-robot collaboration relationships corresponding to the future target continuous performance window. According to the prediction results of human-robot collaboration relationships, candidate robot motion trajectories are generated in the unified digital twin, and rolling optimization is performed based on motion reachability constraints, collaboration relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory. The deviation between the physical performance feedback and the simulation output of the unified digital twin is compared, and the timing parameters, motion parameters, risk thresholds, and constraint boundaries corresponding to the unified digital twin are updated according to the deviation results. Based on the updated unified digital twin and the target motion trajectory, the real-time collaborative dance motion control results of the robot during the target performance process are generated and output. This application can improve human-robot collaboration adaptability, enhance the dynamic optimization capability of motion trajectory, and improve the accuracy of virtual-real linkage correction. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the human-computer collaborative dance motion optimization method based on digital twins provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the human-computer collaborative dance motion optimization system based on digital twins provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0010] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0011] With the development of humanoid robots, digital twins, and human-robot collaborative control technologies, robots have been gradually applied to scenarios such as dance performances and interactive displays. Existing solutions typically employ pre-programmed trajectory execution, human motion capture redirection, or partial online tracking control to enable the robot to perform corresponding movements according to preset actions or changes in human motion. While some solutions introduce digital twins for simulation verification or state mapping, most remain at the level of offline modeling, local correction, or single motion tracking, lacking sufficient handling of the linkage between changes in human motion during performance, the evolution of performance sequence, changes in human-robot relative relationships, and deviations between virtual and real states.

[0012] When existing technologies are applied to human-robot collaborative dance performances, they are prone to problems such as poor adaptability to human-robot collaboration, insufficient dynamic optimization of motion trajectories, and difficulty in coordinating and correcting virtual-real discrepancies. Specifically, when a live dancer experiences beat drift, movement adjustments, temporary position changes, or changes in interaction relationships during a performance, the robot struggles to make timely and coherent adjustments to its motion trajectory based on the performance context and the state of human-robot collaboration. Simultaneously, timing deviations, motion deviations, and constraint boundary deviations between the virtual model and the physical performance are difficult to perceive and correct synchronously, thus affecting the continuity of robot collaboration and control precision during the target performance.

[0013] In view of the problems existing in the prior art, this application proposes a human-machine collaborative dance motion optimization method based on digital twins. The method first collects human motion data, robot state data, environmental perception data, and performance timing data in the performance scene to construct a unified digital twin corresponding to the human, robot, scene, and time. Then, based on the unified digital twin, it performs associative semantic parsing on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships to generate a prediction result of the human-machine collaborative relationship corresponding to the future target continuous performance window. Based on this, according to the prediction result of the human-machine collaborative relationship, it generates candidate robot motion trajectories in the unified digital twin and performs rolling optimization by combining motion reachability constraints, collaborative relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory. Further, it compares the deviation between the physical performance feedback and the simulation output of the unified digital twin, and updates the timing parameters, motion parameters, risk thresholds, and constraint boundaries corresponding to the unified digital twin based on the deviation results. Finally, based on the updated unified digital twin and the target motion trajectory, it generates and outputs the real-time collaborative dance motion control result of the robot during the target performance process.

[0014] By adopting the above technical solutions, this application can improve human-machine collaboration adaptability, enhance dynamic optimization capabilities of motion trajectories, and improve the accuracy of virtual-real linkage correction. Compared with existing technologies, this application does not merely perform static choreography or simple tracking of robot movements, but integrates human-machine collaboration relationship prediction, trajectory generation and optimization within the digital twin, virtual-real deviation linkage update, and real-time control output into a unified process. This enables the robot to achieve more continuous and coordinated collaborative dance motion control based on changes in human-machine collaboration status and performance sequence during the target performance.

[0015] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0016] Figure 1 This is a flowchart illustrating the human-computer collaborative dance motion optimization method based on digital twins provided in an embodiment of this application. Figure 1 As shown, the method may specifically include: S101 collects human motion data, robot status data, environmental perception data, and performance timing data in the performance scene to construct a unified digital twin corresponding to people, machines, scene, and time; S102, based on a unified digital twin, performs semantic analysis on human motion sequences, performance timing, preset choreography relationships and human-machine relative pose relationships to generate prediction results of human-machine collaboration relationships corresponding to future target continuous performance windows; S103. Based on the prediction results of human-machine collaboration relationship, generate candidate motion trajectories of the robot in the unified digital twin, and perform rolling optimization based on motion reachability constraints, collaboration relationship constraints, beat phase constraints and interaction risk constraints to obtain the target motion trajectory. S104 compares the deviation between the physical performance feedback and the simulation output of the unified digital twin, and updates the timing parameters, motion parameters, risk thresholds and constraint boundaries of the unified digital twin based on the deviation results. S105 generates and outputs the real-time collaborative dance motion control results of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory.

[0017] In some embodiments, human motion data, robot state data, environmental perception data, and performance timing data in the performance scene are collected to construct a unified digital twin corresponding to the human, robot, scene, and time, including: Perform unified time base alignment, cross-modal space registration, and virtual-real parameter calibration on human motion data, robot state data, environmental perception data, and performance timing data to generate corresponding human action states, robot motion states, scene constraint states, and performance phase states. Based on the human body's motion state, the robot's movement state, the scene's constraint state, and the performance phase state, a mapping of the human-machine relative relationship and the scene's temporal sequence relationship are established to generate a unified digital twin corresponding to the physical performance process.

[0018] Specifically, this embodiment performs unified time-base alignment, cross-modal spatial registration, and virtual-real parameter calibration on multi-source heterogeneous data from the performance site, thereby generating a unified digital twin capable of representing the real performance process. Here, human motion data is used to characterize the dancer's skeletal posture changes, center-of-gravity shift trends, and limb movement rhythms during continuous performance; robot state data is used to characterize the posture, velocity, joint response, and body orientation of each robot's execution parts; environmental perception data is used to characterize stage boundaries, obstacle distribution, traversable areas, and relative positional relationships; and performance timing data is used to characterize the temporal phases corresponding to music beats, musical phrases, and action transitions. By incorporating the above data into the same reference frame, subsequent prediction of human-machine collaboration relationships and optimization of motion trajectories can be established on a unified, continuous, and updatable state foundation.

[0019] In the specific implementation process, the collected human motion data, robot state data, environmental perception data, and performance timing data are first aligned to a unified time base. Specifically, the music playback clock can be used as a global reference clock, and a unified timestamp is added to the data frames output by the human motion acquisition end, robot control end, and environmental perception end. Then, based on the phase position between adjacent beats, the data at each moment is resampled and time-aligned, so that the dancer's arm raising, the robot's rotation, the change of stage position, and the landing point of the strong beat of the music are established in the same time coordinate.

[0020] Subsequently, cross-modal spatial registration was performed on various types of data, mapping the human skeleton coordinate system, robot body coordinate system, stage global coordinate system, and virtual simulation coordinate system to a unified stage reference coordinate system. In addition, virtual and real parameter calibration was performed in combination with robot structural dimensions, dancer's initial position, stage boundary parameters, and sensor installation pose to eliminate scale deviation, installation deviation, and viewpoint deviation between different acquisition sources.

[0021] Based on the processed results, human motion state, robot motion state, scene constraint state, and performance phase state are generated respectively. Among them, human motion state represents the dancer's current motion configuration and its changing trend, robot motion state represents the robot's current motion response and executable boundaries, scene constraint state represents the spatial constraint relationship in the stage, and performance phase state represents the current performance's phase position and beat interval in the entire choreography.

[0022] In a specific example, a dancer and a humanoid robot collaborate on a 90-second duet dance performance. As the dancer extends their arms and moves diagonally forward, the system captures real-time changes in key limb points and the trajectory of their center of gravity shift. Simultaneously, it acquires the robot's upper limb joint angles, torso orientation, and chassis displacement. This data, combined with environmental constraints from the prop area on the left side of the stage and the front boundary area, as well as the performance phase data (currently between beats 24 and 28) of the music, enables unified time-base alignment and spatial registration.

[0023] Subsequently, the system establishes the relative orientation, spacing, and phase correspondence between the dancer and the robot, forming a mapping of human-robot relative relationships and scene temporal correlations within this continuous performance window. A unified digital twin, synchronously changing with the real performance, is then generated in the virtual domain. This unified digital twin continuously reflects the complete process of the dancer moving forward, the robot synchronously turning sideways, the gradual contraction of the distance between them, and the transition of the performance phase to the next musical phrase, providing a consistent data foundation for subsequent collaborative relationship prediction and trajectory optimization.

[0024] Through the above implementation methods, the human body, robot, environment, and time information at the performance site can be uniformly expressed under the same digital twin framework, improving the consistency of state modeling, enhancing the accuracy of virtual-real mapping, and providing reliable data support for subsequent optimization of human-machine collaborative dance movements.

[0025] In some embodiments, based on a unified digital twin, semantic analysis is performed on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships to generate prediction results of human-machine collaboration relationships corresponding to future target continuous performance windows, including: Perform temporal association encoding and relational state alignment on human motion sequences, performance timing, preset choreography relationships and human-machine relative pose relationships to generate collaborative semantic representations corresponding to the target continuous performance window; Based on collaborative semantic representation, and combined with unified digital twin, collaborative relationship evolution inference is performed to generate prediction results of human-machine collaborative relationships corresponding to the continuous performance window of future targets.

[0026] Specifically, the aforementioned unified digital twin has already uniformly represented people, machines, venues, and time. Therefore, this embodiment does not directly identify a single human action. Instead, it integrates the changing trends of the human action sequence within a continuous performance window, the relationship between rhythm and musical phrase progression in the performance sequence, the interactive organization methods in the preset choreography relationships, and the evolution of orientation, spacing, and position in the relative posture relationship between humans and machines into the associative semantic parsing process to obtain a collaborative semantic representation for subsequent motion optimization. This collaborative semantic representation is used to characterize the collaborative state of human and machine in the current performance segment in terms of action coordination, rhythmic connection, and spatial interaction, and serves as the input basis for subsequent inference of the evolution of collaborative relationships.

[0027] In the specific implementation process, the sequence of human movements, the performance timing, the preset choreographic relationships, and the relative posture relationship between the human and the machine are first encoded using temporal association. Specifically, the human movement state in the unified digital twin can be divided into several performance segments according to continuous beat points, and the temporal features of the limb unfolding trend, center of gravity shift trend, and orientation change trend in each segment can be extracted. Then, the beat point positions, strong and weak beat distribution, and musical phrase boundaries in the performance phase state are mapped onto the same time axis to establish the correspondence between changes in human movement and the progression of the performance timing.

[0028] Furthermore, by combining the collaborative organizational methods represented by pre-set choreographic relationships, such as following, responding, mirroring, or switching positions, the relative pose relationship between humans and machines within the current continuous performance window is aligned. This establishes a coupled mapping between changes in human movement, rhythmic progression, and changes in human-machine spatial relationships within a unified semantic framework, generating a collaborative semantic representation corresponding to the target continuous performance window. This collaborative semantic representation not only reflects the current action state but also the direction of relationship continuation as the current state evolves into subsequent beats.

[0029] In a specific example, a dancer and a humanoid robot perform a duet of modern dance. The current performance is within a continuous performance window from beat 36 to beat 44. The unified digital twin shows that starting from beat 36, the dancer shifts from a sideways stance to a diagonally forward position, accompanied by a transition in upper limb extension from a retracted state. The robot maintains a corresponding dance-like stance in response to the dancer. The system first performs temporal association coding on the sequence of human movements within this continuous performance window, identifying the continuous changes in the dancer's center of gravity shifting forward, shoulder turning, and increased upper limb extension.

[0030] Subsequently, based on the performance sequence corresponding to the window, the 40th beat is identified as the strong beat switching node and the 44th beat corresponds to the starting position of the new musical phrase. Then, according to the choreography requirement that the performance should maintain a transition from parallel response to cross-positioning in the preset choreography relationship, the relationship state alignment is performed on the relative orientation, spacing contraction trend and path intersection trend between the dancer and the robot to generate the collaborative semantic representation corresponding to the window.

[0031] Based on this collaborative semantic representation, the system further calls the unified digital twin to perform collaborative relationship evolution inference, and performs recursive analysis on the human-machine collaborative state from the 45th to the 52nd beat in the future. It obtains the prediction results that the robot should gradually transition from the current responding position to the cross-giving position, the relative orientation of the two should change from responding in the same direction to crossing and then separating, and the interaction distance should shrink before the 48th beat and then widen again after the 50th beat. Thus, the prediction results of the human-machine collaborative relationship corresponding to the future target continuous performance window are formed.

[0032] Through the above implementation methods, under the support of a unified digital twin, it is possible to perform unified semantic analysis and joint inference on changes in human body movements, performance sequence progression, choreography organization relationships, and human-machine relative pose evolution. This improves the continuity of collaborative relationship recognition and the foresight of prediction, providing more accurate relationship basis for subsequent generation of robot candidate motion trajectories and rolling optimization, thereby improving human-machine collaborative adaptability and motion coordination during performance.

[0033] In some embodiments, based on collaborative semantic representation, and combined with a unified digital twin, collaborative relationship evolution inference is performed to generate a prediction result of the human-machine collaborative relationship corresponding to the future target continuous performance window, including: Based on collaborative semantic representation, and combined with a unified digital twin, the current human-machine collaboration state is subjected to temporal unfolding and relational recursion to generate a collaborative relationship evolution sequence corresponding to the target continuous performance window; Based on the evolution sequence of the collaborative relationship, and combined with the unified digital twin, consistency corrections are performed on the relative pose trend of humans and machines, the performance phase connection state, and the interaction constraint state to generate prediction results of human-machine collaborative relationships.

[0034] Specifically, the aforementioned collaborative semantic representation has already performed unified semantic compression on the human action sequence, performance timing, preset choreography relationship, and human-machine relative pose relationship. Therefore, this embodiment does not extrapolate the human or robot actions in isolation at subsequent moments, but takes the current human-machine collaborative state as the starting point and performs timing expansion and relationship deduction along the target continuous performance window in the unified digital twin, so that the changes in collaborative relationships in the subsequent few beats can continuously evolve according to the current performance context.

[0035] The evolutionary sequence of the collaborative relationship here is used to describe the phased changes in the relative positions, orientation, rhythm, and interaction boundaries of the human and machine within the future continuous performance window. The prediction result of the human-machine collaborative relationship is the target relationship result that can be used for subsequent trajectory generation after consistency correction based on the evolutionary sequence of the collaborative relationship.

[0036] In the specific implementation process, the current human-machine collaboration state is first unfolded based on the collaborative semantic representation and combined with a unified digital twin. Specifically, the end point of the current continuous performance window can be used as the starting point for unfolding. According to the performance phase advancement rules, choreography relationship transmission rules, and human-machine relative relationship continuation rules recorded in the unified digital twin, recursive time segments are established for several subsequent shots. Then, within each time segment, based on the relative orientation, spacing distribution, path tendency, and action connection relationship in the current human-machine collaboration state, the candidate collaboration state corresponding to the next time segment is derived. The scene constraint state and robot motion state in the unified digital twin are used to eliminate states that do not meet the requirements of continuous performance, thereby forming the collaboration relationship evolution sequence corresponding to the target continuous performance window. This collaboration relationship evolution sequence reflects the gradual change of the relationship state, rather than a static judgment at a single moment.

[0037] Furthermore, after obtaining the evolution sequence of collaborative relationships, consistency correction is performed on the relative pose trends of the human and machine, the performance phase connection state, and the interaction constraint state based on this sequence and in conjunction with the unified digital twin. Specifically, the relationship states at each stage in the collaborative relationship evolution sequence are compared with the relative pose change trends in the unified digital twin to verify whether the shrinkage or expansion of the human-machine distance is consistent with the preset transposition direction. At the same time, in conjunction with the performance phase connection state, the relationship switching at strong beat nodes, musical phrase switching nodes, and action connection nodes is verified to meet the beat progression requirements. And in conjunction with the interaction constraint state, the subsequent crossover, parallelism, avoidance, and other relationship changes are verified to ensure that they break through the stage boundaries, close-range interaction boundaries, or robot execution boundaries. For the stage relationship states with deviations, callback correction and boundary convergence processing are performed to output the final human-machine collaborative relationship prediction result that satisfies the current performance context.

[0038] In a specific example, the dancer and the humanoid robot complete a duet segment from beat 52 to beat 60, transitioning from mirrored responses to cross-positional exchanges. Based on the collaborative semantic representation formed in the previous continuous performance window, the system identifies the current human-robot collaborative state as the dancer moving diagonally forward towards the center of the stage, the robot moving along the outer edge, and both sides maintaining a tendency to contract at an angle relative to each other. Subsequently, the unified digital twin, starting from beat 60, performs temporal unfolding and relational recursion on beats 61 to 68, obtaining a collaborative relationship evolution sequence. The collaborative relationship evolution sequence shows that the two sides will reach a convergence and approach state before beat 63, complete the orientation switch at the strong beat of beat 64, and enter a state of separation after beat 66.

[0039] Next, the system uses a unified digital twin to perform consistency correction on the evolution sequence of the collaborative relationship. It finds that the robot's lateral cutting amplitude is too large during the 64th to 65th beats in the original recursive result, which may overlap closely with the dancer's forward arm trajectory. Therefore, the system performs convergence correction on the relative pose trend of this stage according to the interaction constraint state, and fine-tunes the robot's orientation switching time to the second half of the 64th beat according to the performance phase connection state. Finally, it generates the prediction result of the human-machine collaborative relationship corresponding to the future target continuous performance window, which indicates that the robot should complete the position change by going around on the outside after the meeting and restore the separation and response relationship before the 68th beat.

[0040] Through the above implementation methods, continuous recursion and consistency correction of human-machine collaboration relationships can be performed based on collaborative semantic representation and unified digital twins, improving the coherence of collaboration relationship prediction, phase matching accuracy and interaction boundary adaptation capability, and providing a stable and accurate relationship foundation for subsequent robot candidate motion trajectory generation and rolling optimization.

[0041] In some embodiments, based on the prediction results of human-machine collaboration relationships, candidate robot motion trajectories are generated in a unified digital twin, including: Based on the prediction results of human-machine collaboration relationship, combined with the unified digital twin, the correlation mapping is performed on the relative state of human-machine relationship, the state of performance phase connection and the state of robot motion constraint corresponding to the target continuous performance window to generate candidate trajectory generation conditions. Based on the candidate trajectory generation conditions, a unified digital twin is invoked to perform temporal unfolding and boundary extrapolation on the robot's motion evolution process, generating the corresponding trajectory evolution results; Based on the trajectory evolution results, candidate robot motion trajectories that are compatible with the prediction results of human-machine collaboration are determined.

[0042] Specifically, the aforementioned steps have yielded a prediction result of the human-machine collaboration relationship corresponding to the future target continuous performance window. This prediction result is not simply a trend of human movement, but a comprehensive description of the relative positions, orientation changes, rhythm connections, and interaction boundary states of the human and machine during the subsequent continuous performance. Therefore, in generating candidate robot motion trajectories, this embodiment does not directly perform local interpolation on the robot's joint movements. Instead, it first converts the prediction result of the human-machine collaboration relationship into candidate trajectory generation conditions within a unified digital twin, and then performs temporal unfolding and boundary deduction on the robot's motion evolution process based on the candidate trajectory generation conditions, thereby obtaining a set of candidate motion trajectories that are adapted to the current performance context.

[0043] In some examples, the candidate trajectory generation conditions are used to characterize the relative relationship objectives, rhythmic propulsion requirements, and self-motion boundary requirements that the robot should meet within the target continuous performance window, while the trajectory evolution results are used to characterize the continuous evolution state of different trajectory paths in the unified digital twin and the degree of matching with the predicted cooperative relationship.

[0044] In the specific implementation process, based on the prediction results of human-machine collaboration, and combined with a unified digital twin, the relative relationship state of human and machine, the performance phase connection state, and the robot motion constraint state corresponding to the target continuous performance window are first correlated and mapped. Specifically, the relative spacing change trend, relative orientation switching trend, and path intersection trend in the prediction results of human-machine collaboration can be mapped to the stage position domain and time phase domain in the unified digital twin to determine the target standing position interval, target orientation interval, and action switching time period of the robot at each beat point; then, combined with the performance phase connection state, the strong beat landing point, musical phrase boundary, and the action response density corresponding to the transition beat are determined; and combined with the robot motion constraint state, the robot's movement range, rotation range, center of gravity transfer boundary, and continuous action connection boundary under the current execution capability are constrained and mapped to generate candidate trajectory generation conditions.

[0045] Subsequently, based on the candidate trajectory generation conditions, a unified digital twin is invoked to perform temporal unfolding and boundary extrapolation of the robot's motion evolution process. Specifically, the robot's motion state is unfolded frame by frame along the timeline of the target continuous performance window. At each unfolding node, the robot's pose change path and state transition relationship are derived based on the candidate trajectory generation conditions. Combining stage space constraints, human-robot close interaction boundaries, and robot motion continuity requirements, path branches that do not meet the constraints are shrunk or eliminated, generating corresponding trajectory evolution results. Finally, based on the trajectory evolution results, the degree of fit of each trajectory in terms of relative relationship matching degree, performance phase consistency degree, and motion evolution continuity is comprehensively judged to determine the robot candidate motion trajectory that matches the human-robot collaboration relationship prediction results.

[0046] In a specific example, a dancer and a humanoid robot perform a collaborative duo performance. The system generates prediction results of the human-robot collaboration relationship for beats 72 to 80 based on the aforementioned steps. The prediction results indicate that the dancer will move diagonally from the inside of the stage to the center before beat 74, and will briefly intersect with the robot at the strong beat position of beat 76 before continuing to move forward. The robot, on the other hand, needs to transition from a state of accompanying and responding to a state of circling around on the outside.

[0047] The system first maps the predicted human-machine collaboration relationship to a unified digital twin. Combined with the performance phase connection state, it determines that the robot needs to complete the lateral displacement preparation in the 73rd to 75th beat, complete the turning connection in the 76th beat, and enter the detour separation segment after the 77th beat. At the same time, combined with the robot's motion constraint state, it limits its lateral displacement amplitude to the current executable range and limits the rotation speed to the continuous control allowable range, thereby generating candidate trajectory generation conditions.

[0048] Subsequently, the unified digital twin performs a beat-by-beat temporal unfolding of the robot's motion evolution process based on the candidate trajectory generation conditions, and deduce the evolution results of multiple trajectories. One of the trajectories has a close overlap with the dancer's forward extension trajectory near the 76th beat, and another trajectory, although it meets the safety boundary, is inconsistent with the rhythm of the musical phrase after the 77th beat. Based on this, the system eliminates the incompatible trajectory and retains a trajectory that completes the outer cut before the 75th beat, completes the orientation adjustment on the 76th beat, and restores the separation and response relationship after the 78th beat as the robot's candidate motion trajectory.

[0049] Through the above implementation methods, the prediction results of human-machine collaboration relationships can be transformed into executable trajectory generation conditions. By relying on a unified digital twin, the robot's motion evolution process can be continuously deduced and boundary screening can be performed, thereby improving the adaptability, continuity, and constraint consistency of candidate motion trajectory generation and providing a reliable trajectory foundation for subsequent rolling optimization and real-time control.

[0050] In some embodiments, rolling optimization is performed based on motion reachability constraints, cooperative relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory, including: Based on the robot's candidate motion trajectory, combined with the prediction results of human-robot collaboration relationship and the unified digital twin, a time-varying constraint rolling optimization model corresponding to the target continuous performance window is constructed. Based on a time-varying constraint rolling optimization model, continuous window prediction evaluation and iterative correction are performed on candidate robot motion trajectories to generate trajectory optimization results. Based on the trajectory optimization results, the target motion trajectory that matches the prediction results of human-machine collaboration is determined.

[0051] Specifically, the aforementioned steps have generated candidate robot motion trajectories in the unified digital twin that are compatible with the predicted results of human-machine collaboration. However, the candidate robot motion trajectories are still candidate solutions for subsequent control solutions. They need to be further combined with the continuously changing human-machine collaboration state, performance phase advancement state, and scene interaction boundary state within the target continuous performance window to perform dynamic optimization, so as to improve the matching degree with the predicted collaboration relationship while ensuring the continuity of the trajectory.

[0052] Therefore, this embodiment does not perform a one-time static screening of candidate robot motion trajectories. Instead, it uses a unified digital twin as the prediction carrier to construct a time-varying constraint rolling optimization model that progresses with the continuous performance window of the target. In the continuous window, it predicts, evaluates and iteratively corrects the candidate robot motion trajectories to generate the final target motion trajectory that can be used for real-time control output.

[0053] In the specific implementation process, a time-varying constraint rolling optimization model corresponding to the target continuous performance window is first constructed based on the robot's candidate motion trajectory, combined with the prediction results of human-machine collaboration relationship and the unified digital twin. Specifically, the robot's candidate motion trajectory can be divided into several overlapping optimization sub-windows according to continuous beat points within the target continuous performance window. The relative standing position target, orientation connection target, and spacing change target in the human-machine collaboration relationship prediction results are mapped as collaboration relationship constraints. The robot's executable boundary, stage space boundary, and posture transfer boundary in the unified digital twin are mapped as motion reachability constraints. The strong beat nodes, musical phrase transition nodes, and action connection nodes in the performance phase state are mapped as beat phase constraints. At the same time, the close interaction area, path intersection area, and highly sensitive proximity area are mapped as interaction risk constraints. This constitutes a time-varying constraint rolling optimization model that dynamically changes over time. The core of this model is that the constraint weights and boundary ranges under different beat points are not fixed, but are adjusted synchronously according to the current human-machine state and subsequent performance context in the unified digital twin.

[0054] After establishing a time-varying constrained rolling optimization model, continuous window prediction evaluation and iterative correction are performed on the candidate robot motion trajectory based on this model. Specifically, within the first optimization sub-window, forward prediction is performed on the displacement trend, rotation trend, limb deployment trend, and beat response trend corresponding to the candidate robot motion trajectory. The deviation between the trajectory and the predicted human-robot collaboration relationship, the phase deviation between the trajectory and the beat phase advancement, and the safety margin between the trajectory and the interaction risk boundary are calculated. Then, based on the deviation and safety margin, constrained corrections are performed on the local path segments, posture switching segments, and action transition segments in the candidate robot motion trajectory to obtain the updated trajectory segments.

[0055] The updated trajectory segment is then written back to the unified digital twin, and the optimization window is scrolled backward along the time axis. Prediction evaluation and iterative correction are repeatedly performed on subsequent optimization sub-windows to generate a complete trajectory optimization result. The iterative correction here includes not only path position adjustment, but also beat placement fine-tuning, attitude switching timing correction, and local motion amplitude convergence processing, so that the entire trajectory maintains consistency of relationships and continuity of action in the continuous window.

[0056] In a specific example, a dancer and a humanoid robot collaborate on a duet dance. The system has already obtained the robot's candidate motion trajectories for beats 84 to 96 based on the aforementioned steps. The prediction results of the human-robot collaboration relationship indicate that the dancer will move diagonally from the center of the stage to the front field before beat 88, and complete the upper limb extension movement at the strong beat position of beat 90. The robot needs to complete the lateral movement and turn while maintaining a responsive relationship.

[0057] The system first constructs a time-varying constraint rolling optimization model corresponding to the 84th to 96th beats based on a unified digital twin. The 84th to 88th beats are set as the first optimization sub-window, and the robot's lateral displacement reachable boundary, the relative distance boundary with the dancer, the alignment requirement of the strong beat of the 90th beat, and the risk boundary of the front intersection area are incorporated into the model.

[0058] Subsequently, continuous window prediction and evaluation were performed on the robot's candidate motion trajectory. It was found that although the original trajectory met the motion reachability constraint near beat 89, it had a relative lag in orientation switching compared to the dancer's advancing trend. At the same time, the turning amplitude around beat 90 was too large, resulting in the risk boundary of the intersection area with the foreground too close. Based on this, the system performed iterative correction on the local path segment of the trajectory from beat 88 to beat 91, shifting the starting time of the turn to the second half of beat 88, and converging the lateral turning amplitude, so that the robot formed an orientation relationship corresponding to the dancer's upper limb extension movement at the strong beat position of beat 90.

[0059] Subsequently, the system continues to perform scrolling optimization on two subsequent optimization sub-windows, namely, the 89th to 93rd and the 92nd to 96th frames, and finally generates a target motion trajectory that matches the prediction result of the human-machine collaboration relationship, and sends the target motion trajectory to the subsequent control module.

[0060] Through the above implementation methods, continuous rolling optimization of robot candidate motion trajectories under time-varying constraints can be performed within the target continuous performance window based on a unified digital twin. This improves the matching degree between the trajectory and the prediction results of human-machine collaboration, enhances the accuracy of rhythm connection and the adaptability of interaction boundaries, and improves the continuity, stability and real-time executability of the target motion trajectory.

[0061] In some embodiments, the physical performance feedback is compared with the simulation output of the unified digital twin to identify discrepancies. Based on the discrepancy results, the timing parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin are updated, including: The physical performance feedback and simulation output are aligned with the state, spatiotemporal correlation mapping and difference analysis under a unified reference framework to generate the virtual-real deviation representation of the corresponding target continuous performance window; Based on the virtual-real deviation characterization, the deviation propagation relationship and parameter perturbation direction corresponding to the unified digital twin are determined, and parameter calibration conditions are generated. Based on the parameter calibration conditions, parameter sensitivity correlation analysis and constraint boundary adaptive inference are performed using a unified digital twin to generate parameter update results. Based on the parameter update results, the temporal parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin are updated collaboratively.

[0062] Specifically, the aforementioned steps have generated the target motion trajectory and driven the robot to perform corresponding dance movements. However, during the actual performance, factors such as temporary beat shifts by the dancers, robot execution lag, changes in ground friction, close-range interaction disturbances, and scene occlusion changes will gradually cause deviations between the physical performance process and the prediction process in the unified digital twin. If these deviations are not continuously identified and updated in conjunction with each other, the virtual state upon which subsequent collaborative relationship prediction, candidate trajectory generation, and rolling optimization are based will gradually deviate from the actual performance state. Therefore, this embodiment constructs a virtual-real deviation representation, parameter calibration conditions, and parameter update results, enabling the unified digital twin to synchronously correct itself with the feedback from the actual performance during continuous performance.

[0063] In the specific implementation process, the physical performance feedback and simulation output are first aligned, spatiotemporally correlated, mapped, and analyzed under a unified reference framework. Specifically, the dancer's key poses, the robot's actual pose, changes in interaction distance, and the current performance beat information in the physical performance feedback are mapped to the simulation output of the unified digital twin at the corresponding moment onto the same stage reference coordinate system and the same performance phase axis, and alignment processing is performed on a target continuous performance window basis.

[0064] Subsequently, the differences in human motion state, robot motion state, relative pose, and performance phase at the same beat point are jointly analyzed to form a virtual-real deviation characterization for the corresponding target continuous performance window. This virtual-real deviation characterization is not a simple single-point error, but a state description reflecting the direction of deviation change, cumulative trend, and mutual coupling relationship within the continuous window.

[0065] Furthermore, based on this virtual-real deviation characterization, the deviation propagation relationship and parameter perturbation direction corresponding to the unified digital twin are determined, and parameter calibration conditions are generated. Here, the deviation propagation relationship is used to characterize the influence path of a certain type of deviation in the unified digital twin to subsequent time-series parameters, motion parameters, and constraint boundaries, while the parameter perturbation direction is used to characterize the incremental correction, decremental correction, or boundary contraction correction that should be performed on the relevant parameters.

[0066] Furthermore, after obtaining the parameter calibration conditions, parameter sensitivity correlation analysis and constraint boundary adaptive inference are performed using the unified digital twin to generate parameter update results. Specifically, based on the parameter calibration conditions, correlation sensitivity analysis is performed on the performance phase advancement parameters, robot motion response parameters, close interaction risk threshold, and local stage constraint boundaries in the unified digital twin to identify which type of parameter change mainly affects the current virtual-real deviation. Then, the update priority is determined based on the degree of influence of each parameter on the subsequent collaborative relationship prediction results and trajectory optimization results.

[0067] Simultaneously, by combining the deviation accumulation trend within the continuous performance window, adaptive inference is performed on the robot's lateral detour boundaries, close-range intersection boundaries, and action switching time limit boundaries to generate parameter update results. Finally, based on the parameter update results, the temporal parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin are collaboratively updated to ensure that the subsequent virtual simulation state and physical performance state are consistent again.

[0068] In a specific example, a dancer and a humanoid robot collaborate on a 96-second duet dance performance. Within the target continuous performance window from beat 88 to beat 96, the system detects that the dancer, due to a temporary increase in emotional expression, completes an upper limb extension movement half a beat before beat 90, while the robot's lateral displacement lags due to increased local friction on the ground. The system first performs state alignment and spatiotemporal correlation mapping with the dancer's actual key point trajectory, the robot's actual rotation angle, the real-time distance between the two, and the performance phase information from beat 88 to beat 96, collected on-site, and with the simulation output of a unified digital twin within the same window. The analysis yields a virtual-real deviation characterization, indicating that there are deviations in the performance phase forward shift, robot lateral displacement lag, and safety margin contraction deviation in the intersection area within the current window.

[0069] Subsequently, based on the virtual-real deviation characterization, the system determines the deviation propagation relationship and parameter perturbation direction, and generates parameter calibration conditions, including forward correction for the performance phase advancement parameters, incremental correction for the robot's lateral response parameters, and contraction correction for the intersection area risk threshold. Next, the system invokes the unified digital twin to perform parameter sensitivity correlation analysis and adaptive constraint boundary inference, determining the phase advancement parameters and lateral response parameters as highly sensitive parameters for the current window, and adaptively converges the intersection boundary from the 90th to the 94th beat, finally generating parameter update results and completing the collaborative update of the unified digital twin accordingly. The updated unified digital twin can more accurately reflect the real performance state of dancers unfolding prematurely, robot displacement lagging, and increased risk in the intersection area, providing a reliable basis for subsequent collaborative relationship prediction and target motion trajectory update.

[0070] Through the above implementation methods, the differences between physical performance feedback and simulation output can be continuously analyzed under a unified reference framework. Based on the virtual-real deviation characterization, the timing parameters, motion parameters, risk thresholds and constraint boundaries in the unified digital twin can be updated in a linked manner, thereby improving the consistency between virtual and real states, enhancing the accuracy of parameter correction, and providing more stable dynamic support for subsequent human-computer collaborative dance motion optimization.

[0071] In some embodiments, based on the updated unified digital twin and the target motion trajectory, real-time collaborative dance motion control results of the robot during the target performance are generated and output, including: Based on the updated unified digital twin and the target motion trajectory, a hierarchical collaborative motion control model corresponding to the target continuous performance window is constructed, and corresponding human-machine collaborative control parameters are generated. Based on the human-machine collaborative control parameters, the collaborative control variables corresponding to the robot actuator are solved for correlation and time sequence allocation, and real-time motion control commands corresponding to the target motion trajectory are generated. Real-time motion control commands are sent to the robot control terminal to drive the robot to output corresponding real-time collaborative dance motion control results during the target performance.

[0072] Specifically, the aforementioned steps have already updated the timing parameters, motion parameters, risk thresholds, and constraint boundaries in the unified digital twin, and further obtained the target motion trajectory that matches the current human-machine collaboration relationship. Therefore, this embodiment does not directly output the target motion trajectory as the robot's underlying execution command. Instead, under the constraints of the updated unified digital twin, the target motion trajectory is first converted into human-machine collaboration control parameters that adapt to the target continuous performance window. Then, based on the human-machine collaboration control parameters, the collaboration control variables corresponding to the robot's actuator are correlated and time-series allocated, and finally, real-time motion control commands that can be continuously output during the actual performance are generated.

[0073] The human-robot collaboration control parameters here are used to characterize the relative orientation maintenance requirements, beat response requirements, posture switching requirements, and interaction boundary maintenance requirements that the robot should meet within the current continuous performance window. The real-time motion control commands are used to drive the robot to complete the collaborative performance with the dancer without deviating from the target motion trajectory.

[0074] In the specific implementation process, a hierarchical collaborative motion control model corresponding to the target continuous performance window is first constructed based on the updated unified digital twin and the target motion trajectory. Specifically, in the unified digital twin, taking the human-machine collaboration state within the current performance window and the target motion trajectory as input, upper-level control constraints for tracking collaborative relationships, middle-level coordination and solution relationships for continuous pose transitions, and lower-level drive mapping relationships for actuator responses are established. This enables the robot to not only meet the requirements of path position and posture changes when executing the target motion trajectory, but also simultaneously meet the constraints of beat landing point, relative pose maintenance, and interaction boundary.

[0075] Subsequently, based on the hierarchical collaborative motion control model, and considering the distribution of strong beat nodes, action transition intervals, and risk boundary changes within the target continuous performance window, corresponding human-machine collaborative control parameters are generated. These parameters characterize the target tracking intensity, attitude switching time limit, local convergence boundary, and risk suppression magnitude of each control level within the current continuous performance window, thus providing a unified constraint basis for subsequent control variable solutions.

[0076] Furthermore, after obtaining the human-machine collaborative control parameters, the collaborative control variables corresponding to the robot actuators are solved and time-series allocated. Specifically, the human-machine collaborative control parameters can be mapped to the robot's torso rotation, limb swing, gait switching, and end-effector posture adjustment processes. The displacement, velocity, and posture correction variables of each actuator within the continuous performance window are jointly solved, and the effective time, duration, and switching boundaries of each control variable are time-series allocated according to the performance phase progression sequence, so that each actuator forms an orderly and connected real-time motion control command during the continuous beat-point progression process.

[0077] Subsequently, real-time motion control commands are sent to the robot control unit, which executes corresponding posture adjustments, path following, and beat response actions according to the current control cycle. This results in real-time collaborative dance motion control that corresponds to the dancer's movements during the target performance. The output process here is not a one-time trajectory playback, but rather involves continuous verification and correction of the control commands within each control cycle, using an updated unified digital twin. This allows the robot to dynamically collaborate with the dancer's movements and the changing rhythm of the performance.

[0078] In a specific example, a dancer and a humanoid robot perform a duet. Within the target continuous performance window from beat 96 to beat 108, the updated unified digital twin indicates that the dancer will perform a diagonal advance from the center to the front field before beat 100, and complete upper limb extension and torso turning movements at the strong beat position of beat 102. The target motion trajectory requires the robot to complete a lateral pre-displacement before beat 99, complete a synchronized turn on beat 102, and enter a separated response state after beat 104, while maintaining a relative responsive relationship. The system first constructs a hierarchical collaborative motion control model based on the updated unified digital twin and the target motion trajectory, generating corresponding human-robot collaborative control parameters. These parameters enable the robot to increase orientation tracking intensity, tighten close interaction boundaries, and shorten local posture switching time during beats 100 to 103.

[0079] Subsequently, the system performs correlation solving and timing allocation on the collaborative control variables corresponding to the robot's actuators. The trunk turning variables, gait lateral shift variables, and upper limb extension variables are allocated to the continuous control cycle according to the beat sequence, and corresponding real-time motion control commands are generated and sent to the robot's control terminal. Based on the real-time motion control commands, the robot completes a lateral pre-displacement on beat 99, achieves a synchronized turning response with the dancer at the strong beat position on beat 102, and exits the convergence area along a separation trajectory after beat 104. Finally, it outputs a real-time collaborative dance motion control result consistent with the target motion trajectory.

[0080] Through the above implementation methods, a hierarchical collaborative motion control model for continuous performance windows can be constructed based on the updated unified digital twin and the target motion trajectory. The target trajectory can be stably converted into motion control commands that can be executed in real time, improving the consistency between the robot control output and the predicted results of the collaborative relationship, and enhancing the beat matching ability, posture connection ability and real-time collaborative adaptation ability of the control process.

[0081] In some embodiments, based on the updated unified digital twin and the target motion trajectory, a hierarchical collaborative motion control model corresponding to the target continuous performance window is constructed, and corresponding human-machine collaborative control parameters are generated, including: Based on the updated unified digital twin, the target motion trajectory, and the human-machine collaboration state corresponding to the target continuous performance window, a collaborative control association between different control levels is established to generate a hierarchical collaborative motion control model. Based on the hierarchical cooperative motion control model, the target motion trajectory is decomposed into control variables, temporal coupling correlation and execution constraint consistency check are performed to generate parameter solution conditions. Based on the parameter solution conditions, determine the human-machine collaborative control parameters corresponding to the target continuous performance window.

[0082] Specifically, the aforementioned steps have completed the collaborative update of the unified digital twin and obtained the target motion trajectory that matches the prediction result of the human-machine collaboration relationship. Therefore, this embodiment does not directly issue control commands to the robot actuator based on the target motion trajectory. Instead, it first combines the updated unified digital twin and the human-machine collaboration state corresponding to the target continuous performance window to establish a collaborative control association between different control levels, so that the pose changes, beat progression relationships and interaction boundary requirements in the target motion trajectory can be mapped to the control layer in a parameterized manner.

[0083] The hierarchical collaborative motion control model here is used to divide the collaborative motion task within the target continuous performance window into control problems of different levels. The human-machine collaborative control parameters are used to characterize the tracking intensity, phase correction, attitude switching boundary and interaction suppression boundary of each control level under the current continuous performance window.

[0084] In the specific implementation process, firstly, based on the updated unified digital twin, the target motion trajectory, and the human-machine collaborative state corresponding to the target continuous performance window, a collaborative control association between different control levels is established to generate a hierarchical collaborative motion control model. Specifically, in the updated unified digital twin, based on the human-machine relative pose trend within the target continuous performance window, the performance phase connection state, and the robot execution boundary state, the control task is divided into a collaborative relationship maintenance layer, a motion trajectory coordination layer, and an actuator response layer.

[0085] In some examples, the collaboration relationship maintenance layer describes the requirements that the robot should meet within the current continuous performance window, including maintaining the human-robot relative orientation, converging relative spacing, and responding to beats; the motion trajectory coordination layer describes the pose transition, path connection, and action switching relationships of the target motion trajectory during continuous beat point advancement; and the actuator response layer describes the dynamic response capabilities and boundary constraints of each actuator of the robot to the upper-level collaborative control target. Through the above hierarchical associations, a hierarchical collaborative motion control model corresponding to the target continuous performance window is generated.

[0086] Furthermore, after generating the hierarchical collaborative motion control model, based on the model, control variable decomposition, temporal coupling correlation, and execution constraint consistency verification are performed on the target motion trajectory to generate parameter solution conditions. Specifically, the path displacement changes, torso turning changes, limb extension changes, and gait switching changes in the target motion trajectory can be decomposed into control variables corresponding to different control levels. Then, combined with the beat progression sequence, strong beat node position, and action connection time period in the target continuous performance window, the temporal coupling correlation of the sequential dependency and synchronous effect relationship between each control variable is performed.

[0087] Furthermore, based on the updated unified digital twin, the consistency of execution constraints is checked on each decomposed control variable to determine whether it meets the requirements of the current robot motion boundary, near-field interaction boundary, and beat phase boundary. For combinations of control variables with local conflicts or unreasonable activation sequences, their switching boundaries and activation periods are converged and corrected to generate parameter solution conditions. The parameter solution conditions are used to limit the solvable range of each control variable and its coupling relationship within the current target continuous performance window.

[0088] Based on the parameter solution conditions, the human-machine collaborative control parameters corresponding to the target continuous performance window are determined. Specifically, based on the parameter solution conditions, the trajectory tracking gain, phase correction weight, attitude transition time limit, and risk suppression amplitude at different control levels can be jointly solved to obtain human-machine collaborative control parameters that meet the requirements of the current human-machine collaborative state and the target motion trajectory. These human-machine collaborative control parameters then serve as the input basis for the generation of subsequent real-time motion control commands.

[0089] In a specific example, a dancer and a humanoid robot perform a collaborative dance. Within the target continuous performance window from beat 108 to beat 120, the updated unified digital twin shows that the dancer will perform a diagonal unfolding movement from the center outwards before beat 112, and complete the connecting movement of torso turning and upper limb lifting on beat 114. The target motion trajectory requires the robot to complete a lateral pre-displacement before beat 111, form a synchronous response orientation with the dancer on beat 114, and enter a retreating separation state after beat 116.

[0090] The system first establishes a hierarchical collaborative motion control model based on the updated unified digital twin, the target motion trajectory, and the current human-machine collaboration state. It maps the human-machine relative orientation maintenance and spacing control to the collaborative relationship maintenance layer, maps lateral displacement, steering switching, and upper limb extension to the motion trajectory coordination layer, and maps gait drive, joint response, and posture correction to the actuator response layer.

[0091] Subsequently, the system performs control variable decomposition and temporal coupling correlation on the target motion trajectory. It determines that the lateral displacement variable and the trunk turning variable have a synchronous coupling relationship during beats 111 to 114, and that the upper limb extension variable and the retreat switching variable have a sequential dependency relationship during beats 114 to 116. Using the updated unified digital twin, the system performs constraint consistency checks on the above control variables, finding that the original retreat switching time was too early, potentially leading to insufficient responsiveness with the dancer's extension movements. Therefore, the retreat switching boundary is postponed to the second half of beat 116, thus generating parameter solution conditions. Finally, the system solves for the human-machine collaborative control parameters corresponding to the continuous performance window of the target based on the parameter solution conditions, and uses these parameters for subsequent real-time motion control command generation.

[0092] Through the above implementation methods, multi-level collaborative associations can be established based on the updated unified digital twin, the target motion trajectory, and the current human-machine collaboration status. Reliable human-machine collaboration control parameters can be generated through control variable decomposition, temporal coupling association, and execution constraint consistency verification, thereby improving the pertinence and consistency of control parameter solutions and enhancing the robot's rhythm response capability, posture connection capability, and human-machine collaboration adaptation capability during continuous performance.

[0093] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0094] Figure 2 This is a schematic diagram of the structure of the human-computer collaborative dance motion optimization system based on digital twins provided in an embodiment of this application. Figure 2 As shown, the system includes: The data acquisition module 201 is used to collect human motion data, robot status data, environmental perception data and performance timing data in the performance scene, and to construct a unified digital twin corresponding to human, machine, scene and time; The parsing module 202 is used to perform associated semantic parsing on human motion sequences, performance timing, preset choreography relationships and human-machine relative pose relationships based on a unified digital twin, and generate prediction results of human-machine collaboration relationships corresponding to future target continuous performance windows; The generation module 203 is used to generate candidate motion trajectories of the robot in the unified digital twin based on the prediction results of human-machine collaboration relationship, and to perform rolling optimization based on motion reachability constraints, collaboration relationship constraints, beat phase constraints and interaction risk constraints to obtain the target motion trajectory; The update module 204 is used to compare the deviation between the physical performance feedback and the simulation output of the unified digital twin, and update the timing parameters, motion parameters, risk thresholds and constraint boundaries of the unified digital twin according to the deviation results. The output module 205 is used to generate and output the real-time collaborative dance motion control results of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory.

[0095] In some embodiments, Figure 2 The acquisition module 201 performs unified time base alignment, cross-modal space registration, and virtual-real parameter calibration on human motion data, robot state data, environmental perception data, and performance timing data to generate corresponding human action states, robot motion states, scene constraint states, and performance phase states. Based on the human action states, robot motion states, scene constraint states, and performance phase states, it establishes a human-machine relative relationship mapping and scene timing association relationship to generate a unified digital twin corresponding to the physical performance process.

[0096] In some embodiments, Figure 2 The parsing module 202 performs temporal association encoding and relational state alignment on the human motion sequence, performance timing, preset choreography relationship and human-machine relative pose relationship to generate a collaborative semantic representation corresponding to the target continuous performance window; based on the collaborative semantic representation, combined with the unified digital twin, it performs collaborative relationship evolution inference to generate the prediction result of human-machine collaborative relationship corresponding to the future target continuous performance window.

[0097] In some embodiments, Figure 2 The parsing module 202, based on the collaborative semantic representation, combines the unified digital twin to perform temporal unfolding and relational recursion on the current human-machine collaboration state, generating a collaborative relationship evolution sequence corresponding to the target continuous performance window; based on the collaborative relationship evolution sequence, it combines the unified digital twin to perform consistency correction on the human-machine relative pose trend, performance phase connection state and interaction constraint state, generating human-machine collaboration relationship prediction results.

[0098] In some embodiments, Figure 2 The generation module 203, based on the prediction results of human-machine collaboration relationship and combined with the unified digital twin, performs correlation mapping on the relative state of human-machine relationship, performance phase connection state and robot motion constraint state corresponding to the target continuous performance window, and generates candidate trajectory generation conditions; according to the candidate trajectory generation conditions, it calls the unified digital twin to perform temporal unfolding and boundary deduction on the robot motion evolution process, and generates the corresponding trajectory evolution results; based on the trajectory evolution results, it determines the robot candidate motion trajectory that matches the prediction results of human-machine collaboration relationship.

[0099] In some embodiments, Figure 2 The generation module 203 constructs a time-varying constraint rolling optimization model corresponding to the target continuous performance window based on the robot candidate motion trajectory, combined with the prediction results of human-machine collaboration relationship and the unified digital twin; based on the time-varying constraint rolling optimization model, it performs continuous window prediction evaluation and iterative correction on the robot candidate motion trajectory to generate trajectory optimization results; and based on the trajectory optimization results, it determines the target motion trajectory that matches the prediction results of human-machine collaboration relationship.

[0100] In some embodiments, Figure 2The update module 204 performs state alignment, spatiotemporal correlation mapping, and difference analysis under a unified reference framework on the physical performance feedback and simulation output, generating a virtual-real deviation representation for the corresponding target continuous performance window; based on the virtual-real deviation representation, it determines the deviation propagation relationship and parameter perturbation direction corresponding to the unified digital twin, and generates parameter calibration conditions; according to the parameter calibration conditions, it performs parameter sensitivity correlation analysis and adaptive inference of constraint boundaries in conjunction with the unified digital twin, generating parameter update results; based on the parameter update results, it collaboratively updates the temporal parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin.

[0101] In some embodiments, Figure 2 The output module 205 constructs a hierarchical collaborative motion control model corresponding to the target continuous performance window based on the updated unified digital twin and the target motion trajectory, and generates corresponding human-machine collaborative control parameters. According to the human-machine collaborative control parameters, it performs correlation solving and timing allocation on the collaborative control variables corresponding to the robot actuator, and generates real-time motion control commands corresponding to the target motion trajectory. The real-time motion control commands are sent to the robot control terminal to drive the robot to output the corresponding real-time collaborative dance motion control results during the target performance.

[0102] In some embodiments, Figure 2 The output module 205 establishes collaborative control associations between different control levels based on the updated unified digital twin, the target motion trajectory, and the human-machine collaboration state corresponding to the target continuous performance window, generating a hierarchical collaborative motion control model. Based on the hierarchical collaborative motion control model, it performs control variable decomposition, temporal coupling association, and execution constraint consistency verification on the target motion trajectory, generating parameter solution conditions. According to the parameter solution conditions, it determines the human-machine collaborative control parameters corresponding to the target continuous performance window.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various system embodiments described above.

[0105] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0106] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0107] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0110] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for human-robot collaboration dance motion optimization based on digital twin, characterized in that, include: Collect human motion data, robot status data, environmental perception data, and performance timing data in the performance scene to construct a unified digital twin corresponding to people, machines, scene, and time; Based on the unified digital twin, the associated semantic parsing is performed on the human motion sequence, performance sequence, preset choreography relationship and human-machine relative pose relationship to generate the prediction result of human-machine collaboration relationship corresponding to the future target continuous performance window; Based on the predicted results of human-machine collaboration, candidate robot motion trajectories are generated in the unified digital twin, and rolling optimization is performed based on motion reachability constraints, collaboration relationship constraints, beat phase constraints, and interaction risk constraints to obtain the target motion trajectory. The physical performance feedback is compared with the simulation output of the unified digital twin, and the timing parameters, motion parameters, risk thresholds and constraint boundaries of the unified digital twin are updated according to the deviation results. Based on the updated unified digital twin and the target motion trajectory, the real-time collaborative dance motion control results of the robot during the target performance are generated and output.

2. The method of claim 1, wherein, The process involves collecting human motion data, robot state data, environmental perception data, and performance timing data from the performance scene to construct a unified digital twin corresponding to the human, robot, scene, and time, including: The human motion data, robot state data, environmental perception data, and performance timing data are subjected to unified time base alignment, cross-modal space registration, and virtual-real parameter calibration to generate corresponding human action states, robot motion states, scene constraint states, and performance phase states. Based on the human body's motion state, robot's motion state, scene constraint state, and performance phase state, a mapping of the human-machine relative relationship and a scene temporal correlation relationship are established to generate a unified digital twin corresponding to the physical performance process.

3. The method of claim 1, wherein, Based on the unified digital twin, the process involves performing semantic analysis on the human motion sequence, performance timing, preset choreography relationships, and human-machine relative pose relationships to generate prediction results of human-machine collaboration relationships corresponding to future target continuous performance windows, including: Perform temporal association encoding and relational state alignment on the human motion sequence, performance timing, preset choreography relationship and human-machine relative pose relationship to generate a collaborative semantic representation corresponding to the target continuous performance window; Based on the aforementioned collaborative semantic representation, and combined with the unified digital twin, collaborative relationship evolution inference is performed to generate prediction results of human-machine collaborative relationships corresponding to the future target continuous performance window.

4. The method of claim 3, wherein, The step of performing collaborative relationship evolution inference based on the collaborative semantic representation and the unified digital twin to generate a human-machine collaborative relationship prediction result corresponding to the future target continuous performance window includes: Based on the aforementioned collaborative semantic representation, and combined with the unified digital twin, the current human-machine collaboration state is subjected to temporal unfolding and relational recursion to generate a collaborative relationship evolution sequence corresponding to the target continuous performance window; Based on the evolution sequence of the collaborative relationship, and combined with the unified digital twin, consistency correction is performed on the human-machine relative pose trend, performance phase connection state, and interaction constraint state to generate the prediction result of the human-machine collaborative relationship.

5. The method of claim 1, wherein, The step of generating candidate robot motion trajectories in the unified digital twin based on the human-machine collaboration relationship prediction results includes: Based on the predicted results of human-machine collaboration, and combined with the unified digital twin, an association mapping is performed on the relative state of human-machine relationship, the state of performance phase connection, and the state of robot motion constraint corresponding to the target continuous performance window to generate candidate trajectory generation conditions. Based on the candidate trajectory generation conditions, the unified digital twin is invoked to perform temporal unfolding and boundary deduction of the robot's motion evolution process, generating the corresponding trajectory evolution results; Based on the trajectory evolution results, candidate robot motion trajectories that are compatible with the human-machine collaboration relationship prediction results are determined.

6. The method according to claim 1, characterized in that, The rolling optimization based on motion reachability constraints, cooperative relationship constraints, beat phase constraints, and interaction risk constraints yields the target motion trajectory, including: Based on the robot's candidate motion trajectory, combined with the prediction results of the human-machine collaboration relationship and the unified digital twin, a time-varying constraint rolling optimization model corresponding to the target continuous performance window is constructed. Based on the time-varying constraint rolling optimization model, continuous window prediction evaluation and iterative correction are performed on the candidate motion trajectory of the robot to generate trajectory optimization results. Based on the trajectory optimization results, a target motion trajectory that matches the human-machine collaboration relationship prediction results is determined.

7. The method of claim 1, wherein, The step of comparing the physical performance feedback with the simulation output of the unified digital twin, and updating the timing parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin based on the deviation results, includes: The physical performance feedback and the simulation output are aligned with a unified reference framework, and spatiotemporal correlation mapping and difference analysis are performed to generate a virtual-real deviation representation for the corresponding target continuous performance window. Based on the virtual-real deviation characterization, the deviation propagation relationship and parameter perturbation direction corresponding to the unified digital twin are determined, and parameter calibration conditions are generated. Based on the parameter calibration conditions, parameter sensitivity correlation analysis and constraint boundary adaptive inference are performed using the unified digital twin to generate parameter update results; Based on the parameter update results, the temporal parameters, motion parameters, risk thresholds, and constraint boundaries of the unified digital twin are updated collaboratively.

8. The method of claim 1, wherein, The step of generating and outputting the real-time collaborative dance motion control results of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory includes: Based on the updated unified digital twin and the target motion trajectory, a hierarchical collaborative motion control model corresponding to the target continuous performance window is constructed, and corresponding human-machine collaborative control parameters are generated. Based on the human-machine collaborative control parameters, the collaborative control variables corresponding to the robot actuator are correlated and time-series allocated to generate real-time motion control commands corresponding to the target motion trajectory. The real-time motion control commands are sent to the robot control terminal to drive the robot to output corresponding real-time collaborative dance motion control results during the target performance.

9. The method of claim 8, wherein, Based on the updated unified digital twin and the target motion trajectory, a hierarchical collaborative motion control model corresponding to the target continuous performance window is constructed, and corresponding human-machine collaborative control parameters are generated, including: Based on the updated unified digital twin, the target motion trajectory, and the human-machine collaboration state corresponding to the target continuous performance window, a collaborative control association between different control levels is established to generate a hierarchical collaborative motion control model. Based on the hierarchical cooperative motion control model, control variable decomposition, temporal coupling correlation and execution constraint consistency verification are performed on the target motion trajectory to generate parameter solution conditions. Based on the parameter solution conditions, determine the human-machine collaboration control parameters corresponding to the target continuous performance window.

10. A human-machine collaboration dance motion optimization system based on digital twin, characterized in that, include: The data acquisition module is used to collect human motion data, robot status data, environmental perception data, and performance timing data in the performance scene, and to build a unified digital twin corresponding to people, machines, scene, and time. The parsing module is used to perform associated semantic parsing on the human motion sequence, performance sequence, preset choreography relationship and human-machine relative pose relationship based on the unified digital twin, and generate the human-machine collaboration relationship prediction result corresponding to the future target continuous performance window; The generation module is used to generate candidate robot motion trajectories in the unified digital twin based on the prediction results of the human-machine collaboration relationship, and to perform rolling optimization based on motion reachability constraints, collaboration relationship constraints, beat phase constraints and interaction risk constraints to obtain the target motion trajectory. The update module is used to compare the deviation between the physical performance feedback and the simulation output of the unified digital twin, and update the timing parameters, motion parameters, risk thresholds and constraint boundaries of the unified digital twin according to the deviation results. The output module is used to generate and output the real-time collaborative dance motion control results of the robot during the target performance process based on the updated unified digital twin and the target motion trajectory.

Images