Real-time driving method and system for music performance gesture based on time-space collaboration

Abstract

The present invention relates to the field of virtual reality technology, and specifically to a real-time driving method and system for music performance postures based on spatiotemporal collaboration. The method captures performer movements through multi-angle video streams, combines them with audio signals, and uses a lightweight convolutional neural network to extract key joint coordinates. A multi-scale time window is innovatively used to analyze historical frame data to extract short-range, medium-range, and long-range temporal features. A key joint importance calculation mechanism is introduced to assign higher weights to performance-related key joints. Two-way enhancement of spatiotemporal information is achieved through a spatiotemporal deep collaborative interaction mechanism. Prediction parameters are dynamically adjusted based on real-time system delay information, and future postures are actively predicted to compensate for system delays. The expressiveness of predicted postures is enhanced by combining music rhythm information. Natural and realistic virtual character driving data is generated through physical constraints and inverse kinematics processing. The present invention significantly reduces the perception of system delay and improves sound and image synchronization and performance expressiveness.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and specifically to a real-time driving method and system for music performance postures based on spatiotemporal collaboration, which is particularly suitable for application scenarios such as virtual idol live broadcasts, VR / AR concerts, digital human performances, and music education. Background Art

[0002] With the rapid development of virtual idols, digital human performances, and immersive music experiences, achieving high-precision, low-latency, real-time motion capture and virtual character driving has become a pressing technical challenge. Existing real-time motion capture systems are primarily divided into two categories: professional motion capture systems based on specialized equipment and standard camera capture systems based on computer vision.

[0003] While professional motion capture systems offer high accuracy, they are expensive and complex to deploy, and performers must wear special markers or sensors, limiting the naturalness of their performances. Computer vision-based systems, on the other hand, suffer from two major issues: high latency and insufficient accuracy. This latency stems primarily from three factors: processing delays caused by the complexity of pose estimation algorithms, network transmission delays caused by cloud-based processing, and the inherent delay between image acquisition and processing. In fast-paced musical performances, this accumulated latency can lead to audio and video desynchronization, stuttering, or drifting movements, severely disrupting both the audience's immersion and the performer's experience.

[0004] Existing technologies typically employ simple linear extrapolation or buffering strategies to address latency issues. However, these methods struggle to cope with the complex and varied movements of musical performances, often at the expense of accuracy or by adding additional latency. Furthermore, most existing methods focus on single-frame processing accuracy, neglecting temporal information and the rhythmic characteristics of music, making it difficult to fully capture the artistic expression of musical performances.

[0005] Therefore, there is an urgent need for a real-time driving method for music performance postures that can simultaneously solve the problems of high latency and insufficient precision, so as to improve the real-time, accuracy and expressiveness of virtual music performances. Summary of the Invention

[0006] The purpose of the present invention is to provide a real-time driving method and system for music performance posture based on spatiotemporal collaboration, which effectively solves the problems of high delay and insufficient precision existing in the existing technology by integrating innovative technologies such as lightweight edge computing, spatiotemporal deep collaborative processing and active delay compensation prediction.

[0007] The present invention proposes a real-time driving method for music performance posture based on spatiotemporal collaboration, comprising:

[0008] Obtain multi-angle video streams and audio signals;

[0009] Based on the multi-angle video stream, extracting coordinate data of the performer's key joints through a lightweight convolutional neural network;

[0010] Analyzing the coordinate data of the key joints in the historical frame includes:

[0011] Extract multi-scale time series features based on time windows of different lengths;

[0012] Calculating the importance weight of the key joint;

[0013] Through the spatiotemporal depth collaborative interaction mechanism, the multi-scale temporal features and the current frame spatial features are fused, and the importance weights of the key joints are applied to the fusion process to perform weighted enhancement on the key joint features related to the performance;

[0014] Obtaining acquisition delay information, processing delay information, and transmission delay information in the real-time driving process of music performance gestures, and adjusting prediction parameters based on the delay information;

[0015] Based on the fused features and adjusted prediction parameters, the key joint coordinate data at the future moment is predicted;

[0016] Combining the music rhythm information in the audio signal, the predicted key joint coordinate data is enhanced in terms of expressiveness;

[0017] The enhanced key joint coordinate data is transmitted to a rendering end; and at the rendering end, the enhanced key joint coordinate data is converted into virtual character driving data based on physical constraints and an inverse kinematics model.

[0018] Preferably, the extracting coordinate data of the performer's key joints by using a lightweight convolutional neural network specifically includes:

[0019] Extracting two-dimensional key joint coordinates from each frame of the multi-angle video stream;

[0020] A confidence score is assigned to each key joint coordinate; and based on the two-dimensional key joint coordinates and their confidence scores from multiple perspectives, more three-dimensional key joint coordinate data is generated through triangulation.

[0021] Preferably, the extracting multi-scale time series features based on time windows of different lengths specifically includes:

[0022] Set short-range time window, medium-range time window and long-range time window to capture the motion characteristics of different time scales respectively;

[0023] Extracting instantaneous change features from the key joint coordinate sequence within the short-range time window;

[0024] Extracting complete action unit features from the key joint coordinate sequence within the mid-term time window;

[0025] Extracting rhythmic pattern features from the key joint coordinate sequence within the long-range time window; and fusing the instantaneous change features, the complete action unit features, and the rhythmic pattern features to generate comprehensive multi-scale temporal features.

[0026] Preferably, calculating the importance weight of the key joint specifically includes:

[0027] Establish key joint connection topology based on the human skeletal structure;

[0028] Calculate the relative position relationship between key joints in the current frame;

[0029] Based on the relative position relationship, generating attention scores between key joints;

[0030] Identify the current instrument type and performance status; and enhance the attention scores of key joints related to the performance based on the instrument type and the performance status to generate final key joint importance weights.

[0031] Preferably, the fusing of the multi-scale temporal features and the current frame spatial features through the spatiotemporal depth collaborative interaction mechanism specifically includes:

[0032] Using the current frame spatial features as the query and the multi-scale temporal features as the key and value, we calculate the time-to-space cross attention and obtain the spatial features enhanced by temporal information.

[0033] Using multi-scale temporal features as queries and temporally enhanced spatial features as keys and values, we calculate the spatial-to-temporal cross-attention to obtain spatially enhanced temporal features.

[0034] The key joint importance weights are applied to the temporal features enhanced by the spatial information; and through multiple rounds of interactive iterations, the fusion of spatiotemporal information is further deepened to generate a unified spatiotemporal collaborative representation.

[0035] Preferably, the step of obtaining the acquisition delay information, processing delay information, and transmission delay information in the real-time driving process of the music performance posture, and adjusting the prediction parameters based on the delay information specifically includes:

[0036] Real-time monitoring of video acquisition delay information, processing delay information and network transmission delay information;

[0037] encoding the delay information into a conditional vector;

[0038] Based on different delay magnitudes, multiple prediction branches are set; and according to the current actual delay situation, the weight of each prediction branch is dynamically adjusted to generate prediction parameters that adapt to the current system state.

[0039] Preferably, said enhancing the expressiveness of said predicted key joint coordinate data by combining the music rhythm information in said audio signal specifically comprises:

[0040] extracting beat position, beat intensity and music emotion features from the audio signal;

[0041] Calculate the time relationship between the current moment and the most recent beat to obtain beat phase information;

[0042] determining a motion amplitude enhancement factor according to the beat phase information and the beat intensity;

[0043] Based on the music emotion characteristics, the motion speed and range parameters are adjusted; and the motion amplitude enhancement factor, the motion speed and the range parameters are applied to the predicted key joint coordinates to generate key joint coordinate data with enhanced expressiveness.

[0044] Preferably, the converting the enhanced key joint coordinate data into virtual character driving data based on physical constraints and inverse kinematics model specifically includes:

[0045] Apply key joint angle limits to ensure that key joint movements are consistent with human physiological structure;

[0046] Apply motion smoothness constraints to avoid changes in key joint positions between adjacent frames;

[0047] Handle self-collision situations to prevent the virtual character's limbs from penetrating each other; and through the inverse kinematics algorithm, convert the three-dimensional key joint coordinate data into the virtual character's bone rotation data for driving the virtual character model.

[0048] Preferably, transmitting the enhanced key joint coordinate data to the rendering end specifically includes:

[0049] Compressing and encoding the enhanced key joint coordinate data;

[0050] Add timestamp and delay compensation information;

[0051] The data packets are transmitted to the rendering end via a low-latency network protocol; and the received data packets are parsed at the rendering end to restore the key joint coordinate data.

[0052] The real-time driving system for music performance gestures based on spatiotemporal coordination includes:

[0053] Video and audio acquisition unit, used to obtain multi-angle video streams and audio signals;

[0054] A lightweight posture estimation unit, configured to extract coordinate data of key joints of the performer based on the multi-angle video stream using a lightweight convolutional neural network;

[0055] A multi-scale temporal feature extraction unit, configured to analyze the coordinate data of the key joints in the historical frames based on time windows of different lengths to extract multi-scale temporal features;

[0056] A key joint importance calculation unit, used to calculate the importance weight of the key joint;

[0057] a spatiotemporal collaborative processing unit for fusing the multi-scale temporal features and the current frame spatial features through a spatiotemporal deep collaborative interaction mechanism, and applying the importance weights of the key joints to the fusion process to perform weighted enhancement on the key joint features related to the performance;

[0058] A delay perception unit is used to obtain acquisition delay information, processing delay information and transmission delay information in the real-time driving process of the music performance posture, and adjust the prediction parameters based on the delay information;

[0059] The posture prediction unit is used to predict the key joint coordinate data at the future moment based on the fused features and adjusted prediction parameters;

[0060] a music-driven enhancement unit, configured to enhance the expressiveness of the predicted key joint coordinate data by combining the music rhythm information in the audio signal;

[0061] A data transmission unit is used to transmit the enhanced key joint coordinate data to the rendering end; and an inverse kinematics processing unit is used to convert the enhanced key joint coordinate data into virtual character driving data based on physical constraints and inverse kinematics models at the rendering end.

[0062] The beneficial effects of the present invention include:

[0063] 1. Significantly reduce system end-to-end latency: Achieve near real-time synchronization through lightweight edge deployment and active delay compensation prediction.

[0064] 2. Improve the positioning accuracy of key joints: Through the adaptive weighting mechanism of key joint importance, key performance-related joints are optimized, significantly improving the positioning accuracy and stability of these key joints.

[0065] 3. Enhance the expressiveness and naturalness of movements: Through a posture correction enhancement mechanism driven by music rhythm, the movement performance is adjusted in combination with the music beat and emotional characteristics, making the virtual character's performance more artistic and appealing.

[0066] 4. Improve system robustness and adaptability: Through multi-perspective fusion and spatiotemporal depth collaborative processing, the system's adaptability to occlusion, lighting changes, and rapid movements is enhanced.

[0067] 5. Reduce deployment threshold and cost: Compared with traditional professional motion capture systems, this invention uses ordinary cameras and edge computing devices, greatly reducing hardware costs and deployment complexity.

[0068] Overall, the present invention provides an efficient, accurate, and expressive real-time driving solution for fields such as virtual idols, digital human performances, and VR / AR music applications, and has important technical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of the real-time driving method for music performance gestures based on spatiotemporal collaboration of the present invention;

[0070] Figure 2 Schematic diagram of the structure of the multi-scale temporal feature extraction mechanism of the present invention;

[0071] Figure 3 This is a calculation flow chart of the key joint importance adaptive weighting mechanism of the present invention;

[0072] Figure 4 Schematic diagram of the structure of the spatiotemporal depth collaborative interaction mechanism of the present invention;

[0073] Figure 5 This is a workflow diagram of the dynamic delay-aware prediction and adjustment mechanism of the present invention;

[0074] Figure 6 A schematic diagram of the music rhythm-driven posture correction enhancement mechanism of the present invention;

[0075] Figure 7 This is an architectural diagram of the real-time driving system for music performance gestures based on spatiotemporal collaboration of the present invention. DETAILED DESCRIPTION

[0076] Please refer to Figure 1-Figure 7 The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0077] Reference Figure 1 The present invention provides a real-time driving method for music performance posture based on time-space collaboration, comprising the following steps:

[0078] Acquiring Multi-Angle Video Streams and Audio Signals: In a preferred embodiment of the present invention, the system captures the performer's multi-angle video streams and audio signals via the video and audio acquisition unit 1. Preferably, three to four high-frame-rate industrial cameras (60fps or higher, 1920×1080 resolution) are used to synchronously capture the performer's video images from different angles. Simultaneously, professional audio acquisition equipment is used to capture the performance audio (sampling rate 44.1kHz or 48kHz). Each frame of image and audio data is accurately timestamped to ensure synchronization of subsequent processing.

[0079] Furthermore, camera placement must ensure full coverage of the performance area, preferably at 45-degree intervals around the performer, 2-3 meters from the performer, and at chest level for optimal viewing angles. Camera placement also varies for different instrument types. For example, during a piano performance, at least one camera should be looking down at the keyboard to capture detailed finger movements.

[0080] Based on the multi-angle video stream, the coordinate data of the performer's key joints is extracted using a lightweight convolutional neural network: The present invention uses a lightweight pose estimation unit 2 to process the multi-angle video stream. In one embodiment, this unit is deployed on an edge computing device (such as the NVIDIA Jetson Xavier NX) and extracts the coordinates of key joints in real time using an optimized lightweight convolutional neural network.

[0081] In this invention, the performer's key joints refer to the human body joints that have a decisive influence on the accurate reproduction of the musical performance posture. These key joints vary depending on the instrument playing scene, and the system will intelligently identify and optimize them based on the instrument type and playing status. Specifically:

[0082] For the piano playing scene, the key joints mainly include:

[0083] Finger joints: These include the fingertips, interphalangeal joints, and metacarpophalangeal joints of each finger, totaling 20 joints (10 on each hand). These joints directly determine the accuracy and strength of keystrokes and are the most critical part of piano playing.

[0084] Wrist joints: There is one on each wrist, which controls the position and posture of the entire hand and affects the overall distribution of fingers on the keyboard.

[0085] Forearm joints: These include the elbow joint and forearm rotation point, which are crucial for arm posture and force transfer.

[0086] Shoulder joint: Controls a wide range of motion of the entire upper limb, especially important when playing passages across a large key range.

[0087] For the guitar playing scene, the key joints focus on:

[0088] Left hand finger joints: responsible for pressing the strings, requiring extremely high precision to accurately locate the fret position and avoid noise.

[0089] Right hand finger joints: responsible for plucking or playing strings, controlling tone and volume.

[0090] Wrist joint: The flexibility of the left wrist affects the smoothness of changing positions, while the right wrist affects the plucking technique.

[0091] Forearm joints: Especially the left forearm, which controls pressure and position on the fretboard.

[0092] For the violin playing scene, the key joints include:

[0093] Right hand fingers, wrist and elbow: control the movement of the bow and directly affect the tone and expression.

[0094] Left hand finger joints: responsible for pressing the strings and tuning the pitch, requiring extremely high precision.

[0095] Left shoulder and neck joint: Controls the stable support of the violin.

[0096] The present invention uses a key joint importance calculation unit so that the system can dynamically adjust the importance weight of each key joint according to the currently identified instrument type and playing status.

[0097] This differentiated key joint weighting mechanism enables the system to focus limited computing resources on the body parts that are most critical to musical performance, significantly improving the accuracy and expressiveness of the virtual character's playing movements.

[0098] Specifically, for each video frame from each perspective, a lightweight CNN model (based on architectures such as MobileNetV3 or EfficientNet) extracts the 2D coordinates of 17 to 25 key human joints. This model is preferably quantized (INT8) and has computational graph optimization, keeping single-frame processing time under 15 milliseconds (at 1080p), meeting real-time requirements. The model also outputs a confidence score for each key joint coordinate, which is used for subsequent multi-view fusion.

[0099] The system uses triangulation technology to fuse the extracted multi-view 2D key joint coordinates to generate more accurate 3D coordinate data. Preferably, the fusion process considers the confidence score of each key joint and assigns lower weights to low-confidence areas (such as occluded areas) to improve overall accuracy. The fusion algorithm can be expressed as:

[0100] ,

[0101] in: is the fused three-dimensional coordinate point, the dimension is A vector representing the x, y, and z coordinates in space; The number of cameras is usually 3 to 4; For the The projection matrix of the camera has the dimension , including the camera's internal and external parameters; is the two-dimensional coordinate point observed by the camera, with dimension A vector representing the x and y coordinates on the image plane; It is a weight factor based on confidence, ranging from 0 to 1. The larger the value, the higher the reliability of the key joint detection result corresponding to the perspective. Represents the parameters that minimize the objective function ; Represents the square of the Euclidean distance.

[0102] Analyze the key joint coordinate data in the historical frame:

[0103] Extract multi-scale time series features based on time windows of different lengths, refer to Figure 2 The multi-scale temporal feature extraction unit 3 of the present invention performs multi-scale analysis on the key joint coordinate data in the historical frame. Unlike the traditional single time window method, the present invention uses three time windows of different lengths to capture the motion characteristics at different time scales.

[0104] In a preferred embodiment, the system sets the following three time windows:

[0105] Short-range window: 5 frames (about 83 milliseconds @ 60fps), used to capture instantaneous changes (such as keystrokes, string plucking, and other fast actions);

[0106] Mid-range window: 15 frames (about 250 milliseconds @ 60fps), used to capture complete action units (such as chord changes, position changes, etc.);

[0107] Long-range window: 30 frames (approximately 500 milliseconds @ 60fps), used to capture rhythmic patterns and periodic features.

[0108] For short-range windows, the system uses a one-dimensional convolutional network to extract instantaneous change features. Preferably, multi-size convolution kernels (3 / 5 / 7 frames) are used for parallel processing to capture motion changes at different speeds. For key joint coordinate sequences (J is the number of key joints), short-range feature extraction can be expressed as:

[0109] ,

[0110] in: is a short-range feature with a dimension of The matrix represents the features extracted in the time dimension; is the key joint coordinate sequence within the short-range window, with dimension where 5 represents the time window length. Indicates the number of key joints, 3 represents the x, y, z three-dimensional coordinates of each key joint; is the set of convolution kernel sizes (e.g. ), represents the different convolution kernel sizes used in the time dimension; is the number of output channels (for example, 64), indicating the feature dimension; Conv1D represents a one-dimensional convolution operation, which extracts features along the time dimension.

[0111] For the mid-range window, the system uses a bidirectional gated recurrent unit network to extract the complete action unit features. , mid-range feature extraction can be expressed as:

[0112] ,

[0113] in: is a medium-range feature with a dimension of The matrix represents the features extracted in the time dimension; is the key joint coordinate sequence in the mid-range window, with dimension where 15 represents the time window length. Indicates the number of key joints, 3 represents the x, y, z three-dimensional coordinates of each key joint; is the hidden layer dimension (e.g. 128), which represents the state vector dimension of the GRU unit; is the number of network layers (for example, 2), indicating the number of stacked GRU layers; BiGRU represents a bidirectional gated recurrent unit network that can capture timing information from both the forward and backward directions, and 2H represents the concatenation of forward and backward features.

[0114] For long-range windows, the system uses a lightweight Transformer network to extract rhythmic pattern features. , long-range feature extraction can be expressed as:

[0115] ,

[0116] in: is a long-range feature, and its dimension is The matrix of is the output dimension of the Transformer model (e.g. 256); is the key joint coordinate sequence in the long-range window, with dimension where 30 represents the time window length. Indicates the number of key joints, 3 represents the x, y, z three-dimensional coordinates of each key joint; is the number of attention heads (e.g. 4), indicating the number of parallel processing units in the self-attention mechanism; is the number of network layers (for example, 2), indicating the number of stacked Transformer encoder layers; Transformer represents the Transformer network structure, which can effectively capture long-distance dependencies.

[0117] Finally, the system fuses the three scale features to generate comprehensive multi-scale temporal features:

[0118] ,

[0119] ,

[0120] in: It is the multi-scale feature after splicing, and its dimension depends on the dimension of each feature and the splicing method; It is the final feature after fusion by the fully connected layer, with a dimension of 256; Concat represents the tensor splicing operation, which splices features of different scales in the feature dimension; FC represents the fully connected layer operation, which is used to map the spliced ​​features to a unified dimensional space.

[0121] Reference Figure 3 The key joint importance calculation unit 4 of the present invention implements an adaptive weighting mechanism to assign higher importance to key joints related to performance. Unlike the traditional method of treating all key joints equally, this mechanism can significantly improve the accuracy and expressiveness of key joints.

[0122] In one embodiment, the system first establishes a topological diagram of key joint connections based on the human skeletal structure. Preferably, the physical connection relationship between key joints is defined based on human anatomical knowledge to form a skeletal topological diagram. ,in is a set of key joint points, Connect the edges to the skeleton.

[0123] Next, the system calculates the relative position relationship between the key joints in the current frame. and , its relative position vector can be expressed as:

[0124] ,

[0125] in: For key joints Relative to key joints The position vector of , represents the relative position in three-dimensional space; is the key joint coordinate matrix of the current frame, with dimensions of ,in is the number of key joints (usually 17 to 25), 3 represents the x, y, z three-dimensional coordinates of each key joint; Representation matrix Middle The coordinate vector of the key joints is , Representation matrix Middle The coordinate vector of the key joints is .

[0126] Based on the relative position relationship, the system calculates the degree of mutual influence between key joints through the attention mechanism. Specifically, a multi-layer perceptron is used to calculate the attention score:

[0127] ,

[0128] in: is the spatial attention matrix with dimension , represents the attention relationship between key joints, and the value of each element ranges from 0 to 1; Indicates key joints For key joints Attention score; For key joints Relative to key joints The position vector of ; MLP represents a multi-layer perceptron, which maps the relative position vector to the attention score. The typical structure is a two-layer fully connected network with a ReLU activation function in the middle; Softmax represents the softmax normalization function to ensure that for each key joint , which is connected to all key joints The sum of the attention scores is 1.

[0129] The system then identifies the current instrument type and playing status. Preferably, by analyzing key joint positions and movement patterns, combined with pre-set instrument templates, the system identifies the instrument type (e.g., piano, guitar, violin, etc.) and playing status (e.g., fast strumming, sustained key press, tremolo, etc.) currently being played.

[0130] Based on the recognition results, the system enhances the attention scores of key joints related to performance. For example, for piano playing, the weight enhancement factor of key finger joints can be set to 2.0, and the key wrist joints to 1.5; for guitar playing, the key right hand joints to 1.8, and the key left hand joints to 1.5. Specifically:

[0131] ,

[0132] in: is the key joint importance weight vector, with dimension , represents the importance weight of each key joint; Represents a vector Middle The weight value of each key joint; is the key joint in the spatial attention matrix For key joints Attention score; For the enhancement factor based on the instrument type and playing status, set a larger value for key joints (for example, 2.0 for finger key joints and 1.5 for wrist key joints), and set non-key joints to 1.0. Indicates that all A key joint summation operation.

[0133] ,

[0134] in: is the normalized key joint importance weight vector, with dimension , is the key joint importance weight vector before normalization, with dimension , Represents the simple sum of all key joint weight values, used as a normalization factor; is the total number of key joints; is the index of the key joint, ranging from 1 to This operation ensures that the sum of all key joint weights is 1, that is, .

[0135] This L1 normalization (normalization to 1) ensures that the weights can be interpreted as probability distributions or attention allocations, which is crucial for the subsequent spatiotemporal deep collaborative interaction process. For example, in a piano playing scenario, the normalized weight distribution might be: fingertips account for 40% of the total weight, interphalangeal joints account for 25%, wrist joints account for 20%, and other joints account for 15%. This allows the system to focus on the key joints that have the greatest impact on the performance.

[0136] Compared to L2 normalization (Euclidean norm normalization), L1 normalization is more suitable for the application scenarios of this invention because it maintains the relative importance of weights while ensuring that the sum is a fixed value, which facilitates subsequent weighting operations. In practical applications, this normalization method enables the system to adaptively adjust the focus point, maintaining accurate tracking of key joints even in complex and changing performances.

[0137] Through the spatial-temporal depth collaborative interaction mechanism, the multi-scale temporal features and the current frame spatial features are integrated:

[0138] Reference Figure 4 The spatiotemporal collaborative processing unit 5 of the present invention implements an innovative spatiotemporal deep collaborative interaction mechanism. Unlike traditional methods that treat time and space as independent dimensions, this mechanism achieves deep interaction and mutual enhancement of spatiotemporal information through bidirectional cross-attention.

[0139] In a preferred embodiment, the system first calculates the temporal-to-spatial cross-attention using the current frame spatial features as the query and the multi-scale temporal features as the key and value:

[0140] ,

[0141] ,

[0142] in: is the time-to-space attention output, with dimensions same; is the spatial feature of the current frame, with a dimension of ,in is the number of key joints, is the characteristic dimension of each key joint; is the fused multi-scale time series feature, with a dimension of ,in is the number of time steps, is the feature dimension; is the number of attention heads (e.g. 8), indicating the number of attention units computed in parallel; CrossAttention represents the cross-attention operation, the first parameter is the query, the second parameter is the key, the third parameter is the value, and the fourth parameter is the number of attention heads; is the enhanced spatial feature, the dimension and Same; + represents residual connection, that is, element-level addition operation.

[0143] The system then calculates the spatial-to-temporal cross attention using the multi-scale temporal features as queries and the enhanced spatial features as keys and values:

[0144] ,

[0145] ,

[0146] in: is the spatial-to-temporal attention output, with dimensions same; is the fused multi-scale time series feature, with a dimension of , is the enhanced spatial feature, the dimension is , is the number of attention heads (e.g. 8); is the enhanced time feature, dimension and Same; + represents residual connection, that is, element-level addition operation.

[0147] Next, the system applies key joint importance weights to the enhanced features:

[0148] ,

[0149] in: is the weighted time feature, dimension and same; is the enhanced time feature, the dimension is , is the key joint importance weight vector, with dimension , Represents an element-wise multiplication operation, which broadcasts the weight vector to the appropriate dimension and then performs element-wise multiplication.

[0150] To further deepen the integration of spatiotemporal information, the system performs multiple rounds of interactive iterations (preferably 2-3 rounds). Each round of iteration can be expressed as:

[0151] ,

[0152] ,

[0153] Among them: the first equation means using weighted temporal features to further enhance spatial features; the second equation means using the enhanced spatial features to further enhance weighted temporal features; CrossAttention means cross attention operation, and the parameters have the same meaning as before; + means residual connection, that is, element-level addition operation.

[0154] Finally, the system combines the enhanced temporal and spatial features to generate a unified spatiotemporal collaborative representation:

[0155] ,

[0156] in: For the spatial and temporal co-representation, the dimension depends on the splicing method; is the final enhanced spatial feature, the dimension is , The enhanced temporal and spatial features are concatenated in appropriate dimensions.

[0157] This mechanism's innovation lies in its bidirectional interaction and mutual reinforcement of temporal and spatial information, enabling the capture of more complex spatiotemporal dependencies. This approach is particularly well-suited to the rhythmic and expressive demands of musical performances. Experiments have shown that compared to traditional simple fusion methods, this mechanism significantly improves motion prediction accuracy (by an average of approximately 28%) and expressiveness.

[0158] Obtain processing delay information for each link in the system and adjust prediction parameters based on the delay information:

[0159] Reference Figure 5 The delay perception unit 6 of the present invention implements a dynamic delay perception prediction adjustment mechanism, which can adaptively adjust the prediction parameters according to the real-time status of the system, thereby solving the limitations of the traditional fixed delay compensation method.

[0160] In a preferred embodiment, the system monitors three types of key delay information in real time:

[0161] Video capture latency: the time from when the physical action occurs to when the image capture is completed (typically 8-12 milliseconds at 60fps).

[0162] Processing latency: the time from image acquisition to pose estimation and prediction processing (optimized to 15-25 milliseconds).

[0163] Network transmission latency information: Data transmission time from the edge device to the rendering end (preferably 5-15 milliseconds under wired network conditions).

[0164] The system encodes the monitored delay information into a condition vector:

[0165] ,

[0166] in: is the delay encoding vector, with dimension (e.g. 32); is the acquisition delay, in milliseconds; For processing delay, the unit is milliseconds; is the network delay in milliseconds; FC represents the fully connected layer operation, and the three delay values ​​are mapped to dimensional encoding vector; Represents a vector of three delay values ​​concatenated together, with dimensions is the encoding dimension (e.g. 32).

[0167] Based on different delay levels, the system sets up multiple prediction branches, each branch is optimized for a specific delay situation:

[0168] ,

[0169] ,

[0170] ,

[0171] in: 、 、 They are the prediction results under short / medium / long delay respectively, and the dimensions are , indicating the predicted 3D coordinates of key joints; It is a spatiotemporal collaborative representation, which contains the spatiotemporal feature information of the action; It provides delay-related conditional information for the delay encoding vector; PredHead represents the prediction head network, which is usually implemented as a multi-layer perceptron. The last parameter (50ms / 100ms / 150ms) represents the target delay time.

[0172] Based on the current actual delay, the system dynamically adjusts the weight of each prediction branch:

[0173] ,

[0174] ,

[0175] ,

[0176] ,

[0177] in: 、 、 are the original weights of short / medium / long delay prediction branches respectively; is the total delay ( ), in milliseconds; Indicates the absolute difference between the actual total delay and the short delay target (50ms); Indicates the absolute difference between the actual total delay and the medium delay target (100ms); Indicates the absolute difference between the actual total delay and the long delay target (150ms); Indicates taking 0 and The larger value in ensures that the weight is non-negative; Normalize represents the normalization operation to ensure that the sum of the three weights is 1; The normalized weight vector dimension is 3×1.

[0178] The final prediction result is the weighted sum of each branch:

[0179] ,

[0180] in: is the final prediction parameter, and its dimension is the same as the prediction results of each branch; 、 、 are the normalized short / medium / long delay prediction branch weights respectively; 、 、 are the prediction results under short / medium / long delay respectively; · represents the operation of scalar multiplication by vector / matrix.

[0181] The advantage of this mechanism is that it can dynamically adjust prediction parameters based on the real-time system status, adapting to different latency conditions and improving the system's robustness and adaptability. In practical applications, this mechanism can expand the adaptability range of the system's end-to-end latency to 40-160 milliseconds, significantly enhancing the system's applicability under different network and hardware conditions.

[0182] Based on the fused features and the adjusted prediction parameters, the key joint coordinates at the future moment are predicted:

[0183] The posture prediction unit 7 of the present invention generates predictions of key joint coordinates at future moments based on the spatiotemporal collaborative representation and the adjusted prediction parameters. This is the core link in solving the latency problem of the present invention, compensating for the inherent system latency by actively predicting the future posture.

[0184] In one embodiment, the prediction process can be expressed as:

[0185] ,

[0186] in: For the predicted future moment The key joint coordinates of ,in is the number of key joints, 3 represents the x, y, z three-dimensional coordinates of each key joint; For the prediction function (preferably implemented as a multilayer perceptron or temporal convolutional network), the spatiotemporal co-representation, delay encoding, and prediction parameters are mapped to future key joint coordinates; It is a spatiotemporal collaborative representation, which contains the spatiotemporal feature information of the action; Providing delay-related condition information for the delay encoding vector; It is the final prediction parameter, which controls the specific behavior of the prediction; The prediction time window is in milliseconds and is dynamically adjusted to the current total delay of the system.

[0187] Preferably, the prediction time window Dynamically adjust to the current total system latency, with a slight margin (e.g., a safety factor of 1.2) to ensure full coverage of the latency. In practical applications, the prediction time window is typically in the range of 50-80 milliseconds.

[0188] The innovation of this prediction mechanism lies in that, unlike traditional methods that passively accept delays, this invention proactively compensates for delays by predicting future gestures, addressing real-time performance issues at the source. Experiments have shown that in fast-paced music performances (over 160 beats per minute), this mechanism can reduce perceived delay by approximately 65%, significantly improving audio-visual synchronization.

[0189] Combined with the music rhythm information in the audio signal, the predicted key joint coordinate data is enhanced in expressiveness:

[0190] Reference Figure 6 The music-driven enhancement unit 8 of the present invention implements an innovative music rhythm-driven posture correction enhancement mechanism, which integrates audio information into the posture prediction process and enhances the artistic expression of virtual performances.

[0191] In a preferred embodiment, the system first extracts three types of key features from the audio signal:

[0192] Beat location: Identify the timing of music beats through time domain energy analysis and spectrum analysis;

[0193] Beat intensity: quantifies the strength of the beat, ranging from 0 to 1;

[0194] Musical emotions: Identify musical emotions (such as excitement, calmness, etc.) through spectrum characteristics and timbre analysis.

[0195] The system calculates the time relationship between the current moment and the most recent beat and obtains the beat phase information:

[0196] ,

[0197] in: The beat phase ranges from 0 to 1, indicating the relative position of the current moment in the beat cycle. The current time, in milliseconds; The most recent beat time point, in milliseconds; Indicates the minimum time difference between the current moment and all beat points; The average beat interval in milliseconds, usually calculated from the BPM (beats per minute) of the music.

[0198] Based on the beat phase information and beat intensity, the system determines the motion amplitude enhancement factor:

[0199] ,

[0200] in: is the action amplitude enhancement factor, which is a dimensionless coefficient; is the beat intensity, ranging from 0 to 1, the larger the value, the stronger the beat; Indicates that the current moment is close to the beat point (within the first 10% of the beat cycle); It means that the movement amplitude is enhanced near the beat point, and the enhancement amount is proportional to the beat intensity; 1.0 means that the original movement amplitude remains unchanged near the non-beat point.

[0201] Based on the emotional characteristics of the music, the system adjusts the speed and range parameters of the action:

[0202] ,

[0203] ,

[0204] in: is the action speed factor, which is a dimensionless coefficient; is the action range factor, which is a dimensionless coefficient; emotion is the recognized music emotion type, "excited" represents the excited / excited emotion type; "calm" represents the calm / soothing emotion type; otherwise represents other emotion types. Specifically, when the music emotion is "excited", the action speed increases by 20% , Movement range increased by 15% When the music mood is calm, the action speed is reduced by 10% ( , action range reduced by 5% For other emotion types, keep the original movement speed and range unchanged.

[0205] Finally, the system applies these parameters to the predicted key joint coordinates to generate key joint coordinates with enhanced expressiveness:

[0206] ,

[0207] ,

[0208] in: is the motion vector, with dimension , indicating that from the current moment To the predicted time displacement of key joints; is the key joint coordinate at the current moment, and its dimension is , is the key joint coordinate at the prediction moment, with dimension , is the enhanced key joint coordinate, the dimension is , is the action amplitude enhancement factor; is the motion range factor; is the action speed factor; represents the operation of multiplying a scalar by a vector / matrix.

[0209] The innovation of this mechanism lies in its deep integration of audio information and motion prediction, achieving a synergistic enhancement of music and movement. Experiments have shown that this mechanism can significantly enhance the artistic expression of virtual performances and the audience's sense of immersion.

[0210] The data transmission unit 9 of the present invention is responsible for efficiently transmitting the enhanced predicted key joint coordinates to the rendering end. In a preferred embodiment, the transmission process includes the following steps:

[0211] First, the system compresses and encodes the enhanced predicted key joint coordinates. Preferably, differential encoding is used to transmit only the changes in key joint positions, and a floating-point compression algorithm is applied to maintain accuracy to the minimum required for visual performance (typically millimeter-level). This approach can reduce data volume by approximately 70% to 80%, significantly reducing bandwidth requirements.

[0212] Secondly, the system adds a timestamp and delay compensation information to the data packet. The timestamp indicates the theoretical time corresponding to the predicted pose (current time + prediction time window), and the delay compensation information contains prediction parameters and confidence, which is used for fine-tuning on the rendering end.

[0213] The system then transmits the data packets via a low-latency network protocol. Preferably, optimized WebSocket or UDP protocols are used to minimize transmission latency while ensuring reliability. For critical data packets, an adaptive reliability mechanism is implemented, prioritizing low latency when network conditions are good and increasing reliability appropriately when the network fluctuates.

[0214] Finally, the rendering end receives and parses the data packets to recover the key joint coordinate data. Preferably, the rendering end implements a buffer management strategy to dynamically adjust the buffer size based on timestamps and system status to minimize additional delay while ensuring smooth playback.

[0215] In actual applications, this transmission mechanism can control network transmission delay within the range of 5-15 milliseconds (wired network environment), and reduce data bandwidth requirements to approximately 2-5Mbps, meeting the real-time transmission requirements of most network environments.

[0216] The inverse kinematics processing unit 10 of the present invention implements inverse kinematics processing based on physical constraints at the rendering end, ensuring the naturalness and rationality of the virtual character's movements.

[0217] In a preferred embodiment, the system first applies key joint angle limits to ensure that key joint movements conform to human physiological structure. For example, the flexion and extension range of the elbow key joint is limited to 0-145 degrees, and the flexion and extension range of the wrist key joint is limited to 60-70 degrees to avoid excessive bending or unnatural postures. The specific implementation is:

[0218] ,

[0219] in: is the final key joint angle, in degrees; is the calculated angle in degrees; is the minimum allowable angle of the key joint, in degrees; The maximum allowed angle of the key joint is in degrees; clamp means that the clamping function limits the input value to between the specified minimum and maximum values.

[0220] Secondly, the system applies motion smoothness constraints to avoid drastic changes in the positions of key joints between adjacent frames. Preferably, a weighted smoothing algorithm is used:

[0221] ,

[0222] in: is the key joint position after smoothing, and the dimension is , is the current frame position, and its dimension is , is the previous frame position, and the dimension is , is the smoothing factor, ranging from 0 to 1, with a preferred value of 0.7 to 0.9. It is dynamically adjusted according to the speed of the action. A larger value indicates a weaker smoothing effect, and a smaller value indicates a stronger smoothing effect. · represents the operation of multiplying a scalar by a vector / matrix.

[0223] The system then handles self-collision to prevent the avatar's limbs from penetrating each other. Preferably, a simplified collision detection and response mechanism is implemented, modeling key limbs (such as arms and hands) as capsules or spheres to detect and resolve collisions.

[0224] Finally, the system uses an inverse kinematics algorithm to convert the 3D key joint coordinates into the virtual character's skeletal rotation data. Preferably, a damped least squares (DLS) or forward-backward arrival inverse kinematics (FABRIK) algorithm is used to meet the target position while considering the aforementioned physical constraints.

[0225] For musical performances, the system also optimizes for specific instruments. For example, for piano playing, the focus is on optimizing the contact accuracy between fingers and keys; for guitar playing, the focus is on optimizing the interaction between fingers and strings. These optimizations are achieved by adjusting the weights and constraint parameters of the corresponding key joints.

[0226] The advantage of this mechanism is that it ensures that even based on prediction and enhancement, the movements of virtual characters still conform to the laws of physics and human kinematics, avoiding unnatural or physical violations of movements and improving the credibility and viewing quality of virtual performances.

[0227] Reference Figure 7 The present invention also provides a real-time driving system for music performance gestures based on time-space collaboration, comprising:

[0228] Video and audio acquisition unit 1, used to obtain multi-angle video streams and audio signals;

[0229] A lightweight posture estimation unit 2, configured to extract coordinate data of key joints of the performer based on the multi-angle video stream through a lightweight convolutional neural network;

[0230] A multi-scale temporal feature extraction unit 3 is used to analyze the key joint coordinate data in the historical frame based on time windows of different lengths to extract multi-scale temporal features;

[0231] A key joint importance calculation unit 4 is used to calculate the importance weight of the key joint;

[0232] A spatiotemporal collaborative processing unit 5 is configured to fuse the multi-scale temporal features and the current frame spatial features through a spatiotemporal depth collaborative interaction mechanism;

[0233] Delay perception unit 6, used to obtain processing delay information of each link in the system and adjust prediction parameters based on the delay information;

[0234] A posture prediction unit 7, configured to predict the key joint coordinates at a future moment based on the fused features and the adjusted prediction parameters;

[0235] a music-driven enhancement unit 8, configured to enhance the expressiveness of the predicted key joint coordinate data by combining the music rhythm information in the audio signal;

[0236] The data transmission unit 9 is used to transmit the enhanced predicted key joint coordinates to the rendering end; and the inverse kinematics processing unit 10 is used to convert the enhanced predicted key joint coordinates into virtual character driving data based on physical constraints and inverse kinematics models at the rendering end.

[0237] In this embodiment, the system is divided into two parts: the edge side and the rendering side. The edge side is deployed on an edge computing device close to the performers (such as the NVIDIA Jetson Xavier NX / AGX); the rendering side is deployed on a high-performance host (such as a PC equipped with an RTX 3060 or higher graphics card). Both sides are connected via a low-latency network, preferably a wired connection to ensure stability.

[0238] The preferred hardware configuration for the edge side is: a six-core processor (1.4GHz or higher), 8GB or more of memory, a GPU with 384 or more CUDA cores, and support for INT8 quantization acceleration. The preferred configuration for the rendering side is: an eight-core, sixteen-thread processor (3.0GHz or higher), 16GB or more of memory, and an RTX3060 or higher performance graphics card.

[0239] In terms of the system software framework, the edge side uses optimized TensorRT or ONNX Runtime to run lightweight models, OpenCV to process video streams, librosa to analyze audio signals, and optimized WebSocket or ZeroMQ for data transmission. The rendering side uses Unity3D or Unreal Engine for virtual character rendering and physical constraint processing.

[0240] The system's advantages include relatively low hardware costs (approximately 70% to 80% lower than professional motion capture systems), ease of deployment (no specialized markers or wearable devices required), and strong adaptability and scalability. Practical applications have proven that the system can operate stably for over four hours in various musical performance scenarios with no noticeable performance degradation, meeting the requirements of commercial applications.

[0241] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A real-time driving method for music performance gestures based on spatiotemporal collaboration, characterized in that: include: Obtain multi-angle video streams and audio signals; Based on the multi-angle video stream, extracting coordinate data of the performer's key joints through a lightweight convolutional neural network; The coordinate data of the key joints in the historical frame are analyzed, including: extracting multi-scale temporal features based on time windows of different lengths; calculating the importance weights of the key joints; fusing the multi-scale temporal features and the current frame spatial features through a spatiotemporal depth collaborative interaction mechanism, and applying the importance weights of the key joints to the fusion process to weightedly enhance the key joint features related to the performance; obtaining acquisition delay information, processing delay information and transmission delay information in the real-time driving processing flow of the music performance posture, and adjusting the prediction parameters based on the delay information; predicting the key joint coordinate data at the future moment based on the fused features and the adjusted prediction parameters; enhancing the expressiveness of the predicted key joint coordinate data in combination with the music rhythm information in the audio signal; transmitting the enhanced key joint coordinate data to the rendering end; and at the rendering end, converting the enhanced key joint coordinate data into virtual character driving data based on physical constraints and inverse kinematics models; The extracting of multi-scale temporal features based on time windows of different lengths specifically includes: setting a short-term time window, a medium-term time window, and a long-term time window, respectively for capturing motion characteristics of different time scales; extracting instantaneous change features for the key joint coordinate sequence in the short-term time window; extracting complete motion unit features for the key joint coordinate sequence in the medium-term time window; extracting rhythm pattern features for the key joint coordinate sequence in the long-term time window; and fusing the instantaneous change features, the complete motion unit features, and the rhythm pattern features to generate a comprehensive multi-scale temporal feature; Calculating the importance weights of the key joints specifically includes: establishing a key joint connection topology map based on the human skeletal structure; calculating the relative positional relationship between the key joints in the current frame; generating attention scores between the key joints based on the relative positional relationship; identifying the current instrument type and playing state; and enhancing the attention scores of the key joints related to the performance based on the instrument type and the playing state to generate the final key joint importance weights; The method of obtaining acquisition delay information, processing delay information, and transmission delay information in the real-time driving process of music performance posture and adjusting prediction parameters based on the delay information specifically includes: real-time monitoring of video acquisition delay information, processing delay information, and network transmission delay information; encoding the delay information into a conditional vector; setting multiple prediction branches based on different delay magnitudes; and dynamically adjusting the weight of each prediction branch according to the current actual delay situation to generate prediction parameters adapted to the current system state.

2. The method according to claim 1, characterized in that The method of extracting coordinate data of the performer's key joints through a lightweight convolutional neural network specifically includes: extracting two-dimensional key joint coordinates for each frame image of the multi-angle video stream; assigning a confidence score to each key joint coordinate; and generating more three-dimensional key joint coordinate data through triangulation based on the two-dimensional key joint coordinates and their confidence scores from multiple perspectives.

3. The method according to claim 1, characterized in that The method of fusing the multi-scale temporal features and the current frame spatial features through the deep spatiotemporal collaborative interaction mechanism specifically includes: using the current frame spatial features as queries and the multi-scale temporal features as keys and values, calculating the cross-attention from time to space, and obtaining spatial features enhanced by temporal information; using the multi-scale temporal features as queries and the spatial features enhanced by temporal information as keys and values, calculating the cross-attention from space to time, and obtaining temporal features enhanced by spatial information; applying the key joint importance weights to the temporal features enhanced by spatial information; and further deepening the fusion of spatiotemporal information through multiple rounds of interactive iterations to generate a unified spatiotemporal collaborative representation.

4. The method according to claim 1, wherein The method of enhancing the expressiveness of the predicted key joint coordinate data by combining the music rhythm information in the audio signal specifically includes: extracting the beat position, beat intensity and music emotion characteristics from the audio signal; calculating the time relationship between the current moment and the most recent beat to obtain the beat phase information; determining the motion amplitude enhancement factor based on the beat phase information and the beat intensity; adjusting the motion speed and range parameters based on the music emotion characteristics; and applying the motion amplitude enhancement factor, the motion speed and the range parameters to the predicted key joint coordinates to generate key joint coordinate data with enhanced expressiveness.

5. The method according to claim 1, wherein The method of converting the enhanced key joint coordinate data into virtual character driving data based on physical constraints and inverse kinematics models specifically includes: applying key joint angle restrictions to ensure that key joint movements conform to the physiological structure of the human body; applying motion smoothness constraints to avoid changes in key joint positions between adjacent frames; handling self-collision situations to prevent the virtual character's limbs from penetrating each other; and converting the three-dimensional key joint coordinate data into virtual character bone rotation data through an inverse kinematics algorithm for driving the virtual character model.

6. The method according to claim 1, characterized in that The transmitting of the enhanced key joint coordinate data to the rendering end specifically includes: compressing and encoding the enhanced key joint coordinate data; adding timestamp and delay compensation information; transmitting to the rendering end via a low-latency network protocol; and parsing the received data packet at the rendering end to restore the key joint coordinate data.

7. A real-time driving system for music performance gestures based on spatiotemporal coordination, for executing the method according to any one of claims 1 to 6, characterized in that: include: Video and audio acquisition unit, used to obtain multi-angle video streams and audio signals; A lightweight posture estimation unit, configured to extract coordinate data of key joints of the performer based on the multi-angle video stream using a lightweight convolutional neural network; A multi-scale temporal feature extraction unit, configured to analyze the coordinate data of the key joints in the historical frames based on time windows of different lengths to extract multi-scale temporal features; A key joint importance calculation unit, used to calculate the importance weight of the key joint; A spatiotemporal collaborative processing unit is configured to fuse the multi-scale temporal features and the current frame spatial features through a spatiotemporal deep collaborative interaction mechanism, and to apply the importance weights of the key joints to the fusion process to perform weighted enhancement on the key joint features related to the performance; a delay perception unit is configured to obtain acquisition delay information, processing delay information, and transmission delay information in the real-time driving processing flow of the music performance posture, and to adjust the prediction parameters based on the delay information; The posture prediction unit is used to predict the key joint coordinate data at the future moment based on the fused features and adjusted prediction parameters; a music-driven enhancement unit, configured to enhance the expressiveness of the predicted key joint coordinate data by combining the music rhythm information in the audio signal; A data transmission unit, used to transmit the enhanced key joint coordinate data to the rendering end; and an inverse kinematics processing unit, which is used to convert the enhanced key joint coordinate data into virtual character driving data based on physical constraints and inverse kinematics models at the rendering end.

Images