Robot action control method and system based on deep learning

Abstract

The invention provides a robot action control method and system based on deep learning, and belongs to the technical field of artificial intelligence and robot control crossing. The method comprises the steps that S1, a robot action generation model is created based on a music encoder, an action encoder, an action generator, an action discriminator and a fusion layer; s2, acquiring a large amount of historical music and historical action videos to construct a data set; s3, training a robot action generation model based on the data set; s4, deploying a robot action generation model passing the test; and S5, collecting real-time music, preprocessing the real-time music, inputting the preprocessed real-time music into the deployed robot action generation model to obtain a command action, performing coordinate mapping on the command action to obtain a robot control instruction, and controlling the robot based on the robot control instruction. The method has the advantages that the anthropomorphic degree of robot action display is greatly improved, and action distortion is reduced.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of artificial intelligence and robot control, and particularly to a robot motion control method and system based on deep learning. Background Art

[0002] In the field of music performing arts, the conductor's body movements are not only the physical carrier of beat control, but also the core medium for expressing musical emotions. In recent years, with the penetration and development of artificial intelligence technology in the art field, the automatic generation technology of music conducting movements has gradually become a hot topic in interdisciplinary research. Traditional methods mainly rely on MIDI (Musical Instrument Digital Interface) file parsing technology, and generate conducting movements by establishing a mapping relationship between note durations and movement amplitudes. Although such technologies can complete the basic beat indication function, they have the following significant defects:

[0003] First of all, in terms of the motion generation mechanism, the existing technologies generally adopt a rule - based rigid mapping system. This mechanical mapping ignores the crucial dynamic expression elements in music performance, resulting in the generated conducting movements showing obvious mechanical repetition characteristics.

[0004] Secondly, in the dimension of emotional expression, the existing technologies lack effective music semantic parsing capabilities, which directly leads to the lack of necessary artistic expressiveness in the generated conducting movements.

[0005] Then, at the physical implementation level, the existing technologies are mostly limited to the animation rendering of virtual characters, and seriously lack the consideration of the motion constraints of physical robots. When attempting to transplant such algorithms to a physical robot platform, it directly affects the motion completion degree.

[0006] Therefore, how to provide a robot motion control method and system based on deep learning to improve the anthropomorphism of robot motion display and reduce motion distortion has become an urgent technical problem to be solved. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a robot motion control method and system based on deep learning to improve the anthropomorphism of robot motion display and reduce motion distortion.

[0008] In the first aspect, the present invention provides a robot motion control method based on deep learning, including the following steps:

[0009] Step S1: Create a robot motion generation model based on a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot motion generation model;

[0010] The music encoder is used to extract audio features from the input music; the action encoder is used to extract action features of the conducting actions from the input action video, and based on the action features, train the music encoder to enable the music encoder to learn the synchronization relationship between the music and the conducting actions; the action generator is used to output conducting actions according to the audio features extracted by the music encoder; the action discriminator is used to determine whether the conducting actions output by the action generator are real actions or simulated actions; the fusion layer is used to determine the synchronization of the audio features extracted by the music encoder and the action features extracted by the action encoder;

[0011] Step S2: Obtain a large number of historical music pieces and the corresponding historical action videos, and construct a data set based on each of the historical music pieces and historical action videos;

[0012] Step S3: Divide the data set into a training set, a validation set, and a test set, train the robot action generation model through the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing through the validation set and the test set respectively;

[0013] Step S4: Deploy the robot action generation model that passes the test;

[0014] Step S5: Collect real-time music, preprocess the real-time music and then input it into the deployed robot action generation model to obtain conducting actions carrying the two-dimensional coordinates of each joint point, perform coordinate mapping on the conducting actions to obtain robot control instructions, and control the robot to execute actions matching the real-time music based on the robot control instructions.

[0015] Further, in step S1, the music encoder is constructed based on a three-level downsampling module, and each downsampling module is constructed based on three residual blocks and a biaxial pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and the time axis of the two-dimensional Mel spectrogram to extract audio features;

[0016] The action encoder is used to extract action features including spatial structure features and temporal dynamic features of the conducting actions from the input action video through a spatio-temporal graph convolutional network;

[0017]

[0018] ​The action discriminator is constructed based on a joint dynamic feature extraction module, a joint space feature extraction module, and a discrimination module; the joint dynamic feature extraction module is used to extract joint dynamic features for characterizing joint dynamic changes from the command actions output by the action generator through group convolution; the joint space feature extraction module is used to extract joint space features for characterizing the spatial relationship between joints from the command actions output by the action generator through one-dimensional convolution; the discrimination module is used to reason about the joint dynamic features and joint space features through a fully connected layer, and output the discrimination result of whether the command action is a real action or a simulated action.

[0019] Further, step S2 is specifically as follows:

[0020] Obtain a large number of historical music and historical action videos corresponding to the historical music, perform preprocessing on each historical music including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement, perform preprocessing on each historical action video including at least video cleaning, video conversion, and video enhancement, perform annotation of command actions on each preprocessed historical music and historical action video, and construct a dataset based on each annotated historical music and historical action video.

[0021] Further, step S3 is specifically as follows:

[0022] Divide the dataset into a training set, a validation set, and a test set based on a ratio of 7:2:1, train the robot action generation model through the training set, continuously optimize the hyperparameters of the robot action generation model during the training process until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition;

[0023] Validate the trained robot action generation model through the validation set, determine whether the action accuracy is greater than a preset accuracy threshold, if not, the validation fails, expand the training set and continue training; if so, the validation passes, and:

[0024] Test the robot action generation model that has passed the validation through the test set, determine whether the F1 score is greater than a preset score threshold, if not, the test fails, expand the training set and continue training; if so, the test passes, and end the training.

[0025] Further, step S4 is specifically as follows:

[0026] Perform compression operations of quantization and dynamic pruning on the robot action generation model that has passed the test, and deploy the compressed robot action generation model to the robot through containerization technology.

[0027] Second aspect, the present invention provides a robot motion control system based on deep learning, including the following modules:

[0028] A robot motion generation model creation module, configured to create a robot motion generation model based on a music encoder, a motion encoder, a motion generator, a motion discriminator, and a fusion layer, and set a loss function of the robot motion generation model;

[0029] The music encoder is used to extract audio features from the input music; the motion encoder is used to extract motion features of the command motion from the input motion video, and train the music encoder based on the motion features to enable the music encoder to learn the synchronization relationship between music and the command motion; the motion generator is used to output a command motion according to the audio features extracted by the music encoder; the motion discriminator is used to determine whether the command motion output by the motion generator is a real motion or a simulated motion; the fusion layer is used to determine the synchronization of the audio features extracted by the music encoder and the motion features extracted by the motion encoder;

[0030] A dataset construction module, configured to obtain a large number of historical music and historical motion videos corresponding to the historical music, and construct a dataset based on each of the historical music and historical motion videos;

[0031] A robot motion generation model training module, configured to divide the dataset into a training set, a validation set, and a test set, train the robot motion generation model through the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing through the validation set and the test set respectively;

[0032] A robot motion generation model deployment module, configured to deploy the robot motion generation model that passes the test;

[0033] A robot motion control module, configured to collect real-time music, preprocess the real-time music and then input it into the deployed robot motion generation model to obtain a command motion carrying the two-dimensional coordinates of each joint point, perform coordinate mapping on the command motion to obtain a robot control instruction, and control the robot to execute an action matching the real-time music based on the robot control instruction.

[0034] Further, in the robot motion generation model creation module, the music encoder is constructed based on a three-level downsampling module, and each of the downsampling modules is constructed based on three residual blocks and a two-axis pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and the time axis of the two-dimensional Mel spectrogram to extract audio features;

[0035] The action encoder is used to extract action features including spatial structure features and temporal dynamic features of the command action from the input action video through a spatio-temporal graph convolutional network;

[0036] The action generator is constructed based on a temporal convolutional network, and the temporal convolutional network includes six dilated one-dimensional convolutional layers with residual connections. Each dilated one-dimensional convolutional layer contains 64 channels, and the kernel size is 5;

[0037] The action discriminator is constructed based on a joint dynamic feature extraction module, a joint spatial feature extraction module, and a discrimination module; the joint dynamic feature extraction module is used to extract joint dynamic features for characterizing joint dynamic changes from the command action output by the action generator through group convolution; the joint spatial feature extraction module is used to extract joint spatial features for characterizing the spatial relationship between joints from the command action output by the action generator through one-dimensional convolution; the discrimination module is used to infer the joint dynamic features and joint spatial features through a fully connected layer, and output the discrimination result of whether the command action is a real action or a simulated action.

[0038] Further, the dataset construction module is specifically used for:

[0039] Obtain a large number of historical music and historical action videos corresponding to the historical music, perform preprocessing on each historical music including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement, perform preprocessing on each historical action video including at least video cleaning, video conversion, and video enhancement, perform annotation of the command action on each preprocessed historical music and historical action video, and construct a dataset based on the annotated historical music and historical action videos.

[0040] Further, the robot action generation model training module is specifically used for:

[0041] Divide the dataset into a training set, a validation set, and a test set according to a ratio of 7:2:1, train the robot action generation model through the training set, and continuously optimize the hyperparameters of the robot action generation model during the training process until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition;

[0042] Verify the trained robot action generation model through the validation set, and judge whether the action accuracy is greater than a preset accuracy threshold. If not, the verification fails, and the training set is expanded and training continues; if so, the verification passes, and:

[0043] Use the test set to test the verified robot action generation model, and determine whether the F1 score is greater than a preset score threshold. If not, the test fails, and the training set is expanded for continued training. If so, the test passes and the training ends.

[0044] Further, the robot action generation model deployment module is specifically used for:

[0045] Perform compression operations of quantization and dynamic pruning on the tested robot action generation model, and deploy the compressed robot action generation model to the robot through containerization technology.

[0046] The advantages of the present invention are as follows:

[0047] 1. Create a robot action generation model through a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot action generation model; the music encoder is used to extract audio features from the input music; the action encoder is used to extract action features of the conducting action from the input action video, and based on the action features, train the music encoder to enable the music encoder to learn the synchronization relationship between music and the conducting action; the action generator is used to output the conducting action according to the audio features extracted by the music encoder; the action discriminator is used to determine whether the conducting action output by the action generator is a real action or a simulated action; the fusion layer is used to determine the synchronization of the audio features extracted by the music encoder and the action features extracted by the action encoder; then obtain a large number of historical music and historical action videos to construct a data set, divide the data set into a training set, a validation set, and a test set, train the robot action generation model through the training set until the loss value of the loss function is less than a preset loss threshold, and then perform verification and testing through the validation set and the test set respectively, and deploy the tested robot action generation model; finally, collect and preprocess the real-time music and input it into the deployed robot action generation model to obtain a conducting action carrying the two-dimensional coordinates of each joint point, perform coordinate mapping on the conducting action to obtain a robot control instruction, and control the robot to execute an action matching the real-time music based on the robot control instruction; that is, command the robot to move based on the conducting action generated by the pre-trained robot action generation model. Through the music encoder and action encoder of the robot action generation model, the synchronization relationship between music and the conducting action can be effectively improved. Combining the discriminative feedback of the action discriminator, it overcomes the obvious mechanical repetition characteristics in the traditional way and has a certain artistic expressiveness. Moreover, the generated conducting action is subjected to coordinate mapping to obtain a robot control instruction, fully considering the motion constraints of the physical robot, and ultimately greatly improving the anthropomorphism of the robot action display and reducing action distortion.

[0048] 2. By jointly judging the synchronization of music features and motion features through the fusion layer, the problem of temporal alignment of cross-modal data (audio and visual motion) is solved, ensuring that the generated motion precisely matches the music rhythm.

[0049] 3. By using a three-level downsampling module (residual block + biaxial pooling) in the music encoder, deep features of the Mel spectrogram are efficiently extracted through double downsampling along the frequency axis and time axis, effectively enhancing the audio representation ability.

[0050] 4. By using a spatio-temporal graph convolutional network in the motion encoder, both the spatial structural features of the conducting motion (such as the geometric relationship between joints) and the temporal dynamic features (such as motion continuity) are captured, effectively enhancing the representation dimension of motion features.

[0051] 5. By using a dilated one-dimensional convolutional layer (residual connection + 6-layer stacking) in the motion generator, the temporal receptive field is expanded, ensuring the coherence and long-range rhythm dependence of the generated motion.

[0052] 6. By setting up a motion discriminator, the time-varying characteristics and spatial correlation of the motion are captured through a joint dynamic feature extraction module (group convolution) and a joint spatial feature extraction module (one-dimensional convolution) respectively, and the two types of features are fused through a fully connected layer, improving the accuracy of judging the authenticity of the generated motion.

[0053] 7. By performing sampling rate conversion, denoising, and feature enhancement on historical music, and cleaning, converting, and enhancing historical motion videos, the data quality and diversity are effectively improved, and thus the generalization ability of the model is effectively enhanced.

[0054] 8. By using a 7:2:1 ratio to divide the dataset and combining the early stopping mechanism for the validation set (presetting an accuracy threshold) and the F1 score evaluation for the test set, overfitting is effectively prevented, ensuring the robustness of the model.

[0055] 9. During the training process, if the validation set or test set does not meet the standard, the training set is dynamically expanded and retrained to form a closed-loop optimization process, improving the model iteration efficiency.

[0056] 10. Before deployment, by dynamically pruning and quantizing the model, the model volume is compressed and the computational complexity is reduced to make it adapt to the limited hardware resources on the robot side while maintaining the inference accuracy; by using containerization technology for deployment, it supports rapid migration and version management, improving the system maintainability and scalability.

[0057] 11. By preprocessing real-time music and generating conducting motions carrying joint two-dimensional coordinates, and then converting them into robot control commands, an end-to-end low-latency response from music input to mechanical motion is achieved, meeting the real-time interaction requirements.

[0058] 12. By setting the action generator to adopt a Temporal Convolutional Network (TCN), whose dilated one-dimensional convolutional layer supports parallel computing (superior to the sequential processing of RNN / LSTM), the inference latency is significantly reduced, meeting the millisecond-level response requirement for real-time music-to-action generation; by directly generating robot control instructions through coordinate mapping, the traditional intermediate encoding link is skipped, reducing the length of the computational chain and further enhancing the real-time performance.

[0059] 13. By setting the two-axis pooling layer (frequency axis + time axis) of the music encoder to compress the Mel spectrogram, while retaining the key audio information, the data dimension is reduced, the model computation amount is decreased, and it adapts to the real-time processing scenario.

[0060] 14. By setting the dual-path feature extraction (dynamic + spatial) of the action discriminator, the generator is forced to simultaneously learn the temporal rationality of the action and the spatial coordination of joint movements, avoiding the generation of mechanical jitters or abnormal actions that violate ergonomics.

[0061] 15. By segmenting the historical music, the long audio is cut into segments aligned with the action video, ensuring that the model learns the local correspondence between music segments and action segments, rather than the global fuzzy association, enhancing the detail accuracy of the generated actions.

[0062] 16. By setting the combination of group convolution and one-dimensional convolution of the action discriminator, while reducing the number of parameters, the spatio-temporal characteristics of joint movements are retained, avoiding the high computational overhead of traditional 3D convolution.

[0063] 17. Through an innovative multi-modal deep learning architecture (music encoder + action generator + fusion layer), high-precision real-time synchronization between music and robot actions is achieved. Among them, the music encoder uses a three-stage downsampling module and a two-axis pooling layer to extract compressed Mel spectrogram features. The action generator is based on a Temporal Convolutional Network (TCN) and dilated one-dimensional convolution to achieve low-latency action generation. At the same time, a Spatio-Temporal Graph Convolutional Network (ST-GCN) and a dual-path discriminator (dynamic + spatial features) are combined to ensure the spatio-temporal rationality of the action and its fit with the music beats; at the training level, the generalization ability of the model is improved through segmented audio alignment, data augmentation, and dynamic expansion strategies. When deployed, dynamic pruning and containerization technologies are used to achieve lightweight adaptation. Finally, robot control instructions are directly generated through joint coordinate mapping, significantly optimizing the real-time response efficiency and being scalable to multiple scenarios such as music conducting, dance robots, and virtual reality interactions, combining both technological innovation and engineering practicality. Description of the Drawings

[0064] The present invention will be further described below with reference to the accompanying drawings in conjunction with embodiments.

[0065] Figure 1 is a flowchart of a robot action control method based on deep learning according to the present invention.

[0066] Figure 2 It is a schematic structural diagram of a robot motion control system based on deep learning according to the present invention. Specific implementation manner

[0067] The technical solution in the embodiment of the present application has the following general idea: Based on the command actions generated by the pre-trained robot motion generation model to command the robot to move. Through the music encoder and action encoder of the robot motion generation model, the synchronization relationship between music and command actions can be effectively improved. Combining the discriminative feedback of the action discriminator, it overcomes the obvious mechanical repetition characteristics in the traditional sense and has a certain artistic expressiveness. And the generated command actions are subjected to coordinate mapping to obtain robot control commands, fully considering the motion constraints of the physical robot to improve the anthropomorphic degree of the robot motion display and reduce motion distortion.

[0068] Please refer to Figures 1 to 2 As shown, a preferred embodiment of a robot motion control method based on deep learning according to the present invention includes the following steps:

[0069] Step S1: Create a robot motion generation model based on a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot motion generation model;

[0070] The music encoder is used to extract audio features from the input music; the action encoder is used to extract action features of command actions from the input action video, and based on the action features, train the music encoder to enable the music encoder to learn the synchronization relationship between music and command actions; the action generator is used to output command actions according to the audio features extracted by the music encoder; the action discriminator is used to judge whether the command actions output by the action generator are real actions or simulated actions; the fusion layer is used to judge the synchronization of the audio features extracted by the music encoder and the action features extracted by the action encoder;

[0071] By jointly judging the synchronization of music features and action features through the fusion layer, the problem of temporal alignment of cross-modal data (audio and visual actions) is solved, ensuring that the generated actions are accurately matched with the music rhythm.

[0072] Step S2: Obtain a large number of historical music and historical action videos corresponding to the historical music, and construct a data set based on each of the historical music and historical action videos; Specifically, the open-source data set ConductorMotion100 can be directly selected.

[0073] Step S3: Divide the dataset into a training set, a validation set, and a test set. Train the robot motion generation model using the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing using the validation set and the test set respectively;

[0074] Step S4: Deploy the robot motion generation model that passes the test;

[0075] Step S5: Collect real-time music. After preprocessing the real-time music, input it into the deployed robot motion generation model to obtain a conducting motion carrying the two-dimensional coordinates of each joint point. Perform coordinate mapping on the conducting motion to obtain a robot control instruction, and control the robot to execute an action matching the real-time music based on the robot control instruction.

[0076] By preprocessing real-time music and generating a conducting motion carrying joint two-dimensional coordinates, and then converting it into a robot control instruction, an end-to-end low-latency response from music input to mechanical action is achieved, meeting the requirements of real-time interaction.

[0077] Through an innovative multi-modal deep learning architecture (music encoder + motion generator + fusion layer), high-precision real-time synchronization between music and robot motion is achieved. The music encoder uses a three-level downsampling module and a two-axis pooling layer to extract compressed Mel spectrogram features. The motion generator is based on a Temporal Convolutional Network (TCN) and dilated one-dimensional convolution to achieve low-latency motion generation. At the same time, a Spatial-Temporal Graph Convolutional Network (ST-GCN) and a dual-path discriminator (dynamic + spatial features) are combined to ensure the spatio-temporal rationality of the motion and its fit with the music beat. At the training level, the model generalization ability is improved through segmented audio alignment, data augmentation, and dynamic expansion strategies. During deployment, dynamic pruning and containerization technologies are used to achieve lightweight adaptation. Finally, robot control instructions are directly generated through joint coordinate mapping, significantly optimizing the real-time response efficiency and being scalable to multiple scenarios such as music conducting, dance robots, and virtual reality interaction, combining both technological innovation and engineering practicality.

[0078] In the said step S1, the music encoder is constructed based on a three-level downsampling module. Each downsampling module is constructed based on three residual blocks and a two-axis pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and the time axis of the two-dimensional Mel spectrogram to extract audio features; the residual block is composed of a two-dimensional convolutional layer with a kernel size of 3×3, batch normalization, a ReLU activation function, and a skip connection; the two-axis pooling layer performs downsampling simultaneously on the frequency axis and the time axis. On the time axis, only one downsampling is performed, and the scaling factor is 3 to ensure that the sampling rate of the output music features in the time dimension is consistent with the corresponding motion features; before performing 3-fold downsampling, the number of channels is 16; after completing the downsampling, the number of channels increases to 32.

[0079] The music encoder adopts a three - level downsampling module (residual block + biaxial pooling) to efficiently extract the deep features of the Mel spectrogram through double downsampling on the frequency axis and the time axis, effectively enhancing the audio representation ability.

[0080] The biaxial pooling layer (frequency axis + time axis) of the music encoder is set to compress the Mel spectrogram, reducing the data dimension while retaining the key audio information, reducing the model's computational load, and adapting to real - time processing scenarios.

[0081] The action encoder is used to extract the action features including spatial structure features and temporal dynamic features of the conducting action from the input action video through a spatio - temporal graph convolutional network; the action encoder contains 10 layers of spatio - temporal graph convolutional networks. In each layer, graph convolution is used to extract spatial structure features, and temporal convolution captures temporal dynamic features; in addition, a 1×1 convolution is introduced in each layer to build residual connections to enhance the feature expression ability. The entire action encoder does not adopt downsampling operations, so the sampling rate of the output action features is 30Hz, consistent with the input, and each convolutional layer contains 32 channels.

[0082] Based on the spatio - temporal graph convolutional network, the action encoder simultaneously captures the spatial structure features (such as the geometric relationship between joints) and temporal dynamic features (such as action continuity) of the conducting action, effectively enhancing the representation dimension of the action features.

[0083] The action generator is constructed based on a temporal convolutional network. The temporal convolutional network contains six dilated one - dimensional convolutional layers with residual connections. Each dilated one - dimensional convolutional layer contains 64 channels and the kernel size is 5; the action generator generates the conducting action according to the music features extracted by the music encoder and a sequence of noise vectors sampled from the standard normal distribution; the noise vector is an 8 - dimensional vector sampled at a frequency of 1Hz, and then upsampled through several layers of transposed convolution, and its frame rate is increased to 30fps and then input to the TCN; the action generator is constructed based on a temporal convolutional network, which significantly reduces the computational overhead while achieving performance similar to that of a standard LSTM.

[0084] The action generator adopts dilated one - dimensional convolutional layers (residual connection + 6 - layer stacking) to expand the temporal receptive field, ensuring the coherence and long - range rhythm dependence of the generated actions.

[0085] By setting the action generator to adopt a temporal convolutional network (TCN), its dilated one - dimensional convolutional layer supports parallel computing (superior to the sequential processing of RNN / LSTM), significantly reducing the inference latency and meeting the millisecond - level response requirement for real - time music - to - action generation; directly generating robot control instructions through coordinate mapping, skipping the traditional intermediate encoding link, reducing the length of the computational chain, and further improving the real - time performance.

[0086] The action discriminator is constructed based on a joint dynamic feature extraction module, a joint spatial feature extraction module, and a discrimination module. The joint dynamic feature extraction module is used to extract joint dynamic features characterizing joint dynamic changes from the command actions output by the action generator through group convolution. The joint spatial feature extraction module is used to extract joint spatial features characterizing the spatial relationship between joints from the command actions output by the action generator through one-dimensional convolution. The discrimination module is used to infer the joint dynamic features and joint spatial features through a fully connected layer and output a discrimination result as to whether the command action is a real action or a simulated action.

[0087] By setting that the action discriminator captures the time-varying characteristics and spatial correlation of the action through the joint dynamic feature extraction module (group convolution) and the joint spatial feature extraction module (one-dimensional convolution) respectively, and fuses the two types of features through a fully connected layer, the accuracy of judging the authenticity of the generated action is improved.

[0088] By setting the dual-path feature extraction (dynamic + spatial) of the action discriminator, the generator is forced to learn both the temporal rationality of the action and the spatial coordination of joint movements simultaneously, avoiding the generation of mechanical jitters or abnormal actions that violate ergonomics.

[0089] By setting the combination of group convolution and one-dimensional convolution of the action discriminator, while reducing the number of parameters, the spatio-temporal characteristics of joint movements are retained, avoiding the high computational cost of traditional 3D convolution.

[0090] In the contrastive learning stage, the music encoder and the action encoder are trained through self-supervised learning to learn the synchronization relationship between music and actions. In the generation learning stage, the music representation learned in the contrastive learning stage is used to provide semantic information for the action generator, and at the same time, the action representation is used to calculate the proposed perceptual training metric.

[0091] The contrastive learning stage includes the following steps: First, initialize the music encoder, the action encoder, and the fusion layer. Subsequently, through iterative training until the model converges, positive (synchronous) and negative (asynchronous) music-action sample pairs are sampled from the dataset, and corresponding labels are assigned to each pair of samples. When predicting the music-action synchrony, the fusion layer is used to combine the output of the music encoder to determine whether the input pair is synchronous. Finally, the weights of the music encoder, the action encoder, and the fusion layer are updated according to the loss function, continuously optimizing the model performance.

[0092] The generation learning stage includes the following steps: First, initialize the action generator and the action discriminator, and load the weights of the music encoder and the action encoder trained in the contrast learning stage; then, perform iterative training until the model converges; in each iteration, sample music-action sample pairs from the dataset, and sample noise vectors from the normal distribution; use MusicEncoder(X) and the noise vector as inputs, and generate conducting actions through the action generator; subsequently, update the weights of the action generator using the action generator loss function LG, and update the weights of the action discriminator using the action discriminator loss function LD; through this adversarial training process, continuously optimize the performance of the action generator and the action discriminator, and finally achieve high-quality conducting action generation.

[0093] The specific steps of step S2 are as follows:

[0094] Obtain a large number of historical music and historical action videos corresponding to the historical music, perform preprocessing on each of the historical music, including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement, perform preprocessing on each of the historical action videos, including at least video cleaning, video conversion, and video enhancement, annotate the conducting actions for each of the preprocessed historical music and historical action videos, and construct a dataset based on the annotated historical music and historical action videos.

[0095] By performing sampling rate conversion, denoising, and feature enhancement on historical music, and cleaning, converting, and enhancing historical action videos, the data quality and diversity are effectively improved, and thus the generalization ability of the model is effectively enhanced.

[0096] By performing segmentation processing on historical music, cutting long audio into segments aligned with action videos, ensuring that the model learns the local correspondence between music paragraphs and action segments, rather than the global fuzzy association, and enhancing the detail accuracy of the generated actions.

[0097] The specific steps of step S3 are as follows:

[0098] Divide the dataset into a training set, a validation set, and a test set based on a ratio of 7:2:1, train the robot action generation model using the training set, and continuously optimize the hyperparameters of the robot action generation model during the training process until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition;

[0099] Validate the trained robot action generation model using the validation set, and determine whether the action accuracy is greater than a preset accuracy threshold. If not, the validation fails, and the training set is expanded and training continues; if so, the validation passes, and:

[0100] Test the verified robot motion generation model with the test set, and determine whether the F1 score is greater than the preset score threshold. If not, the test fails, and the training set is expanded for continued training; if so, the test passes and the training ends.

[0101] By dividing the dataset in a 7:2:1 ratio, combined with the early stopping mechanism (preset accuracy threshold) of the validation set and the F1 score evaluation of the test set, overfitting is effectively prevented and the robustness of the model is ensured.

[0102] During the training process, if the validation set or the test set does not meet the standard, the training set is dynamically expanded and retrained to form a closed-loop optimization process, improving the model iteration efficiency.

[0103] The specific step S4 is as follows:

[0104] Perform quantization and dynamic pruning compression operations on the tested robot motion generation model, and deploy the compressed robot motion generation model to the robot through containerization technology.

[0105] By performing dynamic pruning and quantization on the model before deployment, the model volume is compressed and the computational complexity is reduced, making it adapt to the limited hardware resources of the robot side while maintaining the inference accuracy; by using containerization technology for deployment, it supports fast migration and version management, improving the system maintainability and scalability.

[0106] In the step S5, the coordinate mapping process is specifically as follows:

[0107] The key point coordinates in the Motion matrix are normalized. The distance between the left and right shoulders is about 0.3, the length of the upper arm (from shoulder to elbow) is about 1.2 times the head length, and the length of the forearm (from elbow to wrist) is about 1 times the head length; according to the principles of human anatomy, the normalized 3D coordinates can be calculated, and the hip of the motion 2D data is located at (0.5, 0.75).

[0108] In the three-dimensional coordinate system, combined with the x and y axis directions of the two-dimensional coordinate system, define the positive direction of the x axis as left, the positive direction of the y axis as down, and the facing direction as the positive direction of the z axis; define the 3D data (x, y, z), where x = x_2D, y = -y_2D, and z is unknown.

[0109] Combined with the actual command actions, when the commander makes gesture expressions, the movement of the arm is mainly concentrated in the space area in front of the body, and there are very few cases of stretching backward. Therefore, in the process of reconstructing the three-dimensional coordinates of the two-dimensional key points, in order to simplify the modeling and calculation, it is assumed that the movement of the arm in the z axis direction (i.e., the front-back direction of the body) only unfolds forward, that is, only the positive direction coordinates of the z axis are considered. This assumption not only conforms to the physiological laws of actual command actions but also can effectively reduce the uncertainty and computational complexity during pose estimation.

[0110] In addition, considering the structural constraints of the robot and its joint movement capabilities, especially the degree-of-freedom limitations of the shoulder, the coordinates of the shoulder key points in the z-axis direction are further constrained and uniformly set to 0. That is, it is assumed that the shoulder is always located on the body plane and does not participate in the displacement in the z-axis direction. This processing method helps to ensure the realizability of the generated three-dimensional posture during actual robot control and at the same time enhances the correspondence between posture restoration and robot control.

[0111] 1) Elbow coordinate calculation: Taking the left shoulder and elbow as an example, given that the left shoulder coordinates are (LS_x, LS_y, LS_z) and the length of the upper arm (from shoulder to elbow) is usually 1.2 times the head length. Let the head length be X, the left shoulder coordinates: (LS_x, LS_y, LS_z), and the left elbow coordinates: (LE_x, LE_y, f(LE)), then the z coordinate f(LE) is:

[0112]

[0113] 2) Wrist coordinate calculation: Taking the left elbow and wrist as an example, given that the left elbow coordinates are (LE_x, LE_y, f(LE)). The length of the forearm (from elbow to wrist) is usually 1 times the head length. Let the head length be X, the left elbow coordinates: (LE_x, LE_y, f(LE)), and the left wrist coordinates: (LW_x, LW_y, f(LW)), then the z coordinate f(LW) is:

[0114]

[0115] 3) Lateral raise angle

[0116] Definition: The arm spreads outwards in the X-Y plane and rotates around the Z axis. Due to robot limitations, the maximum lateral raise angle is 70 degrees.

[0117] Upper arm vector:

[0118]

[0119] Among them, x e represents the x-axis coordinate of the elbow; x s represents the x-axis coordinate of the shoulder; y e represents the y-axis coordinate of the elbow; y s represents the y-axis coordinate of the shoulder; z e represents the z-axis coordinate of the elbow; z s represents the z-axis coordinate of the shoulder;

[0120] Projection of the upper arm vector in the X-Y plane:

[0121]

[0122] Unit vector of the y-axis:

[0123]

[0124] Angle formula:

[0125]

[0126] 4) Angle of the upper arm lifting

[0127] Definition: The arm is lifted forward in the Y-Z plane and rotates around the X axis. Due to robot limitations, the angle of the upper arm lifting is from 0 to 170 degrees.

[0128] Upper arm vector:

[0129]

[0130] Projection of the upper arm vector in the Y-Z plane:

[0131]

[0132] Unit vector of the y axis:

[0133]

[0134] Angle formula:

[0135]

[0136] 5) Angle of the elbow joint bending

[0137] Definition: When the arm is straight, the corresponding joint angle on the simulation platform is 0 degrees. Therefore, the flexion angle of the elbow joint is the angle between the upper arm (shoulder-elbow vector) and the forearm (elbow-wrist vector), which reflects the bending degree of the elbow joint. Due to robot limitations, the maximum angle of the elbow joint bending is 85 degrees.

[0138] Upper arm vector:

[0139]

[0140] Forearm vector:

[0141]

[0142] Among them, x w , y w , z w respectively represent the x-axis, y-axis, and z-axis coordinates of the wrist;

[0143] Angle formula:

[0144]

[0145] 6) Angle of the forearm rotation

[0146] Definition: The rotation angle of the forearm around its own long axis, which reflects the change in the palm orientation, is one of the important parameters for achieving complex gestures and enhancing the expressiveness of command actions. Due to robot limitations, the rotation angle of the forearm is from -85 degrees to 85 degrees, with the outward rotation angle being positive and the inward rotation angle being negative.

[0147] Upper arm vector:

[0148]

[0149] Forearm vector:

[0150]

[0151] Reference axis z-axis unit vector:

[0152]

[0153] Normal vector of the plane perpendicular to the upper arm vector:

[0154]

[0155] Projection of the forearm vector in the plane perpendicular to the upper arm vector:

[0156]

[0157] Projection of the reference axis z-axis unit vector in the plane perpendicular to the upper arm vector:

[0158]

[0159] Judging the positive and negative of the rotation angle:

[0160] For the right forearm:

[0161]

[0162] For the left forearm:

[0163]

[0164] Angle formula:

[0165]

[0166]

[0167] A preferred embodiment of a robot motion control system based on deep learning according to the present invention includes the following modules:

[0168] A robot motion generation model creation module, which is used to create a robot motion generation model based on a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot motion generation model;

[0169] The music encoder is used to extract audio features from the input music; the action encoder is used to extract action features of the conducting action from the input action video, and train the music encoder based on the action features to enable the music encoder to learn the synchronization relationship between music and the conducting action; the action generator is used to output the conducting action according to the audio features extracted by the music encoder; the action discriminator is used to judge whether the conducting action output by the action generator is a real action or a simulated action; the fusion layer is used to judge the synchronization of the audio features extracted by the music encoder and the action features extracted by the action encoder;

[0170] By jointly judging the synchronization of music features and action features through the fusion layer, the problem of temporal alignment of cross-modal data (audio and visual actions) is solved, ensuring that the generated actions are accurately matched with the music rhythm.

[0171] A dataset construction module, which is used to obtain a large number of historical music and historical action videos corresponding to the historical music, and construct a dataset based on each historical music and historical action video; specifically, the open-source dataset ConductorMotion100 can be directly selected;

[0172] A robot motion generation model training module, which is used to divide the dataset into a training set, a validation set, and a test set, train the robot motion generation model through the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing through the validation set and the test set respectively;

[0173] A robot motion generation model deployment module, which is used to deploy the robot motion generation model that has passed the test;

[0174] A robot motion control module, which is used to collect real-time music, preprocess the real-time music and then input it into the deployed robot motion generation model to obtain a conducting action carrying the two-dimensional coordinates of each joint point, perform coordinate mapping on the conducting action to obtain a robot control instruction, and control the robot to execute an action matching the real-time music based on the robot control instruction.

[0175] By preprocessing the real-time music and generating a conducting action carrying the joint two-dimensional coordinates, and then converting it into a robot control instruction, an end-to-end low-latency response from music input to mechanical action is realized, meeting the real-time interaction requirements.

[0176] High-precision real-time synchronization of music and robot movements is achieved through an innovative multi-modal deep learning architecture (music encoder + motion generator + fusion layer). The music encoder uses a three-level downsampling module and a two-axis pooling layer to extract compressed Mel spectrogram features. The motion generator is based on a Temporal Convolutional Network (TCN) and dilated one-dimensional convolution to achieve low-latency motion generation. At the same time, a Spatio-Temporal Graph Convolutional Network (ST-GCN) and a dual-path discriminator (dynamic + spatial features) are combined to ensure the spatio-temporal rationality of the motion and its alignment with the music beats. At the training level, the generalization ability of the model is improved through segmented audio alignment, data augmentation, and dynamic expansion strategies. During deployment, dynamic pruning and containerization technologies are used to achieve lightweight adaptation. Finally, robot control commands are directly generated through joint coordinate mapping, significantly optimizing the real-time response efficiency. It can be extended to multiple scenarios such as music conducting, dance robots, and virtual reality interactions, combining both technological innovation and engineering practicality.

[0177] In the robot motion generation model creation module, the music encoder is constructed based on a three-level downsampling module. Each downsampling module is constructed based on three residual blocks and a two-axis pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and time axis of the two-dimensional Mel spectrogram to extract audio features. The residual block consists of a two-dimensional convolutional layer with a kernel size of 3×3, batch normalization, a ReLU activation function, and a skip connection. The two-axis pooling layer performs downsampling simultaneously on the frequency axis and time axis. On the time axis, only one downsampling is performed, and the scaling factor is 3 to ensure that the sampling rate of the output music features in the time dimension is consistent with the corresponding motion features. Before the 3-fold downsampling, the number of channels is 16. After the downsampling is completed, the number of channels increases to 32.

[0178] The music encoder adopts a three-level downsampling module (residual block + two-axis pooling), and efficiently extracts the deep features of the Mel spectrogram through double downsampling on the frequency axis and time axis, effectively enhancing the audio representation ability.

[0179] By setting the two-axis pooling layer (frequency axis + time axis) of the music encoder to compress the Mel spectrogram, while retaining the key audio information, the data dimension is reduced, the model calculation amount is reduced, and it adapts to the real-time processing scenario.

[0180] The action encoder is used to extract action features including spatial structure features and temporal dynamic features of the conducting action from the input action video through a spatio-temporal graph convolutional network. The action encoder contains 10 layers of spatio-temporal graph convolutional networks. In each layer, graph convolution is used to extract spatial structure features, and temporal convolution captures temporal dynamic features. In addition, a 1×1 convolution is introduced in each layer to build residual connections to enhance the feature expression ability. The entire action encoder does not adopt downsampling operations. Therefore, the sampling rate of the output action features is 30Hz, which is consistent with the input. Each convolutional layer contains 32 channels.

[0181] Based on the spatio-temporal graph convolutional network, the action encoder simultaneously captures the spatial structure features (such as the geometric relationship between joints) and temporal dynamic features (such as action continuity) of the conducting action, effectively enhancing the representation dimension of the action features.

[0182] The action generator is constructed based on a temporal convolutional network. The temporal convolutional network contains six dilated one-dimensional convolutional layers with residual connections. Each dilated one-dimensional convolutional layer contains 64 channels and the kernel size is 5. The action generator generates the conducting action according to the music features extracted by the music encoder and the noise vector sequence sampled from the standard normal distribution. The noise vector is an 8-dimensional vector sampled at a frequency of 1Hz, and then is upsampled through several transposed convolutional layers. After its frame rate is increased to 30fps, it is input into the TCN. The action generator is constructed based on a temporal convolutional network. While achieving performance similar to that of the standard LSTM, it significantly reduces the computational overhead.

[0183] By using dilated one-dimensional convolutional layers (residual connections + 6-layer stacking) in the action generator, the temporal receptive field is expanded to ensure the coherence and long-range rhythm dependence of the generated action.

[0184] By setting the action generator to use a temporal convolutional network (TCN), its dilated one-dimensional convolutional layers support parallel computing (superior to the sequential processing of RNN / LSTM), significantly reducing the inference latency and meeting the millisecond-level response requirement for real-time music-to-action generation. By directly generating robot control instructions through coordinate mapping, the traditional intermediate encoding link is skipped, reducing the length of the computational chain and further improving the real-time performance.

[0185] The action discriminator is constructed based on a joint dynamic feature extraction module, a joint spatial feature extraction module, and a discrimination module. The joint dynamic feature extraction module is used to extract joint dynamic features characterizing joint dynamic changes from the command actions output by the action generator through group convolution. The joint spatial feature extraction module is used to extract joint spatial features characterizing the spatial relationship between joints from the command actions output by the action generator through one-dimensional convolution. The discrimination module is used to infer the joint dynamic features and joint spatial features through a fully connected layer and output the discrimination result of whether the command action is a real action or a simulated action.

[0186] By setting that the action discriminator captures the time-varying characteristics and spatial correlation of actions through the joint dynamic feature extraction module (group convolution) and the joint spatial feature extraction module (one-dimensional convolution) respectively, and fuses the two types of features through a fully connected layer, the accuracy of judging the authenticity of the generated actions is improved.

[0187] By setting the dual-path feature extraction (dynamic + spatial) of the action discriminator, the generator is forced to learn the temporal rationality of actions and the spatial coordination of joint movements simultaneously, avoiding the generation of mechanical jitters or abnormal actions that violate ergonomics.

[0188] By setting the combination of group convolution and one-dimensional convolution of the action discriminator, while reducing the number of parameters, the spatio-temporal characteristics of joint movements are retained, avoiding the high computational cost of traditional 3D convolution.

[0189] In the contrast learning stage, the music encoder and the action encoder are trained through self-supervised learning to learn the synchronization relationship between music and actions. In the generation learning stage, the music representation learned in the contrast learning stage is used to provide semantic information for the action generator, and at the same time, the action representation is used to calculate the proposed perceptual training metric.

[0190] The contrast learning stage includes the following steps: First, initialize the music encoder, the action encoder, and the fusion layer. Subsequently, through iterative training until the model converges, positive (synchronous) and negative (asynchronous) music-action sample pairs are sampled from the dataset, and corresponding labels are assigned to each pair of samples. When predicting the music-action synchrony, the fusion layer is used to combine the output of the music encoder to judge whether the input pair is synchronous. Finally, according to the loss function, the weights of the music encoder, the action encoder, and the fusion layer are updated to continuously optimize the model performance.

[0191] The generation learning stage includes the following steps: First, initialize the action generator and the action discriminator, and load the weights of the music encoder and the action encoder trained in the contrast learning stage; then, perform iterative training until the model converges; in each iteration, sample music-action sample pairs from the dataset and sample noise vectors from a normal distribution; use MusicEncoder(X) and the noise vector as inputs to generate conducting actions through the action generator; subsequently, update the weights of the action generator using the action generator loss function LG and update the weights of the action discriminator using the action discriminator loss function LD; through this adversarial training process, continuously optimize the performance of the action generator and the action discriminator, and finally achieve high-quality conducting action generation.

[0192] The dataset construction module is specifically used for:

[0193] Obtain a large number of historical music and historical action videos corresponding to the historical music, perform preprocessing on each of the historical music, including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement, perform preprocessing on each of the historical action videos, including at least video cleaning, video conversion, and video enhancement, perform annotation of conducting actions on the preprocessed historical music and historical action videos, and construct a dataset based on the annotated historical music and historical action videos.

[0194] By performing sampling rate conversion, denoising, and feature enhancement on historical music, and cleaning, conversion, and enhancement on historical action videos, the data quality and diversity are effectively improved, and thus the generalization ability of the model is effectively enhanced.

[0195] By performing segmentation processing on historical music, cutting long audio into segments aligned with action videos, ensuring that the model learns the local correspondence between music paragraphs and action segments rather than global fuzzy associations, and enhancing the detail accuracy of the generated actions.

[0196] The robot action generation model training module is specifically used for:

[0197] Divide the dataset into a training set, a validation set, and a test set based on a ratio of 7:2:1, train the robot action generation model through the training set, and continuously optimize the hyperparameters of the robot action generation model during the training process until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition;

[0198] Verify the trained robot action generation model through the validation set, determine whether the action accuracy is greater than a preset accuracy threshold, if not, the verification fails, expand the training set and continue training; if so, the verification passes, and:

[0199] Test the verified robot motion generation model with the test set, and determine whether the F1 score is greater than the preset score threshold. If not, the test fails, and the training set is expanded for continued training; if so, the test passes and the training ends.

[0200] By dividing the dataset in a 7:2:1 ratio, combining the early stopping mechanism (preset accuracy threshold) of the validation set and the F1 score evaluation of the test set, overfitting is effectively prevented and the model robustness is ensured.

[0201] During the training process, if the validation set or the test set does not meet the standard, the training set is dynamically expanded and retrained to form a closed-loop optimization process, improving the model iteration efficiency.

[0202] The robot motion generation model deployment module is specifically used for:

[0203] Perform quantization and dynamic pruning compression operations on the tested robot motion generation model, and deploy the compressed robot motion generation model to the robot through containerization technology.

[0204] By performing dynamic pruning and quantization on the model before deployment, the model volume is compressed and the computational complexity is reduced, making it adapt to the limited hardware resources of the robot side while maintaining the inference accuracy; by using containerization technology for deployment, fast migration and version management are supported, improving the system maintainability and scalability.

[0205] In step S5, the coordinate mapping process is specifically as follows:

[0206] The key point coordinates in the Motion matrix are normalized. The distance between the left and right shoulders is about 0.3, the length of the upper arm (from shoulder to elbow) is about 1.2 times the head length, and the length of the forearm (from elbow to wrist) is about 1 times the head length; according to the principles of human anatomy, the normalized 3D coordinates can be calculated, and the hip of the motion 2D data is located at (0.5, 0.75).

[0207] In the three-dimensional coordinate system, combined with the x and y axis directions of the two-dimensional coordinate system, define the positive direction of the x axis as left, the positive direction of the y axis as down, and the facing direction as the positive direction of the z axis; define the 3D data (x, y, z), where x = x_2D, y = -y_2D, and z is unknown.

[0208] Combined with the actual command actions, when the commander makes gesture expressions, the movement of the arm is mainly concentrated in the space area in front of the body, and there are very few cases of stretching backward. Therefore, in the process of reconstructing the three-dimensional coordinates of the two-dimensional key points, in order to simplify the modeling and calculation, it is assumed that the movement of the arm in the z axis direction (i.e., the front-back direction of the body) only unfolds forward, that is, only the positive direction coordinates of the z axis are considered. This assumption not only conforms to the physiological laws of actual command actions but also can effectively reduce the uncertainty and computational complexity during pose estimation.

[0209] In addition, considering the structural constraints of the robot and its joint movement capabilities, especially the degree-of-freedom limitations of the shoulder, the coordinates of the shoulder key points in the z-axis direction are further constrained and uniformly set to 0. That is, it is assumed that the shoulder is always located on the body plane and does not participate in the displacement in the z-axis direction. This processing method helps to ensure the realizability of the generated three-dimensional pose during actual robot control, and at the same time enhances the correspondence between pose restoration and robot control.

[0210] 1) Elbow coordinate calculation: Taking the left shoulder and elbow as an example, given that the left shoulder coordinates are (LS_x, LS_y, LS_z) and the length of the upper arm (from shoulder to elbow) is usually 1.2 times the head length. Let the head length be X, the left shoulder coordinates: (LS_x, LS_y, LS_z), and the left elbow coordinates: (LE_x, LE_y, f(LE)), then the z coordinate f(LE) is:

[0211]

[0212] 2) Wrist coordinate calculation: Taking the left elbow and wrist as an example, given that the left elbow coordinates are (LE_x, LE_y, f(LE)). The length of the forearm (from elbow to wrist) is usually 1 times the head length. Let the head length be X, the left elbow coordinates: (LE_x, LE_y, f(LE)), and the left wrist coordinates: (LW_x, LW_y, f(LW)), then the z coordinate f(LW) is:

[0213]

[0214] 3) Lateral raise angle

[0215] Definition: The arm spreads out in the X-Y plane and rotates around the Z axis. Due to robot limitations, the maximum lateral raise angle is 70 degrees.

[0216] Upper arm vector:

[0217]

[0218] where, x e represents the x-axis coordinate of the elbow; x s represents the x-axis coordinate of the shoulder; y e represents the y-axis coordinate of the elbow; y s represents the y-axis coordinate of the shoulder; z e represents the z-axis coordinate of the elbow; z s represents the z-axis coordinate of the shoulder;

[0219] Projection of the upper arm vector in the X-Y plane:

[0220]

[0221] Unit vector of the y-axis:

[0222]

[0223] Angle formula:

[0224]

[0225] 4) Angle of the upper arm lifting

[0226] Definition: The arm lifts forward in the Y-Z plane and rotates around the X axis. Due to robot limitations, the angle of the upper arm lifting is from 0 to 170 degrees.

[0227] Upper arm vector:

[0228]

[0229] Projection of the upper arm vector in the Y-Z plane:

[0230]

[0231] Unit vector of the y-axis:

[0232]

[0233] Angle formula:

[0234]

[0235] 5) Angle of the elbow joint bending

[0236] Definition: When the arm is straight, the corresponding joint angle on the simulation platform is 0 degrees. Therefore, the flexion angle of the elbow joint is the angle between the upper arm (shoulder-elbow vector) and the forearm (elbow-wrist vector), which reflects the bending degree of the elbow joint. Due to robot limitations, the maximum bending angle of the elbow is 85 degrees.

[0237] Upper arm vector:

[0238]

[0239] Forearm vector:

[0240]

[0241] Among them, x w 、y w 、z w respectively represent the x-axis, y-axis, and z-axis coordinates of the wrist;

[0242] Angle formula:

[0243]

[0244] 6) Angle of the forearm rotation

[0245] Definition: The rotation angle of the forearm around its own long axis, which reflects the change in the palm orientation, is one of the important parameters for realizing complex gestures and enhancing the expressiveness of command actions. Due to robot limitations, the rotation angle of the forearm is from -85 degrees to 85 degrees, with the outward rotation angle being positive and the inward rotation angle being negative.

[0246] Vector of the upper arm:

[0247]

[0248] Vector of the forearm:

[0249]

[0250] Unit vector of the reference axis z:

[0251]

[0252] Normal vector of the plane perpendicular to the vector of the upper arm:

[0253]

[0254] Projection of the vector of the forearm in the plane perpendicular to the vector of the upper arm:

[0255]

[0256] Projection of the unit vector of the reference axis z in the plane perpendicular to the vector of the upper arm:

[0257]

[0258] Judging the positive and negative of the rotation angle:

[0259] For the right forearm:

[0260]

[0261] For the left forearm:

[0262]

[0263] Angle formula:

[0264]

[0265]

[0266] In summary, the advantages of the present invention are as follows:

[0267] 1. A robot action generation model is created through a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and the loss function of the robot action generation model is set. The music encoder is used to extract audio features from the input music. The action encoder is used to extract the action features of the conducting actions from the input action video, and based on the action features, the music encoder is trained to enable the music encoder to learn the synchronization relationship between the music and the conducting actions. The action generator is used to output conducting actions according to the audio features extracted by the music encoder. The action discriminator is used to determine whether the conducting actions output by the action generator are real actions or simulated actions. The fusion layer is used to determine the synchronization between the audio features extracted by the music encoder and the action features extracted by the action encoder. Then, a large number of historical music and historical action videos are obtained to construct a dataset, which is divided into a training set, a validation set, and a test set. The robot action generation model is trained through the training set until the loss value of the loss function is less than a preset loss threshold, and then verified and tested through the validation set and the test set respectively. The robot action generation model that passes the test is deployed. Finally, the real-time music is collected and preprocessed and then input into the deployed robot action generation model to obtain the conducting actions carrying the two-dimensional coordinates of each joint point. The conducting actions are subjected to coordinate mapping to obtain robot control instructions, and the robot is controlled based on the robot control instructions to execute actions matching the real-time music. That is, the robot is commanded to move based on the conducting actions generated by the pre-trained robot action generation model. Through the music encoder and the action encoder of the robot action generation model, the synchronization relationship between the music and the conducting actions can be effectively improved. Combined with the discriminative feedback of the action discriminator, the obvious mechanical repetition features in the tradition are overcome, and certain artistic expressiveness is possessed. Moreover, the generated conducting actions are subjected to coordinate mapping to obtain robot control instructions, fully considering the motion constraints of the physical robot, and ultimately greatly improving the anthropomorphism of the robot action display and reducing action distortion.

[0268] 2. By jointly judging the synchronization of music features and action features through the fusion layer, the problem of temporal alignment of cross-modal data (audio and visual actions) is solved, ensuring that the generated actions are accurately matched with the music rhythm.

[0269] 3. The music encoder adopts a three-level downsampling module (residual block + biaxial pooling), and through double downsampling on the frequency axis and the time axis, the deep features of the Mel spectrogram are efficiently extracted, effectively enhancing the audio representation ability.

[0270] 4. Through the action encoder based on the spatio-temporal graph convolutional network, the spatial structure features (such as the geometric relationship between joints) and the temporal dynamic features (such as action continuity) of the conducting actions are simultaneously captured, effectively enhancing the representation dimension of the action features.

[0271] 5. The action generator uses dilated one-dimensional convolutional layers (residual connection + 6-layer stack) to expand the temporal receptive field, ensuring the coherence of the generated actions and the long-range rhythm dependence.

[0272] 6. By setting up an action discriminator, the time-varying characteristics and spatial correlation of the actions are captured through a joint dynamic feature extraction module (group convolution) and a joint spatial feature extraction module (one-dimensional convolution) respectively. The two types of features are fused through a fully connected layer to improve the accuracy of judging the authenticity of the generated actions.

[0273] 7. By performing sampling rate conversion, denoising, and feature enhancement on historical music, and cleaning, converting, and enhancing historical action videos, the data quality and diversity are effectively improved, and thus the generalization ability of the model is effectively enhanced.

[0274] 8. By dividing the dataset in a 7:2:1 ratio, combined with the early stopping mechanism for the validation set (preset accuracy threshold) and the F1 score evaluation for the test set, overfitting is effectively prevented, ensuring the robustness of the model.

[0275] 9. During the training process, if the validation set or the test set does not meet the standards, the training set is dynamically expanded and retrained to form a closed-loop optimization process, improving the model iteration efficiency.

[0276] 10. Before deployment, the model is dynamically pruned and quantized to compress the model size and reduce the computational complexity, making it adaptable to the limited hardware resources on the robot side while maintaining the inference accuracy. By using containerization technology for deployment, it supports rapid migration and version management, improving the system maintainability and scalability.

[0277] 11. By preprocessing real-time music and generating conducting actions carrying joint two-dimensional coordinates, and then converting them into robot control instructions, an end-to-end low-latency response from music input to mechanical actions is achieved, meeting the real-time interaction requirements.

[0278] 12. By setting up the action generator to use a Temporal Convolutional Network (TCN), its dilated one-dimensional convolutional layers support parallel computing (superior to the sequential processing of RNN / LSTM), significantly reducing the inference latency and meeting the millisecond-level response requirement from real-time music to action generation. By directly generating robot control instructions through coordinate mapping, skipping the traditional intermediate encoding link and reducing the length of the computational chain, the real-time performance is further improved.

[0279] 13. By setting up the two-axis pooling layer (frequency axis + time axis) of the music encoder to compress the Mel spectrogram, while retaining the key audio information, the data dimension is reduced, the model computational amount is decreased, and it adapts to the real-time processing scenario.

[0280] 14. By setting the dual-path feature extraction (dynamic + spatial) of the action discriminator, the generator is forced to learn both the temporal rationality of the action and the spatial coordination of joint movements simultaneously, avoiding the generation of mechanical jitters or abnormal movements that violate ergonomics.

[0281] 15. By segmenting historical music, the long audio is cut into segments aligned with the action video, ensuring that the model learns the local correspondence between music segments and action segments, rather than global fuzzy associations, enhancing the detail accuracy of the generated actions.

[0282] 16. By setting the combination of group convolution and one-dimensional convolution of the action discriminator, while reducing the number of parameters, the spatio-temporal characteristics of joint movements are retained, avoiding the high computational cost of traditional 3D convolution.

[0283] 17. Through an innovative multi-modal deep learning architecture (music encoder + action generator + fusion layer), high-precision real-time synchronization of music and robot actions is achieved. The music encoder uses a three-stage downsampling module and a biaxial pooling layer to extract compressed Mel spectrogram features. The action generator is based on a temporal convolutional network (TCN) and dilated one-dimensional convolution to achieve low-latency action generation. At the same time, a spatio-temporal graph convolutional network (ST-GCN) and a dual-path discriminator (dynamic + spatial features) are combined to ensure the spatio-temporal rationality of the actions and their fit with the music beats. At the training level, the generalization ability of the model is improved through segmented audio alignment, data augmentation, and dynamic expansion strategies. During deployment, dynamic pruning and containerization technologies are used to achieve lightweight adaptation. Finally, robot control instructions are directly generated through joint coordinate mapping, significantly optimizing the real-time response efficiency, and can be extended to multiple scenarios such as music conducting, dance robots, and virtual reality interactions, combining technological innovation and engineering practicality.

[0284] Although the specific implementation manners of the present invention have been described above, those skilled in the art of this technology should understand that the specific embodiments we described are illustrative only and not used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope protected by the claims of the present invention.

Claims

1. A robot motion control method based on deep learning, characterized in that: The steps are as follows: Step S1: Create a robot action generation model based on a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot action generation model; The music encoder is used to extract audio features from the input music; The action encoder is used to extract the action features of the conducting actions from the input action video, and train the music encoder based on the action features to enable the music encoder to learn the synchronization relationship between the music and the conducting actions; the action generator is used to output the conducting actions according to the audio features extracted by the music encoder; the action discriminator is used to determine whether the conducting actions output by the action generator are real actions or simulated actions; the fusion layer is used to determine the synchronization of the audio features extracted by the music encoder and the action features extracted by the action encoder; Step S2: Obtain a large number of historical music and the corresponding historical action videos, and construct a data set based on each piece of historical music and the historical action videos; Step S3: Divide the data set into a training set, a validation set, and a test set, train the robot action generation model through the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing through the validation set and the test set respectively; Step S4: Deploy the robot action generation model that passes the test; Step S5: Collect real-time music, preprocess the real-time music and then input it into the deployed robot action generation model to obtain the conducting actions carrying the two-dimensional coordinates of each joint point, perform coordinate mapping on the conducting actions to obtain robot control instructions, and control the robot to perform actions matching the real-time music based on the robot control instructions.

2. The method for controlling the actions of a robot based on deep learning according to claim 1, characterized in that: In the step S1, the music encoder is constructed based on a three-level downsampling module, and each downsampling module is constructed based on three residual blocks and a two-axis pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and the time axis of the two-dimensional Mel spectrogram to extract audio features; The action encoder is used to extract the action features including spatial structure features and temporal dynamic features of the conducting actions from the input action video through a spatio-temporal graph convolutional network; The action generator is constructed based on a temporal convolutional network, and the temporal convolutional network includes six dilated one-dimensional convolutional layers with residual connections, and each dilated one-dimensional convolutional layer includes 64 channels and a convolutional kernel size of 5; The action discriminator is constructed based on a joint dynamic feature extraction module, a joint spatial feature extraction module, and a discriminant module; the joint dynamic feature extraction module is used to extract joint dynamic features for characterizing the dynamic changes of joints from the conducting actions output by the action generator through group convolution; the joint spatial feature extraction module is used to extract joint spatial features for characterizing the spatial relationship between joints from the conducting actions output by the action generator through one-dimensional convolution; the discriminant module is used to reason about the joint dynamic features and the joint spatial features through a fully connected layer and output the discriminant result of whether the conducting actions are real actions or simulated actions.

3. The method for controlling a robot's actions based on deep learning according to claim 1, characterized in that: The specific steps of step S2 are as follows: Obtain a large number of historical music and historical action videos corresponding to the historical music. Perform preprocessing on each piece of historical music, including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement. Perform preprocessing on each historical action video, including at least video cleaning, video conversion, and video enhancement. Perform annotation of conducting actions on the preprocessed historical music and historical action videos, and construct a dataset based on the annotated historical music and historical action videos.

4. The method for controlling the movement of a robot based on deep learning according to claim 1, wherein: The specific steps of step S3 are as follows: Divide the dataset into a training set, a validation set, and a test set according to a ratio of 7:2:

1. Train the robot action generation model using the training set. During the training process, continuously optimize the hyperparameters of the robot action generation model until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition. Validate the trained robot action generation model using the validation set. Determine whether the action accuracy is greater than a preset accuracy threshold. If not, the validation fails, and the training set is expanded and training continues. If so, the validation passes, and: Test the robot action generation model that has passed the validation using the test set. Determine whether the F1 score is greater than a preset score threshold. If not, the test fails, and the training set is expanded and training continues. If so, the test passes, and the training ends.

5. The method for controlling the actions of a robot based on deep learning according to claim 1, characterized in that: The specific steps of step S4 are as follows: Perform compression operations of quantization and dynamic pruning on the robot action generation model that has passed the test, and deploy the compressed robot action generation model to the robot through containerization technology.

6. A robot motion control system based on deep learning, characterized in that: It includes the following modules: A robot action generation model creation module, which is used to create a robot action generation model based on a music encoder, an action encoder, an action generator, an action discriminator, and a fusion layer, and set the loss function of the robot action generation model. The music encoder is used to extract audio features from the input music. The action encoder is used to extract action features of conducting actions from the input action video, and train the music encoder based on the action features to enable the music encoder to learn the synchronization relationship between music and conducting actions. The action generator is used to output conducting actions according to the audio features extracted by the music encoder. The action discriminator is used to determine whether the conducting actions output by the action generator are real actions or simulated actions. The fusion layer is used to determine the synchronization between the audio features extracted by the music encoder and the action features extracted by the action encoder. A dataset construction module, which is used to obtain a large number of historical music and historical action videos corresponding to the historical music, and construct a dataset based on each historical music and historical action video. A robot action generation model training module, which is used to divide the dataset into a training set, a validation set, and a test set, train the robot action generation model using the training set until the loss value of the loss function is less than a preset loss threshold, and then perform validation and testing using the validation set and the test set respectively. A robot action generation model deployment module for deploying the robot action generation model that has passed the test; A robot action control module for collecting real-time music, preprocessing the real-time music and then inputting it into the deployed robot action generation model to obtain a conducting action carrying the two-dimensional coordinates of each joint point, performing coordinate mapping on the conducting action to obtain a robot control instruction, and controlling the robot to execute an action matching the real-time music based on the robot control instruction.

7. The robot motion control system based on deep learning according to claim 6, characterized in that: In the robot action generation model creation module, the music encoder is constructed based on a three-level downsampling module, and each downsampling module is constructed based on three residual blocks and a biaxial pooling layer, and is used to convert the input music into a two-dimensional Mel spectrogram, and perform downsampling on the frequency axis and the time axis of the two-dimensional Mel spectrogram to extract audio features; The action encoder is used to extract action features including spatial structure features and temporal dynamic features of the conducting action from the input action video through a spatio-temporal graph convolutional network; The action generator is constructed based on a temporal convolutional network, and the temporal convolutional network includes six dilated one-dimensional convolutional layers with residual connections, and each dilated one-dimensional convolutional layer includes 64 channels and a convolutional kernel size of 5; The action discriminator is constructed based on a joint dynamic feature extraction module, a joint spatial feature extraction module, and a discrimination module; the joint dynamic feature extraction module is used to extract joint dynamic features for characterizing joint dynamic changes from the conducting action output by the action generator through group convolution; the joint spatial feature extraction module is used to extract joint spatial features for characterizing the spatial relationship between joints from the conducting action output by the action generator through one-dimensional convolution; the discrimination module is used to infer the joint dynamic features and the joint spatial features through a fully connected layer and output a discrimination result as to whether the conducting action is a real action or a simulated action.

8. A robot motion control system based on deep learning according to claim 6, characterized in that: The dataset construction module is specifically used for: Obtaining a large number of historical music and historical action videos corresponding to the historical music, performing preprocessing on each historical music including at least sampling rate conversion, normalization, denoising, segmentation, feature extraction, and feature enhancement, performing preprocessing on each historical action video including at least video cleaning, video conversion, and video enhancement, performing annotation of conducting actions on the preprocessed historical music and historical action videos, and constructing a dataset based on the annotated historical music and historical action videos.

9. The robot motion control system based on deep learning according to claim 6, characterized in that: The robot action generation model training module is specifically used for: Dividing the dataset into a training set, a validation set, and a test set according to a ratio of 7:2:1, training the robot action generation model through the training set, and continuously optimizing the hyperparameters of the robot action generation model during the training process until the loss value of the loss function is less than a preset loss threshold or meets a preset early stopping condition; Validating the trained robot action generation model through the validation set, judging whether the action accuracy is greater than a preset accuracy threshold, if not, the validation fails, and the training set is expanded and training continues; if so, the validation passes, and: Test the verified robot motion generation model with the test set to determine whether the F1 score is greater than a preset score threshold. If not, the test fails and the training set is expanded for continued training. If so, the test passes and the training ends.

10. A robot motion control system based on deep learning according to claim 6, characterized in that: The robot motion generation model deployment module is specifically used for: Perform compression operations of quantization and dynamic pruning on the robot motion generation model that has passed the test, and deploy the compressed robot motion generation model to the robot through containerization technology.

Images