A somatic intelligent humanoid robot remote control data acquisition and data closed loop system

Abstract

The present application belongs to the technical field of embodied intelligent robots, and discloses a kind of embodied intelligent humanoid robot remote control data acquisition and data closed loop system, and the instruction, action, acquisition ternary time sequence synchronization mechanism is built by time sequence cooperation module, the dynamic cooperation of three is realized using time sensitive network technology, effectively reduce time sequence deviation, while combining the space-time alignment algorithm of data processing module, guarantee the consistency of multi-modal data in time and space dimension;Make the collected data can accurately fit human operation logic and robot action state, improve the optimization efficiency of data closed loop, successfully adapt to the demand of complex assembly, flexible material processing and other high-precision operation scene;Rely on evaluation module to establish the quantitative correlation system of data quality and control effect, can real-time filter invalid data such as sensor failure, action fuzzy, reduce the occupation of redundant storage and computing resources.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of embodied intelligent robots, and particularly relates to a remote control data acquisition and data closed loop system for an embodied intelligent humanoid robot. BACKGROUND

[0002] The remote control data acquisition and data closed loop system for the embodied intelligent humanoid robot guides the robot to complete a task through human remote control, synchronously acquires multi-modal data, and forms a complete closed loop of acquisition, training, optimization and application, thereby providing high-quality data guarantee for improving the autonomous ability of the robot. The current related technology still has the following technical problems in complex scene adaptation and data value mining:

[0003] In the existing system, remote control instruction transmission, robot action execution and multi-source sensor data acquisition are mostly independent time sequence links, and lack a dynamic coordination mechanism. In a high-precision operation scene such as complex assembly and flexible material processing, the execution delay of the remote control instruction will deviate from the time sequence of the sensor data acquisition, so that the acquired data is difficult to accurately match the human operation intention and the robot action state, and the operation and data are misaligned. This will make the action model trained based on the acquired data difficult to reproduce the real operation logic, thereby reducing the optimization efficiency of the data closed loop. In addition, the existing technology mainly focuses on the delay optimization of a single link, and does not carry out systematic design on the coordination time sequence of the instruction, action and data.

[0004] The current remote control data acquisition system mostly only completes data acquisition and storage, and does not establish a real-time feedback mechanism between data quality and control strategy. The effectiveness of the acquired data can only be verified during subsequent offline training, resulting in a large amount of storage and computing resources occupied by invalid data. At the same time, the optimization strategy obtained through training cannot be used to adjust the remote control parameters in real time, so that the remote control process always relies on manual experience and it is difficult to continuously improve the precision through data closed loop. This restricts the rapid iteration of the embodied intelligent robot from remote control demonstration to autonomous operation, especially in a dynamic unknown environment, the remote control control strategy cannot be optimized through real-time data feedback, which ultimately leads to a decrease in the success rate of operation. SUMMARY

[0005] The present application aims to provide a remote control data acquisition and data closed loop system for an embodied intelligent humanoid robot to solve the problems raised in the background.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical solution: a remote control data acquisition and data closed loop system for an embodied intelligent humanoid robot, comprising a time sequence coordination module, a data processing module, a transmission module, an evaluation module, a driving module, an optimization module and a fault-tolerant adaptive module.

[0007] Preferably, the timing coordination module receives multi-modal instructions output by the operator through various types of remote control terminals such as VR devices, force feedback gloves, motion capture systems, etc., adopts a multi-modal instruction fusion algorithm based on a lightweight neural network to complete instruction noise filtering and semantic analysis, and distinguishes between valid instructions and interference signals; based on a real-time identification model of robot dynamics, an event triggering mechanism is introduced, adaptive motion adaptation rules are generated according to the dynamic characteristics of the operator's motion amplitude and speed, and the conversion of human joint space motion to robot joint space motion is realized, supporting plug-and-play of different remote control terminals through standardized interface design;

[0008] The multi-modal instruction fusion algorithm is prioritized according to the order of "pose instruction > force control instruction > voice instruction", the pose instruction and the force control instruction are weighted and fused, the weight proportions are 0.5 and 0.3 respectively, the voice instruction is used as an auxiliary correction signal, and the weight proportion is 0.2. In the fusion process, the fluctuations of different modal instructions are smoothed through a sliding window to ensure the stability of the output instruction.

[0009] The triggering condition of the event triggering mechanism is that the operator's motion amplitude change amount per unit time is ≥15° or the motion speed change amount is ≥0.5 m / s, and the adaptive motion adaptation rule is updated every 20 ms after triggering; the adaptation rule includes joint angle mapping coefficient and motion smoothing transition parameter, wherein the joint angle mapping coefficient is dynamically calculated according to the proportion of robot degrees of freedom and human joint activity range.

[0010] A three-element timing synchronization engine is constructed for instructions, actions, and data collection, and time-sensitive network technology is used to realize timing alignment of remote control instruction sending, robot motion execution, and sensor data collection, real-time monitoring of job scene complexity and dynamic adjustment of timing synchronization threshold; at the same time, the timing synchronization signal is synchronized to the data processing module.

[0011] The job scene complexity is quantified by multiple dimensions, and the core indicators include operation precision requirement (such as millimeter level / centimeter level), motion continuity (number of motion switches per unit time), and environmental dynamics (obstacle moving speed / number), with weight proportions of 0.4, 0.3, and 0.3 respectively. The quantification result is a numerical value of 0-10 points, which is used for precise adaptation of the timing synchronization threshold.

[0012] The lightweight neural network adopts a lightweight architecture that combines CNN and LSTM, only retains the core feature extraction layer and semantic analysis layer, and ensures that the instruction processing delay is ≤20 ms. The dynamic adjustment of the timing synchronization threshold is based on the quantification value of the job scene complexity, the scene complexity is calculated by weighting the operation precision requirement (such as ±0.1 mm for high precision scene) and the motion continuity (such as motion switch frequency ≥3 times per second for high complexity), and the threshold value is adaptively matched in the range of 1 ms-5 ms, the threshold value of high precision scene is ≤2 ms, and the threshold value of regular scene is ≤5 ms.

[0013] Preferably, the data processing module receives the timing synchronization signal of the timing coordination module, integrates the robot body vision sensor, tactile sensor, inertial sensor and environment sensor, and constructs a multi-source heterogeneous data acquisition network; the real-time synchronous acquisition of multi-modal data is realized by using the space-time alignment algorithm;

[0014] Combined with the remote operation instruction priority and the dynamic complexity evaluation results of the operation scene output by the timing coordination module, the acquisition frequency of each sensor is dynamically adjusted through the reinforcement learning algorithm; a data quality pre-evaluation sub-unit is set, and based on the pre-set multi-dimensional indexes such as action consistency and environment recognition, the collected data is filtered in real time, and invalid data such as sensor fault data and action fuzzy data is removed.

[0015] The remote operation instruction priority is divided into three levels, the first priority is emergency operation, such as stop and emergency stop; the second priority is core operation, such as grabbing and assembling; the third priority is auxiliary operation, such as position fine adjustment and attitude adjustment; the priority is distinguished by the priority identification field in the instruction header, and the data processing module responds to the sensor data acquisition demand corresponding to the first and second instructions.

[0016] The reward function of the reinforcement learning takes the data quality compliance rate (weight 0.6), system energy consumption (weight 0.2) and data acquisition real-time performance (weight 0.2) as the core indexes, gives positive reward when the data quality meets the standard, and gives negative punishment when the energy consumption exceeds the standard or the acquisition delay exceeds the threshold.

[0017] The effective data is standardized and the features are preliminarily extracted, and the structured data block is generated and transmitted to the transmission module.

[0018] The feature preliminary extraction extracts core features for different sensor data, the vision sensor extracts target contour, key positioning point and distance information; the tactile sensor extracts contact pressure distribution, contact area and interaction force peak value; the inertial sensor extracts joint angular velocity, acceleration and attitude angle change; the environment sensor extracts obstacle distance, environment light intensity and temperature and humidity data.

[0019] The space-time alignment algorithm adopts time stamp interpolation calibration and space coordinate system double-step implementation, the time dimension is completed by linear interpolation of sensor data time stamp difference, and the space dimension is mapped to a unified coordinate system with the robot base coordinate system as the reference;

[0020] In the multi-dimensional data quality pre-evaluation index, the action consistency judgment threshold is that the joint angle deviation between the actual action and the instruction action is ≤3°, the environment recognition degree judgment threshold is that the image definition is ≥85% and the laser radar point cloud density is ≥100 points / cm², and the data that does not meet the threshold is determined as invalid data.

[0021] Preferably, the transmission module receives the structured data block output by the data processing module and the instruction transmission requirement of the timing coordination module, adopts an edge computing architecture, locally deploys a lightweight data transmission node on the robot, sinks part of the data preprocessing task to the edge end, and realizes edge preprocessing and low-delay transmission of the remote control instruction and collected data.

[0022] The edge preprocessing includes three types of operations: data cropping, outlier removal, and feature simplification. The data volume after preprocessing is reduced by 30-40%, ensuring low-delay transmission. Data cropping refers to removing redundant fields of the sensor; outlier removal refers to filtering data that exceeds the reasonable range based on the 3σ principle; feature simplification specifically refers to retaining core features related to the task and discarding secondary features.

[0023] At the transmission level, a lightweight transmission protocol is adopted, combined with an efficient data compression algorithm to reduce bandwidth occupancy; at the verification level, a bidirectional intelligent verification mechanism is constructed to perform integrity and effectiveness double verification on the issued remote control instruction, identify instruction loss or error through hash verification technology, and perform secondary quality verification on the uploaded collected data, combined with sensor working state data to comprehensively judge the data reliability.

[0024] The lightweight transmission protocol is based on the optimization design of the UDP protocol, which trims the redundant fields of the protocol header and only retains the address identifier, data length, and verification code core fields. The data payload adopts a compact format of "sensor type + timestamp + core data".

[0025] A transmission state monitoring subunit is set to monitor key indicators such as link bandwidth and delay in real time. When transmission anomalies are detected, it automatically switches to a backup transmission link and transmits the verified data and instructions to the evaluation module and the driving module, respectively.

[0026] The efficient data compression algorithm adopts a differential compression architecture for sensor time series data, only storing the difference between adjacent data and key feature points, with a compression ratio controlled between 3:1 and 5:1. The link key indicator monitoring threshold is set as follows: bandwidth below 1 Mbps, delay above 50 ms, and packet loss rate above 1% to determine transmission anomalies, triggering redundant link switching, with a backup link switching time ≤10 ms.

[0027] Preferably, the evaluation module receives the verified multi-source collected data transmitted by the transmission module, adopts a cross-modal attention mechanism to deeply mine the associated features between different modal data, realizes efficient fusion of multi-modal data, and generates a unified robot task state description vector. This vector can depict the robot action posture, environmental interaction force, and surrounding environment features.

[0028] The job state description vector is a 128-dimensional vector, including 32-dimensional action posture features, 32-dimensional environmental interaction force features, 32-dimensional environmental features, and 32-dimensional state evaluation features, and the values of each dimension of the vector are normalized to the interval [0, 1]; the action posture features include joint angles, end effector posture, etc.; the environmental interaction force features include contact point pressures, force magnitudes and directions, etc.; the environmental features include obstacle distribution, work space size, lighting conditions, etc.; and the state evaluation features include data quality scores, action completion progress, device operating status, etc.

[0029] The cross-modal attention mechanism preferentially focuses on modal features related to core job requirements, such as in assembly jobs, preferentially allocating attention weights (total proportion ≥ 70%) of tactile sensors (interaction force data) and visual sensors (positioning data); and in environmental perception scenarios, increasing the weight of lidar data (proportion ≥ 50%).

[0030] The fused data is further denoised by a deep learning model to filter out noise caused by environmental interference, while extracting key features and enhancing the representation ability of the data;

[0031] The deep learning denoising model uses an autoencoder architecture, the input layer receives the fused multi-modal data, the hidden layer separates the signal and noise through the encoding-decoding process, and the output layer outputs the denoised data. In the model training process, the goal is to improve the data signal-to-noise ratio, and to focus on suppressing signal distortion caused by environmental electromagnetic interference and sensor inherent noise.

[0032] A multi-dimensional data quality evaluation model is established to quantitatively correlate data quality with actual robot control effectiveness, output quality, qualified, and poor quality grading results of three levels, and accurately label the reasons for poor quality data. The high-quality and qualified data and quality evaluation results are transmitted to the driving module and the optimization module.

[0033] The specific dimensions of the multi-dimensional data quality evaluation model include data integrity (loss rate ≤ 5%), time sequence consistency (deviation ≤ 5ms), signal-to-noise ratio (≥ 30dB), and control effect matching degree (action accuracy deviation ≤ 0.5mm, job success rate ≥ 80%), and the weights of each dimension are 0.2, 0.2, 0.1, and 0.5, respectively. After weighted calculation, scores ≥ 85 are high-quality data, scores 60-84 are qualified data, and scores < 60 are poor-quality data.

[0034] Preferably, the driving module receives the job state description vector and data quality evaluation results output by the evaluation module, and simultaneously receives the instruction conversion results of the time sequence coordination module, constructs a dynamic weight distribution model, and realizes real-time distribution of the weights of remote control and robot autonomous control;

[0035] A collaborative control model is constructed using deep reinforcement learning algorithms. The control strategy is optimized through scenario training, which improves the autonomous control weight in simple repetitive task scenarios and the remote control weight in complex or dynamically unknown scenarios. At the same time, a model predictive control algorithm is introduced to optimize the robot joint control parameters based on the prediction results of the future short-term task state.

[0036] The model predictive control algorithm first predicts the robot joint motion trajectory for the next 3-5 control cycles based on historical operation data and the current state; then, by comparing the deviation between the predicted trajectory and the expected trajectory, it optimizes control parameters such as joint stiffness and damping coefficient in reverse; finally, it optimizes the parameters in the rolling time domain with the time domain length consistent with the prediction cycle to ensure the real-time performance and adaptability of parameter adjustments.

[0037] The state space of the deep reinforcement learning includes four core parameters: task scenario complexity score, data quality level, robot action deviation value, and remaining task quantity; the action space is the weight adjustment step size of remote control and autonomous control, with each adjustment ranging from 5% to 10%.

[0038] A real-time motion deviation correction subunit is set up. By frequently comparing the deviation between the actual working state and the expected state, an adaptive correction command is generated and fed back to the execution end to further improve the control accuracy. At the same time, the control effect data is fed back to the optimization module.

[0039] The weight calculation of the dynamic weight allocation model is based on the complexity score of the operation scenario (0-10 points) and the data quality level. When the scenario is simple (complexity ≤ 3 points) and the proportion of high-quality data is ≥ 70%, the autonomous control weight is 60%-80%; when the scenario is complex (complexity ≥ 7 points) or the proportion of poor-quality data is ≥ 30%, the remote control weight is 60%-90%. The prediction time domain of the model predictive control is set to 5-10 control cycles, the control time domain is set to 3-5 control cycles, and the action deviation correction response time is ≤ 10ms.

[0040] Preferably, the optimization module receives high-quality data and quality labeling results output by the evaluation module and control effect data fed back by the drive module, generates data labels through an automated labeling algorithm, and stores them in a structured training dataset for incremental training of the robot motion model.

[0041] The incremental training adopts a small-batch training mode, triggering training once every 500 high-quality data points. During the training process, the bottom feature extraction layer of the model is frozen, and only the parameters of the top decision layer are fine-tuned. After training is completed, the model performance is verified (comparing the core indicators before and after training). If the verification is successful, the model parameters are updated; if it fails, the original parameters are retained and training data is supplemented.

[0042] The automatic labeling algorithm has a built-in job type-action feature mapping rule library, which covers typical jobs such as assembly, grabbing and placing. By matching the joint angle sequence in the sensor data, the peak value of the interaction force and other action features with the rule library entries, structured labels are automatically generated, and abnormal feature data triggers manual review marking.

[0043] The optimized model parameters obtained by training are fed back to the time sequence coordination module in real time to optimize and adapt the rules and time sequence synchronization threshold values, and are fed back to the driving module to update the control weights and model predictive control parameters; the reasons for the poor data labeling are analyzed in depth, and targeted acquisition parameter adjustment suggestions are generated in combination with scene features, which are fed back to the data processing module and the time sequence coordination module for parameter updating;

[0044] A closed-loop effect evaluation subunit is arranged to regularly count core indicators such as job success rate, action accuracy and system response speed, establish an evaluation model and dynamically optimize the closed-loop iteration period and training parameters.

[0045] The label format of the automatic labeling algorithm is “job type-action feature-quality level-acquisition scene”, and the structured training data set is stored according to the scene type; the dynamic optimization of the closed-loop iteration period is based on the core indicator compliance rate, when the job success rate is ≥95% and the action accuracy deviation is ≤0.3 mm, the iteration period is extended to 1 hour / time, and when it is not up to standard, it is shortened to 10 minutes / time.

[0046] Preferably, the fault-tolerant adaptive module monitors the running state of the aforementioned modules, including transmission link bandwidth, delay, sensor working state, module computing power occupation, robot action execution accuracy and other key indicators, and constructs a multi-dimensional and all-round state monitoring matrix; a fault prediction model based on machine learning is adopted to analyze historical operation data and real-time state data to give early warning of potential faults;

[0047] The multi-dimensional state monitoring matrix includes 6 core dimensions, namely transmission link, sensor state, module computing power, action execution, data quality and environment state, and each dimension data acquisition frequency is 10-100 Hz and is set according to importance. The transmission link dimension includes bandwidth, delay and packet loss rate; the sensor state dimension includes working voltage, data output frequency and deviation value; the module computing power dimension includes CPU occupancy, memory usage and task processing time consumption; the action execution dimension includes joint angle deviation, end position accuracy and action completion time; the data quality dimension includes integrity, time sequence consistency and signal-to-noise ratio; and the environment state dimension includes temperature, humidity and electromagnetic interference intensity.

[0048] The fault prediction model adopts a gradient boosting decision tree algorithm, and input features include mean, variance, mutation times of sensor data, sliding average, volatility coefficient of link delay, and key features such as duration of module computing power occupation, peak proportion, and the model prediction lead time is 500ms-1s.

[0049] When a module fault or performance degradation is detected, a hierarchical fault-tolerant strategy is automatically triggered, adjustment instructions are issued to the data processing module when a slight fault occurs, and redundant sensors are enabled, and when a serious fault occurs, switching instructions are issued to the drive module, switching to a safe control mode, stopping unnecessary work and timely feeding back manual intervention, recording fault data and fault-tolerant adjustment process, and generating a standardized fault analysis report.

[0050] The input feature dimension of the fault prediction model includes sensor data deviation rate (>=10% is abnormal), link delay fluctuation amplitude (>=20ms is abnormal), module computing power occupation rate (>=90% is abnormal), and action execution precision deviation (>=1mm is abnormal).

[0051] The fault level division standard is: slight fault (single feature abnormality and does not affect work), serious fault (2 or more feature abnormalities or has caused work deviation >=2mm), when a slight fault occurs, instructions are issued to the data processing module to reduce the non-critical data acquisition frequency, and at the same time, redundant sensors are enabled, and when a serious fault occurs, non-core work processes are preferentially interrupted.

[0052] The beneficial effects of the present application are as follows:

[0053] 1、The present application builds a three-element time sequence synchronization mechanism of instructions, actions and collection through a time sequence coordination module, realizes the dynamic coordination of the three by using time-sensitive network technology, effectively reduces the time sequence deviation, and at the same time, combines the space-time alignment algorithm of the data processing module to ensure the consistency of multi-modal data in the time and space dimensions; solves the problem of mispositioning of data and operation intention caused by independent time sequence of the three in the existing system and lack of cooperation, so that the collected data can accurately match the human operation logic and the robot action state, improves the optimization efficiency of the data closed loop, and successfully adapts to the needs of complex assembly, flexible material processing and other high-precision operation scenes.

[0054] 2,The application relies on the evaluation module to establish a quantitative correlation system of data quality and control effect, can real-time filter invalid data such as sensor failure and action ambiguity, reduce the occupation of redundant storage and computing resources;The driving module is based on the data quality evaluation result, adjusts the weight proportion of remote control and autonomous control in real time through a dynamic weight distribution model, strengthens autonomous control in simple repetitive operation to reduce artificial load, improves remote control in complex or dynamic unknown scene to ensure operation safety, and further optimizes control accuracy through real-time deviation correction;Break the limitations of offline verification data and dependence on artificial remote control in the prior art, realize real-time linkage of data quality and control strategy, accelerate the iterative process of robot from remote control demonstration to autonomous operation, and improve the operation success rate in dynamic environment.

[0055] 3,The application transmits module adopts edge computing architecture and lightweight transmission protocol to ensure low-delay transmission of data and instructions, cooperates with a bidirectional intelligent checking mechanism and redundant link switching, effectively avoids instruction errors and data loss, and improves the stability of the transmission link;The fault-tolerant adaptive module constructs a multi-dimensional state monitoring matrix, realizes early warning of potential faults by using machine learning, and combines a hierarchical fault-tolerant strategy to enable redundant sensors in slight faults and switch to a safe control mode in serious faults, so that the core function is not affected;The optimization module uses high-quality data to complete incremental training of the model, feeds back the optimization parameters to each related module in real time, adjusts the acquisition parameters and control strategy, and evaluates the dynamic optimization closed-loop operation parameters through core indicators, so as to continuously improve the operation accuracy, response efficiency and other performances of the system, and adapt to diversified application requirements. BRIEF DESCRIPTION OF DRAWINGS

[0056] Fig. 1 The system flowchart of the application is shown in the figure;

[0057] Fig. 2 The data acquisition and timing coordination flowchart of the application is shown in the figure;

[0058] Fig. 3 The data transmission and quality evaluation flowchart of the application is shown in the figure;

[0059] Fig. 4 The control optimization and fault-tolerant adaptation flowchart of the application is shown in the figure. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0061] As Figs. 1 to 4As shown, the embodiment of the present application provides a body-intelligent humanoid robot remote control data acquisition and data closed loop system, which comprises a time sequence coordination module, a data processing module, a transmission module, an evaluation module, a driving module, an optimization module and a fault-tolerant adaptive module, and the specific implementation of each module is as follows:

[0062] Among them, the time sequence coordination module receives multi-modal instructions output by the operator through VR devices, force feedback gloves, motion capture systems and other multi-type remote control terminals, and the multi-modal instructions include posture instructions, force control instructions and voice instructions; a multi-modal instruction fusion algorithm based on a lightweight neural network is adopted to complete instruction noise filtering and semantic analysis, distinguish effective instructions from interference signals, and improve instruction recognition accuracy; based on a real-time identification model of robot dynamics, an event triggering mechanism is introduced, adaptive motion adaptation rules are generated according to the dynamic characteristics of the operator's motion amplitude and speed, and accurate conversion from human joint space motion to robot joint space motion is realized; through standardized interface design, plug-and-play of different remote control terminals is supported, and system deployment cost is reduced.

[0063] The standardized interface is designed based on the ROS (Robot Operating System) communication protocol, contains three types of data input interface, instruction output interface and state feedback interface, adopts a unified data format (JSON format), defines fields clearly, contains device ID, instruction type, data length, timestamp and check code, and supports plug-and-play adaptation of common remote control terminals such as VR devices and force feedback gloves.

[0064] The real-time identification model of robot dynamics takes robot joint torque, angular velocity and angular acceleration as input characteristics, updates model parameters through online recursive algorithm, adapts to differences in robot load changes and joint friction characteristics, and supports real-time calibration of dynamics parameters of 6-24 degree-of-freedom humanoid robots.

[0065] A three-element time sequence synchronization engine of instruction, motion and acquisition is constructed, time sequence alignment of remote control instruction sending, robot motion execution and sensor data acquisition is realized through TSN (Time Sensitive Network) technology, time sequence deviation is not more than 5ms, real-time monitoring of job scene complexity and dynamic adjustment of time sequence synchronization threshold are realized to adapt to different scene requirements from normal operation to high-precision operation; at the same time, the time sequence synchronization signal is synchronized to the data processing module to ensure the cooperation of data acquisition and instruction execution.

[0066] Among them, the data processing module receives the time sequence synchronization signal of the time sequence coordination module, integrates robot body vision sensors (RGB-D cameras), tactile sensors (fingertip array sensors), inertial sensors (IMUs) and environmental sensors (laser radars), and constructs a multi-source heterogeneous data acquisition network; a space-time alignment algorithm is adopted to realize real-time synchronous acquisition of multi-modal data, and ensure the consistency of different sensor data in time and space dimensions;

[0067] In combination with the work scene dynamic complexity evaluation result output by the remote operation instruction priority and timing coordination module, the sensor collection frequency is dynamically adjusted through a reinforcement learning algorithm, for example, the tactile data collection frequency is increased to 100 Hz in a fine operation scene and reduced to 50 Hz in a regular scene, while ensuring data quality and reducing energy consumption; a data quality pre-evaluation subunit is set, based on multiple-dimensional indexes such as preset motion consistency and environment recognition, real-time filtering of collected data is performed, invalid data such as sensor fault data and motion fuzzy data is removed, and redundant storage pressure is reduced;

[0068] The core indexes of the work scene dynamic complexity evaluation are consistent with the timing coordination module, the quantitative results are synchronously obtained, in combination with the remote operation instruction priority, a dual decision basis for sensor collection frequency adjustment is formed, and accurate matching of frequency adjustment and scene demand is ensured.

[0069] The effective data is subjected to standardization processing and initial feature extraction, a structured data block is generated and transmitted to the transmission module.

[0070] The motion deviation is obtained by calculating the Euclidean distance between the actual angle and the expected angle of each joint of the robot and the spatial distance between the actual position of the end effector and the target position, the deviation calculation frequency is consistent with the sensor collection frequency, and the real-time performance of the deviation feedback is ensured.

[0071] The transmission module receives the structured data block output by the data processing module and the instruction transmission demand of the timing coordination module, adopts an edge computing architecture, locally deploys a lightweight data transmission node on the robot, sinks part of the data preprocessing tasks to the edge, shortens the data transmission path, and realizes edge preprocessing and low-delay transmission of the remote operation instruction and the collected data;

[0072] A lightweight transmission protocol is adopted at the transmission level, redundant transmission fields are removed, high-efficiency data compression algorithms are combined to reduce bandwidth occupation, the transmission delay of the remote operation instruction is ensured to be less than 50 ms, and the real-time control demand is met; a bidirectional intelligent verification mechanism is constructed at the verification level, the integrity and validity of the issued remote operation instruction are double-verified, the instruction loss or error is identified through a hash verification technology, robot misoperation is avoided, and the uploaded collected data is subjected to secondary quality verification, and the data reliability is comprehensively judged in combination with the sensor working state data;

[0073] A transmission state monitoring subunit is set, key indexes such as link bandwidth and delay are monitored in real time, when transmission abnormity is detected, the transmission is automatically switched to a backup transmission link, transmission redundancy is formed, the data and the instruction that pass the verification are respectively transmitted to the evaluation module and the driving module, and the stability and reliability of the entire data transmission link are ensured.

[0074] The evaluation module receives the multi-source collected data after verification transmitted by the transmission module, uses a cross-modal attention mechanism to deeply mine the associated features between different modal data, realizes efficient fusion of multi-modal data, and generates a unified robot operation state description vector, which can accurately depict the robot action posture, environmental interaction force and surrounding environment characteristics.

[0075] The fused data is further denoised by a deep learning model to filter out noise caused by environmental interference, while extracting key features and enhancing the representation ability of the data.

[0076] A multi-dimensional data quality evaluation model is established to quantitatively correlate data quality with actual robot control effect (e.g. action accuracy, operation success rate), output quality, qualified, and inferior quality grading results of three levels, and accurately label the reasons for inferior data, such as time sequence deviation, environmental interference, and non-standard action. The high-quality and qualified data and quality evaluation results are transmitted to the driving module and the optimization module.

[0077] The driving module receives the operation state description vector and data quality evaluation results output by the evaluation module, and simultaneously receives the instruction conversion results of the time sequence coordination module, constructs a dynamic weight distribution model, and realizes real-time and accurate distribution of the weights of remote control and robot autonomous control.

[0078] A collaborative control model is constructed using a deep reinforcement learning algorithm, and the control strategy is optimized through a large number of scene training. In simple and repetitive operation scenarios, the autonomous control weight is increased to a maximum of 80%, significantly reducing the workload of manual operation. In complex or dynamic unknown scenarios, the remote control weight is increased to not less than 60%, ensuring operation safety and accuracy. At the same time, a model predictive control algorithm is introduced to optimize the robot joint control parameters based on the prediction results of the future short-term operation state, effectively suppressing action jitter and ensuring action smoothness and accuracy.

[0079] An action deviation real-time correction subunit is set up to quickly generate adaptive correction instructions by comparing the actual operation state with the expected state at a high frequency, and the correction response time is not more than 10ms, further improving the control accuracy, and at the same time, the control effect data is fed back to the optimization module.

[0080] The optimization module receives the high-quality data and quality labeling results output by the evaluation module and the control effect data fed back by the driving module, generates data labels through an automatic labeling algorithm, and stores them in a structured training data set for incremental training of the robot action model, avoiding resource waste caused by full training.

[0081] The optimized model parameters obtained by training are fed back to the time sequence coordination module in real time to optimize and adapt the rules and time sequence synchronization threshold, improve the action conversion accuracy and coordination synchronization, and are fed back to the driving module to update the control weight and model predictive control parameters, optimize the coordination control effect, and realize dynamic iteration of the control strategy.

[0082] The labeled poor data reasons are analyzed in depth, and targeted acquisition parameter adjustment suggestions are generated combined with scene characteristics, such as adjusting the sensor acquisition frequency and optimizing the time sequence synchronization threshold, which are fed back to the data processing module and the time sequence coordination module for parameter updating.

[0083] A closed-loop effect evaluation subunit is set to regularly count core indicators such as job success rate, action accuracy, and system response speed, establish an evaluation model, dynamically optimize the closed-loop iteration period and training parameters, and ensure the efficiency and pertinence of the closed-loop optimization.

[0084] The fault-tolerant adaptive module monitors the running state of the aforementioned modules, including transmission link bandwidth, delay, sensor working state, module computing power occupation, robot action execution accuracy, and other key indicators, constructs a multi-dimensional and all-around state monitoring matrix, and realizes visual perception of the system running state; a fault prediction model based on machine learning is used to analyze historical running data and real-time state data to provide sufficient disposal time for operation and maintenance personnel by giving early warning of potential faults such as sensor attenuation, link congestion, and insufficient computing power;

[0085] When a module fault or performance decline is detected, a hierarchical fault-tolerant strategy is automatically triggered. When there is a slight fault, adjustment instructions are issued to the data processing module to reduce the non-critical data acquisition frequency and enable redundant sensors to ensure the normal operation of core functions. When there is a serious fault, switching instructions are issued to the driving module to switch to a safe control mode, stop unnecessary work, and timely feedback to human intervention to avoid the expansion of the fault. Fault data and fault-tolerant adjustment process are recorded in detail to generate a standardized fault analysis report.

[0086] It should be noted that, in this text, relationship terms such as first and second are only used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0087] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A somatic intelligent humanoid robot remote control data acquisition and data closed loop system, characterized in that, The system comprises a timing coordination module, a data processing module, a transmission module, an evaluation module, a driving module, an optimization module, and a fault-tolerant adaptive module. The timing coordination module adopts a multi-modal instruction fusion and robot dynamics identification adaptation technology, establishes a three-element timing synchronization mechanism for instructions, actions, and data collection, realizes timing alignment of remote control instruction conversion and transmission, action execution, and data collection, and synchronizes timing synchronization signals to the data processing module. The data processing module receives timing synchronization signals, constructs a multi-source heterogeneous data collection network based on multiple types of sensors, realizes synchronous data collection through space-time alignment and dynamic frequency adjustment, filters effective data through quality pre-evaluation, and standardizes the data to output structured data to the transmission module. The transmission module constructs a transmission link based on an edge computing architecture, verifies the validity of instructions and data through a two-way intelligent verification mechanism, monitors the link status in real time and realizes redundant switching. The evaluation module receives the verified multi-source data, uses a cross-modal fusion mechanism to mine data correlation features to generate job state description information, optimizes data quality, and establishes a quantitative correlation between data quality and control effect, and outputs graded data and evaluation results. The driving module establishes a dynamic weight distribution model based on job state information and data quality evaluation results, optimizes control strategies using reinforcement learning and predictive control algorithms, realizes the adaptation of remote control and autonomous control, improves control accuracy through real-time deviation correction, and feeds back control effect data to the optimization module. The optimization module completes incremental training of the model based on high-quality data, feeds back optimization parameters to the timing coordination module and driving module, analyzes the reasons for poor data and outputs collection parameter adjustment suggestions, and dynamically optimizes closed-loop operation parameters. The fault-tolerant adaptive module uses a fault prediction model to realize potential fault early warning, triggers a hierarchical fault-tolerant strategy based on fault levels, records fault data and fault-tolerant adjustment process, and generates an analysis report.

2. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 1, wherein, The timing coordination module receives multi-modal instructions output by multiple types of remote control terminals, uses a multi-modal instruction fusion algorithm to filter instruction noise and perform semantic analysis, and distinguishes between valid instructions and interference signals. The three-element timing synchronization mechanism adjusts the timing synchronization threshold dynamically to adapt to the complexity of the work scene, ensuring the timing alignment of remote control instruction sending, robot action execution, and sensor data collection.

3. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 2, wherein, The data processing module integrates robot body vision sensors, tactile sensors, inertial sensors, and environmental sensors, and realizes real-time synchronous collection of multi-modal data through space-time alignment algorithms. Based on the remote control instruction priority and the dynamic complexity evaluation results of the work scene, the collection frequency of each sensor is dynamically adjusted. Through the data quality pre-evaluation subunit, the collected data is real-time filtered based on multiple dimensions such as preset action consistency and environmental recognition, and invalid data is removed. The effective data is standardized and feature-extracted to generate structured data blocks.

4. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 3, wherein, The transmission module locally deploys a lightweight data transmission node on the robot to realize edge preprocessing and low-latency transmission of remote control instructions and collected data. A lightweight transmission protocol combined with a data compression algorithm is used to reduce bandwidth occupation; a two-way intelligent verification mechanism is used to complete double verification of the integrity and effectiveness of remote control instructions, identify instruction loss or errors, and perform secondary quality verification on uploaded collected data, and comprehensively judge the data reliability in combination with sensor working state data; The transmission state monitoring subunit monitors the key indicators of the link in real time, automatically switches to the backup transmission link when an exception occurs, and transmits the verified data and instructions to the evaluation module and the driving module, respectively.

5. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 4, wherein, The evaluation module uses a cross-modal attention mechanism to deeply mine the associated features between different modal data and generate a unified robot operation state description vector. The fused data is denoised and key features are extracted to enhance the representation ability of the data. A multi-dimensional data quality evaluation model is established to quantitatively correlate data quality with actual robot control effect, output quality, qualified, and inferior quality three levels of quality classification results, and label the reasons for inferior data.

6. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 5, wherein, The driving module constructs a dynamic weight distribution model based on the operation state description vector and data quality evaluation results output by the evaluation module, and the instruction conversion results output by the timing coordination module, to realize real-time distribution of remote control and robot autonomous control weights. A reinforcement learning algorithm is used to construct a cooperative control model, which dynamically adjusts the control weight according to the complexity of the operation scene; a model predictive control algorithm is introduced to optimize the robot joint control parameters; Through the action deviation real-time correction subunit, the deviation between the actual operation state and the expected state is compared to generate adaptive correction instructions feedback to the execution end, improving control accuracy.

7. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 6, wherein, The optimization module completes incremental training of the robot action model through high-quality data, and feeds back the optimization parameters to the timing coordination module and the driving module; Deeply analyze the reasons for inferior data, generate collection parameter adjustment suggestions combined with scene characteristics, and feedback to the corresponding module; Through the closed-loop effect evaluation subunit, periodically statistics core operation indicators are established to build an evaluation model, dynamically optimize the closed-loop iteration period and training parameters.

8. The embodied intelligent humanoid robot teleoperation data acquisition and data closed loop system of claim 7, wherein, The fault-tolerant adaptive module constructs a multi-dimensional state monitoring matrix to monitor key indicators such as link state, sensor working state, module computing power occupation, and action execution accuracy in real time; a fault prediction model is used to realize early warning of potential faults; Based on the fault level, trigger a hierarchical fault-tolerant strategy; when a slight fault occurs, enable redundant sensors; when a serious fault occurs, switch to a safe control mode; Record fault data and fault-tolerant adjustment process, generate a standardized fault analysis report.

Images