Robot-based data processing system

Abstract

The invention relates to the technical field of data processing, and discloses a data processing system based on a robot, which is used for improving the comprehensiveness and accuracy of data acquisition and fusion. The robot-based data processing system comprises a first robot used for generating and sending an operation instruction; the second robot is used for receiving and moving based on the operation instruction and collecting and sending various modal data in the moving process, and the various modal data comprise joint sensing data, visual sensing data and force sensing data; and the data processing platform is used for receiving the multiple modal data sent by the second robot, and performing fusion based on the multiple modal data to obtain multi-modal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing system based on a robot. BACKGROUND

[0002] In recent years, with the rapid development of robot technology and artificial intelligence technology, people begin to combine robot and artificial intelligence technology and apply it to fields or scenes such as manufacturing, service, home service, etc., which shows great application potential.

[0003] At present, most of the dual-arm robots usually only combine the joint action information of the mechanical arm and the visual information. Since the dual-arm robot obtains less information, there are some limitations, which will limit the comprehensiveness and accuracy of data acquisition and fusion. SUMMARY

[0004] The present application provides a data processing system based on a robot to solve the problem that the information obtained by the robot in the prior art is relatively single, which limits the comprehensiveness and accuracy of data acquisition and fusion.

[0005] The first aspect of the present application provides a data processing system based on a robot, comprising: a first robot, a second robot and a data processing platform, wherein the first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is in communication connection with the second robot; the first robot is used for generating and sending operation instructions; the second robot is used for receiving and moving based on the operation instructions, collecting and sending multiple modal data in the movement process, wherein the multiple modal data includes joint sensing data, visual sensing data and force sensing data; the data processing platform is used for receiving the multiple modal data sent by the second robot, and fusing based on the multiple modal data to obtain multi-modal data.

[0006] In a feasible implementation, the second robot includes a mechanical arm, a three-fingered hand, a plurality of joint sensors, a visual sensor and at least one force sensor, the plurality of joint sensors are arranged on the mechanical arm and the three-fingered hand, and the visual sensor and the plurality of force sensors are arranged on the three-fingered hand; the plurality of joint sensors are used to obtain a plurality of joint information of the mechanical arm and the three-fingered hand to obtain joint sensing data; the visual sensor is used to obtain the visual sensing data; and the at least one force sensor is used to obtain at least one force information of the three-fingered hand to obtain force sensing data.

[0007] In one feasible implementation, acquiring at least one force sensory information of the three-finger dexterous hand includes: acquiring first force sensory information of the palm of the three-finger dexterous hand; and acquiring second force sensory information of the fingers of the three-finger dexterous hand.

[0008] In one feasible implementation, if the force sensor is a piezoresistive force sensor including a distributed array of electrodes, then each force sensor acquires force information by: acquiring the resistance value of each electrode in the distributed array; determining the pressure value corresponding to each electrode based on the resistance value of each electrode; and calculating the corresponding force information based on the pressure value corresponding to each electrode. The force information calculation method is as follows:

[0009] Among them, FSD is the final force sensing data obtained. It is the pressure value corresponding to the j-th electrode in the distributed electrode arrangement of the piezoresistive force sensor. is the weighting coefficient associated with the j-th electrode, n is the total number of electrodes, and DF is the force direction factor calculated based on the pressure distribution of each electrode. The formula for calculating DF is:

[0010] in, Indicates the first j The position vector of each electrode in the sensor coordinate system Indicates the first j The pressure value of each electrode.

[0011] In one feasible implementation, acquiring multiple joint information of the robotic arm and the three-finger dexterous hand to obtain joint sensing data includes: acquiring multiple joint poses and multiple joint velocities of the robotic arm and the three-finger dexterous hand to obtain joint sensing data, wherein the joint sensing data is: ,in, This represents the joint sensing data output by the joint sensors of the second robot. These represent the joint motion poses output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the joint movement speeds output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the left and right arms of the second robot, respectively. This represents the q-th joint sensor at time t.

[0012] In an implementation, the visual sensor is a red-green-blue-depth camera, the acquiring visual sensor data comprises: acquiring an RGB image and a Depth image, outputting visual sensor data based on the RGB image and the Depth image, and the visual sensor data is: wherein, represents visual sensor data output by a visual sensor on the three-fingered hand, represents an RGB image, represents a Depth image, respectively represent a left arm and a right arm of the second robot, represents a cth visual sensor inside at time t.

[0013] In an implementation, the fusing based on the multiple modalities of data comprises: extracting kinematic features from the joint sensor data; extracting image features from the visual sensor data, wherein a formula for image feature extraction using a convolutional neural network is: wherein, VF represents extracted visual features, CNN represents a convolutional neural network, is image data acquired by a visual sensor; extracting force / tactile features from the force / tactile sensor data; fusing based on the kinematic features, the image features, and the force / tactile features, and a fusion formula is wherein, MMD represents multiple modalities of data obtained by final fusion; i = 1, 2, 3 in the formula, respectively represent kinematic features, image features, and force / tactile features, i = 1, 2, 3 in the formula are weighting coefficients of corresponding features, and .

[0014] In an implementation, the fusing based on the kinematic features, the image features, and the force / tactile features comprises: performing normalization processing on the kinematic features, the image features, and the force / tactile features; inputting the normalized kinematic features, image features, and force / tactile features into corresponding processing models respectively to obtain prediction results of each modality; and integrating the prediction results of each modality using a preset fusion strategy to obtain multiple modalities of data.

[0015] In an implementation, the data processing platform is further configured to analyze performance and behavior of the second robot based on the multiple modalities of data to obtain an analysis result, and a formula is: evaluating the robot performance, wherein PI is an integrated performance evaluation index of the second robot, Accuracy represents the accuracy of the robot in performing a task, MaxAccuracy is the maximum accuracy, Speed represents the design speed, MaxSpeed is the maximum design speed, represents a stability index, is the optimal stability index value, 、 、 is a weight coefficient, and + + = 1; and sending the analysis result to the first robot.

[0016] In an available embodiment, the movement based on the operation instruction comprises: converting the operation instruction into a movement parameter in a self joint space by a mapping formula through a joint space mapping method, and performing corresponding movement, the mapping formula being: J=

[0017] wherein J represents the converted joint space movement parameter, is an operation instruction, represents a preset joint mapping relationship parameter set, represents a conversion function.

[0018] In the technical solution provided by the application, the data processing system comprises: a first robot, a second robot and a data processing platform, wherein the first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is in communication connection with the second robot; the first robot is configured to generate and send an operation instruction; the second robot is configured to receive the operation instruction and move based on the operation instruction, collect and send multiple modal data in the movement process, the multiple modal data comprising joint sensing data, visual sensing data and force sensing data; and the data processing platform is configured to receive the multiple modal data sent by the second robot, and fuse the multiple modal data to obtain multimodal data. In the embodiment of the application, the first robot is introduced to control the second robot based on the preset joint mapping relationship, and the data processing platform is combined to realize the collection of multiple modal data including joint sensing data, visual sensing data and force sensing data in the movement process of the robot, thereby improving the comprehensiveness and accuracy of data collection and fusion. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 FIG. 1 is a schematic diagram of an embodiment of a robot-based data processing system in the application; Figure 2Another embodiment of the robot-based data processing system in the embodiments of the present application is shown in the figure. Figure 3 An embodiment of one arm of the second robot in the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] The embodiments of the present application provide a robot-based data processing system, which improves the comprehensiveness and accuracy of data acquisition and fusion by collecting various modal data including joint sensing data, visual sensing data and force sensing data.

[0021] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application and in the above drawings (if any) are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed as interchangeable in order to distinguish between similar objects. It is also to be understood that the use of the terms "including", "comprising" or "having", and variations thereof, are intended to cover the presence of one or more of the stated items, but not the exclusion of one or more of the stated items, e.g., a process, method, system, product or apparatus that comprises a list of steps or units is not necessarily limited to those steps or units specifically listed, but can include additional steps or units not expressly listed or inherent to such process, method, product or apparatus.

[0022] Referring to Figure 1 An embodiment of the robot-based data processing system in the embodiments of the present application includes: The first robot 101, the second robot 102 and the data processing platform 103, wherein the first robot controls the second robot based on a preset joint mapping relationship, the data processing platform is in communication connection with the first robot and the second robot, the first robot is used for generating and sending operation instructions, the second robot is used for receiving and moving based on the operation instructions, and various modal data is collected and sent during the movement, the various modal data includes joint sensing data, visual sensing data and force sensing data, the data processing platform is used for receiving the various modal data sent by the second robot, and the various modal data is fused based on the various modal data to obtain multi-modal data.

[0023] The operator inputs control instructions through the operation interface of the first robot or directly operates its mechanical arm. The first robot can be equipped with intuitive operation interfaces such as touch screens, buttons, and joysticks, etc. The operator can input control instructions through these interface elements. In addition to using the operation interface, the operator can also directly wear special equipment to operate the mechanical arm of the main dual-arm robot. The equipment can be a data glove or an exoskeleton, etc. Direct operation can achieve more intuitive and natural control. The first robot is built-in with high-precision sensors that can capture its own joint angles, positions, speeds, and other motion states in real time. The first robot converts these motion states into a series of digital signals, which contain the action information that the operator wants the second robot to execute. These digital signals are further formatted into operation instructions that the second robot can understand and execute. This process can perform coordinate transformation, scaling, etc. to ensure that the second robot can accurately mimic the actions of the first robot.

[0024] The first robot and the slave second robot interact through a predefined communication protocol. The protocol specifies the format of the instructions, the transmission method, and the error handling mechanism, etc. The operation instructions can be sent from the first robot to the slave second robot through wireless (such as Wi-Fi, Bluetooth) or wired (such as Ethernet, USB) methods. Wireless transmission provides greater flexibility and convenience, while wired transmission can provide more stable and high-speed data transmission.

[0025] The second robot integrates joint sensors, vision sensors, and force sensors. The joint sensors are used to obtain joint sensing data, which mainly reflects the kinematic parameters of the robot's joint angles, angular velocities, angular accelerations, etc. Through these kinematic parameters, the dynamic characteristics of the robot's pose and speed can be understood. The vision sensors are used to obtain vision sensing data, which includes multimedia information such as images and videos. These information provides an intuitive view of the robot's working environment, which helps the robot to perceive the environment and detect obstacles. Through vision sensing data, the robot can identify target objects, locate its own position, and navigate and avoid obstacles in complex environments. In addition, through vision sensing data, the working effect of the robot can also be evaluated, such as checking whether the plates have been successfully wiped clean, etc. The force sensors are used to obtain force sensing data, which reflects the force and torque that the robot receives during interaction with the environment. For example, in the process of wiping the plates, force sensing data can help the robot control the pressure of the sponge against the plates, avoiding excessive pressure that can damage the plates or insufficient pressure that cannot effectively remove stains. In addition, force sensing data can also be used to realize the compliant control of the robot, improving the flexibility and adaptability of the robot.

[0026] In the process of movement, the second robot collects multiple modal data including joint sensing data, visual sensing data and force sensing data in real time through multiple sensors, packs the multiple modal data into a format suitable for transmission, and sends the multiple modal data from the second robot to the data processing platform through a predefined communication protocol. The data processing platform receives the multiple modal data sent from the second robot, and further fuses, analyzes and processes the multiple modal data. Through data fusion technology, the data of different modalities are integrated together to form a comprehensive understanding of the movement state of the robot and the environment.

[0027] In the embodiment of the application, the first robot generates and sends operation instructions to the second robot, the second robot receives and moves based on the operation instructions, and collects and sends multiple modal data in the process of movement. The multiple modal data includes joint sensing data, visual sensing data and force sensing data. The data processing platform receives the multiple modal data sent by the second robot, and fuses the multiple modal data to obtain multi-modal data. The comprehensive and accurate data acquisition and fusion are realized in the process of robot movement.

[0028] Please refer to Figure 2 Another embodiment of the data processing system based on the robot in the embodiment of the application includes: The first robot 201, the second robot 202 and the data processing platform 203, wherein the first robot controls the second robot based on a preset joint mapping relationship, the data processing platform is in communication connection with the first robot and the second robot, the first robot is used for generating and sending operation instructions, the second robot is used for receiving and moving based on the operation instructions, and collecting and sending multiple modal data in the process of movement. The multiple modal data includes joint sensing data, visual sensing data and force sensing data. The data processing platform is used for receiving the multiple modal data sent by the second robot, and fusing the multiple modal data to obtain multi-modal data.

[0029] The second robot includes a mechanical arm 2021 and a three-fingered hand 2022. The mechanical arm and the three-fingered hand of the second robot are both provided with multiple joint sensors. The three-fingered hand of the second robot is also provided with a visual sensor and at least one force sensor. The mechanical arm and the three-fingered hand are both provided with multiple degrees of freedom. Based on the multiple degrees of freedom, the multiple joints of the mechanical arm and the three-fingered hand can be controlled to move. Each joint can be controlled by a Dynamixel servo motor and used as a joint sensor, as shown in Figure 3 Figure 3 ​An embodiment of a mechanical arm of a second robot, the mechanical arm comprising a base 301, a 6-DOF mechanical arm 302, and a 10-DOF three-fingered hand 303, and the three-fingered hand is provided with joint sensors, vision sensors, and force sensors. The second robot uses a joint space mapping method to convert the operation instruction into motion parameters in its own joint space through a mapping formula, and executes corresponding motion, wherein the mapping formula is: J=

[0030] wherein J represents the converted joint space motion parameters, is the operation instruction, represents a set of preset joint mapping relationship parameters, represents a conversion function.

[0031] The second robot receives an operation instruction from the first robot, which specifies the target position, pose, and motion requirements of the second robot. The second robot uses an inverse kinematics algorithm to map the operation instruction from Cartesian space to joint space to obtain motion parameters for each joint, including the angles, velocities, and accelerations that each joint should reach. Based on these motion parameters, the joints are controlled by joint actuators to perform operations.

[0032] A plurality of joint sensors are used to obtain a plurality of joint information of the mechanical arm and the three-fingered hand to obtain joint sensing data; a vision sensor is used to obtain vision sensing data; and at least one force sensor is used to obtain at least one force information of the three-fingered hand to obtain force sensing data.

[0033] The force sensors provided on the three-fingered hand include a first force sensor and a second force sensor, the first force sensor is used to obtain the first force information of the palm of the three-fingered hand; and the second force sensor is used to obtain the second force information of the fingers of the three-fingered hand. Multiple force sensors can be provided on the palm and fingers of the three-fingered hand to improve the accuracy of pressure measurement through multiple force sensors.

[0034] The force sensor can be a piezoresistive force sensor provided with distributed electrodes. For each piezoresistive force sensor provided with distributed electrodes, the force information is obtained by: obtaining the resistance values of each electrode in the distributed electrodes; determining the pressure values corresponding to each electrode based on the resistance values of each electrode; and calculating the corresponding force information based on the pressure values corresponding to each electrode. In the distributed electrodes, each electrode can independently measure the resistance value at the position of the electrode. The corresponding pressure values of each resistance value are found by looking up a pre-established lookup table, and the force information is calculated by the pressure values of each electrode. The force information includes the direction, size and distribution of force. By comparing the pressure values on different electrodes, the direction of the force can be inferred. By calculating the total value or average value of the pressure on all electrodes, the size of the force can be obtained. By analyzing the distribution pattern of the pressure on the electrodes, the distribution characteristics of the force can be understood. The force information calculation method is:

[0035] where FSD is the final force sensing data, is the pressure value corresponding to the jth electrode in the piezoresistive force sensor distributed electrodes, is the weight coefficient related to the jth electrode, n is the total number of electrodes, and DF is the direction factor of the force calculated according to the pressure distribution of each electrode, wherein the calculation formula of DF is:

[0036] wherein, represents the position vector of the jth electrode in the sensor coordinate system, j represents the pressure value of the jth electrode. j

[0037] Obtain multiple joint information of the robot arm and the three-fingered hand to obtain joint sensing data, including: obtaining multiple joint motion poses and multiple joint motion speeds of the robot arm and the three-fingered hand to obtain joint sensing data, and the joint sensing data is:

[0038] wherein, represents the joint sensing data output by the joint sensor of the second robot, respectively represent the joint motion poses output by the joint sensors of the robot arm and the three-fingered hand of the second robot, respectively represent the joint motion speeds output by the joint sensors of the robot arm and the three-fingered hand of the second robot, respectively represent the left arm and the right arm of the second robot, represents the qth joint sensor in the t time. ​​

[0039] If the visual sensor is a red-green-blue-depth camera, acquiring the visual sensor data comprises: acquiring an RGB image and a Depth image, outputting visual sensor data based on the RGB image and the Depth image, and the visual sensor data is:

[0040] wherein, represents the visual sensor output of the visual sensor on the three-fingered hand, represents the RGB image, represents the Depth image, respectively represent the left arm and the right arm of the second robot, represents the cth visual sensor inside at time t.

[0041] Before fusing the multiple modal data, the multiple modal data is also preprocessed, specifically, for text data, spelling errors need to be corrected, naming conventions need to be unified, stop words need to be removed, or stem extraction needs to be performed, etc. For image data, image enhancement, denoising, cropping or normalization need to be performed.

[0042] Based on the multiple modal data, the fusion comprises: extracting kinematic features from the joint sensor data; extracting image features from the visual sensor data, wherein the formula for extracting image features using a convolutional neural network is:

[0043] wherein VF represents the extracted visual features, CNN represents a convolutional neural network, is the image data obtained by the visual sensor; extracting force sensation features from the force sensation data; fusing based on the kinematic features, the image features and the force sensation features, and the fusion formula is:

[0044] wherein MMD represents the final fused multi-modal data; i=1, 2, 3 in the formula, respectively represent the kinematic features, the image features and the force sensation features, i=1, 2, 3 in the formula are weighting coefficients of the corresponding features, and .

[0045] Kinematic features such as joint angles, velocities, accelerations, etc. can be extracted from joint sensor data, which can describe the motion state and trajectory of the object; image features including edges, corners, and textures, etc. can be extracted from image data obtained from vision sensors through image processing techniques, which can reflect the shape, position, and pose of the object; force / torque features such as force magnitude, direction, and application point, etc. can be extracted from force / torque sensor data, which can provide information about the force and pressure experienced by the object; based on these kinematic features, image features, and force / torque features extracted from different sensors, feature fusion techniques such as feature concatenation, feature weighting, or deep learning methods can be used to fuse these features into a unified feature representation, to more comprehensively describe and recognize the state and behavior of the object, thereby improving the accuracy and robustness of robot perception and decision-making.

[0046] Fusion based on kinematic features, image features, and force / torque features includes: normalizing kinematic features, image features, and force / torque features; inputting normalized kinematic features, image features, and force / torque features into corresponding processing models to obtain prediction results for each modality; integrating prediction results for each modality using a pre-set fusion strategy to obtain multi-modal data.

[0047] In order to eliminate the dimensional differences and numerical range differences between different features, these features need to be normalized so that they can be compared and fused on the same scale; normalized kinematic features, image features, and force / torque features are input into corresponding processing models, which can be machine learning models or deep learning models, capable of learning and predicting for each feature to obtain prediction results for each modality; using a pre-set fusion strategy, integrating prediction results for each modality to obtain multi-modal data that integrates kinematic, image, and force / torque information, the fusion strategy includes weighted average, voting mechanism, or feature fusion layer in deep learning, which can improve the accuracy and robustness of overall prediction or decision-making.

[0048] The data processing platform is also used to analyze the performance and behavior of the second robot based on the multi-modal data to obtain an analysis result, using the formula to evaluate the performance of the robot, where PI is the comprehensive performance evaluation index of the second robot, Accuracy represents the accuracy of the robot in executing tasks, MaxAccuracy is the maximum accuracy, Speed represents the design speed, MaxSpeed is the maximum design speed, represents the stability index, is the optimal stability state index value, , , is a weight coefficient, and + + =1; and send the analysis results to the first robot, which is also used to receive and adjust the joint space mapping parameters with the second robot based on the analysis results.

[0049] Feature information is extracted from multimodal data. Based on pre-defined domain knowledge and feature correlation analysis, the corresponding feature data is mapped to an analysis model. Based on the corresponding analysis model, the performance and behavior analysis results of the second robot are output. For example, based on domain knowledge, the performance evaluation of the second robot typically focuses on its motion capabilities, task execution efficiency, and accuracy. Based on this domain knowledge, a series of relevant features can be initially identified, such as speed, acceleration, positional accuracy, and task completion time. In feature correlation analysis, the correlation between the features identified through domain knowledge and the target performance indicators is further analyzed. For example, through correlation coefficients or feature importance assessment methods, it is found that the second robot's maximum speed and positional accuracy are highly correlated with task execution efficiency. Therefore, these two features can be prioritized for input into the performance evaluation model. Behavior prediction aims to predict the robot's behavior based on its historical behavior. To predict the second robot's future actions, based on domain knowledge, its motion trajectory, speed changes, and environmental perception data can be used as relevant features for predicting behavior. Time series analysis or pattern recognition techniques can reveal the inherent patterns and correlations in the second robot's behavior. For example, under specific environmental conditions, the second robot may take a series of specific obstacle avoidance actions. Therefore, the second robot's historical motion trajectory and environmental perception data can be used as input features into the behavior evaluation model to predict its behavior when encountering similar environments in the future. This approach yields analysis results of the second robot's performance and behavior, which can be presented in numerical, graphical, or report formats. The results include performance indicators such as the second robot's motion speed, accuracy, and stability; behavioral characteristics such as path tracking ability and obstacle avoidance response; and the diagnosis of any anomalies or potential problems. By comprehensively analyzing this data, the first robot can gain a comprehensive understanding of the second robot's current state. The first robot needs to evaluate the impact of current joint space mapping parameters on the second robot's performance and behavior, including how parameter settings affect the second robot's motion accuracy, stability, and response speed. By comparing the analysis results with the performance under the current parameter settings, the first robot can identify which parameters may need adjustment to optimize the second robot's performance.

[0050] In the embodiment of the present application, the first robot generates and sends operation instructions to the second robot, the second robot receives and moves based on the operation instructions, collects and sends various modal data in the movement process, the various modal data includes joint sensing data, visual sensing data and force sensing data, the data processing platform receives the various modal data sent by the second robot, and fuses based on the various modal data to obtain multi-modal data, analyzes the performance and behavior of the second robot based on the multi-modal data, obtains the analysis result, and sends the analysis result to the first robot, and the first robot is also used for receiving and adjusting the joint space mapping parameters with the second robot based on the analysis result, which realizes the collection of various modal data including joint sensing data, visual sensing data and force sensing data in the movement process of the robot, and further improves the comprehensiveness and accuracy of data collection and fusion, and optimizes the collaborative performance and adaptability between robots, and effectively improves the performance and intelligent level of the overall system.

[0051] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot-based data processing system, characterized in that, The data processing system includes: a first robot, a second robot, and a data processing platform, wherein the first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is communicatively connected to the second robot; The first robot is used to generate and send operation instructions; The second robot is used to receive and move based on the operation instructions, and to collect and send multiple modal data during the movement, including joint sensing data, visual sensing data and force sensing data. The data processing platform is used to receive multiple modal data sent by the second robot and fuse the multiple modal data to obtain multimodal data.

2. The robot-based data processing system according to claim 1, characterized in that, The second robot includes a robotic arm, a three-finger dexterous hand, multiple joint sensors, a vision sensor, and at least one force sensor. The multiple joint sensors are disposed on the robotic arm and the three-finger dexterous hand, and the vision sensor and the at least one force sensor are disposed on the three-finger dexterous hand. The multiple joint sensors are used to acquire information about multiple joints of the robotic arm and the three-finger dexterous hand to obtain joint sensing data. The visual sensor is used to acquire the visual sensing data; The at least one force sensor is used to acquire at least one force information of the three-finger dexterous hand to obtain force sensing data.

3. The robot-based data processing system according to claim 2, characterized in that, The acquisition of at least one force sensory information of the three-finger dexterous hand includes: Obtain the first force sensation information of the palm of the three-finger dexterous hand; Obtain the second force sensory information of the fingers of the three-finger dexterous hand.

4. The robot-based data processing system according to claim 3, characterized in that, If the force sensor is a piezoresistive force sensor including a distributed array of electrodes, then each force sensor acquires force information in the following ways: Obtain the resistance value of each electrode in the distributed electrode arrangement; The pressure value corresponding to each electrode is determined based on the resistance value of each electrode. The force information is calculated based on the pressure value corresponding to each electrode. The formula for calculating the force information is as follows: Among them, FSD is the final force sensing data obtained. It is the pressure value corresponding to the j-th electrode in the distributed electrode arrangement of the piezoresistive force sensor. is the weighting coefficient associated with the j-th electrode, n is the total number of electrodes, and DF is the force direction factor calculated based on the pressure distribution of each electrode. The formula for calculating DF is: in, Indicates the first j The position vector of each electrode in the sensor coordinate system Indicates the first j The pressure value of each electrode.

5. The robot-based data processing system according to claim 2, characterized in that, The acquisition of multiple joint information of the robotic arm and the three-finger dexterous hand to obtain joint sensing data includes: The robot arm and the three-finger dexterous hand acquire multiple joint poses and speeds to obtain joint sensing data, which is: in, This represents the joint sensing data output by the joint sensors of the second robot. These represent the joint motion poses output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the joint movement speeds output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the left and right arms of the second robot, respectively. This represents the q-th joint sensor at time t.

6. The robot-based data processing system according to claim 2, characterized in that, The visual sensor is a red-green-blue depth camera, and the acquisition of visual sensing data includes: Acquire RGB images and depth images, and output visual sensing data based on the RGB images and depth images. The visual sensing data is as follows: in, This represents the visual sensing data output by the visual sensor on the three-finger dexterity hand. Represents an RGB image. Represents the depth image. These represent the left and right arms of the second robot, respectively. This represents the c-th visual sensor at time t.

7. The robot-based data processing system according to claim 1, characterized in that, The fusion based on the multiple modal data includes: Extract kinematic features from the joint sensing data; Image features are extracted from the visual sensing data, wherein the formula for image feature extraction using a convolutional neural network is as follows: Where VF represents the extracted visual features, and CNN represents a convolutional neural network. It is image data acquired by a vision sensor; Extract force features from the force sensing data; The kinematic features, image features, and force sensory features are fused together using the following formula: MMD represents the final fused multimodal data; In this context, i = 1, 2, and 3 represent kinematic features, image features, and force-sensory features, respectively. In this context, i = 1, 2, and 3 represent the weighting coefficients for the corresponding features, and .

8. The robot-based data processing system according to claim 7, characterized in that, The fusion based on the kinematic features, the image features, and the force features includes: The kinematic features, the image features, and the force sensory features are normalized. The normalized kinematic features, image features, and force features are input into the corresponding processing models to obtain the prediction results for each modality. The prediction results of each modality are integrated using a preset fusion strategy to obtain multimodal data.

9. The robot-based data processing system according to any one of claims 1-8, characterized in that, The data processing platform is also used to analyze the performance and behavior of the second robot based on the multimodal data, obtain analysis results, and use formulas. The robot's performance is evaluated using PI, which is the overall performance evaluation index for the second robot. Accuracy represents the accuracy with which the robot performs the task, MaxAccuracy is the maximum accuracy, Speed ​​represents the design speed, and MaxSpeed ​​is the maximum design speed. Indicates stability index, It is the optimal steady-state index value. , , It is a weighting coefficient, and + + =1; and send the analysis results to the first robot.

10. The robot-based data processing system according to claim 2, characterized in that, The movement based on the operation command includes: Using a joint space mapping method, the operation command is converted into motion parameters in its own joint space through a mapping formula, and the corresponding motion is executed. The mapping formula is as follows: J= Where J represents the converted joint space motion parameters. These are operation instructions. This represents a preset set of joint mapping parameters. This represents the conversion function.

Images