Robot-based data processing system

Abstract

The application relates to the technical field of data processing, and discloses a robot-based data processing system for improving the comprehensiveness and accuracy of data acquisition and fusion. The robot-based data processing system comprises a first robot for generating and sending operation instructions; a second robot for receiving the operation instructions and moving based on the operation instructions, collecting and sending various modal data in the moving process, wherein the various modal data comprises joint sensing data, visual sensing data and force sensing data; and a data processing platform for receiving the various modal data sent by the second robot and fusing the various modal data to obtain multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a robot-based data processing system. Background Technology

[0002] In recent years, with the rapid development of robotics and artificial intelligence, people have begun to combine these technologies and apply them to fields or scenarios such as manufacturing, service, and home services, demonstrating enormous application potential.

[0003] Currently, most dual-arm robots typically combine joint motion information and visual information. Due to the limited information acquired, dual-arm robots have certain limitations, which restrict the comprehensiveness and accuracy of data acquisition and fusion. Summary of the Invention

[0004] This invention provides a robot-based data processing system to address the problem that the information acquired by robots in the prior art is relatively limited, resulting in constraints on the comprehensiveness and accuracy of data collection and fusion.

[0005] The first aspect of this invention provides a robot-based data processing system, comprising: a first robot, a second robot, and a data processing platform, wherein the first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is communicatively connected to the second robot; the first robot is used to generate and send operation commands; the second robot is used to receive and move based on the operation commands, and collect and send multiple modal data during the movement, the multiple modal data including joint sensing data, visual sensing data, and force sensing data; the data processing platform is used to receive the multiple modal data sent by the second robot and fuse the multiple modal data to obtain multimodal data.

[0006] In one feasible implementation, the second robot includes a robotic arm, a three-finger dexterous hand, multiple joint sensors, a vision sensor, and at least one force sensor. The multiple joint sensors are disposed on the robotic arm and the three-finger dexterous hand, and the vision sensor and the multiple force sensors are disposed on the three-finger dexterous hand. The multiple joint sensors are used to acquire multiple joint information of the robotic arm and the three-finger dexterous hand to obtain joint sensing data. The vision sensor is used to acquire the visual sensing data. The at least one force sensor is used to acquire at least one force information of the three-finger dexterous hand to obtain force sensing data.

[0007] In one feasible implementation, acquiring at least one force sensory information of the three-finger dexterous hand includes: acquiring first force sensory information of the palm of the three-finger dexterous hand; and acquiring second force sensory information of the fingers of the three-finger dexterous hand.

[0008] In one feasible implementation, if the force sensor is a piezoresistive force sensor including a distributed array of electrodes, then each force sensor acquires force information by: acquiring the resistance value of each electrode in the distributed array; determining the pressure value corresponding to each electrode based on the resistance value of each electrode; and calculating the corresponding force information based on the pressure value corresponding to each electrode. The force information calculation method is as follows:

[0009]

[0010] Among them, FSD is the final force sensing data obtained. It is the pressure value corresponding to the j-th electrode in the distributed electrode arrangement of the piezoresistive force sensor. is the weighting coefficient associated with the j-th electrode, n is the total number of electrodes, and DF is the force direction factor calculated based on the pressure distribution of each electrode. The formula for calculating DF is:

[0011]

[0012] in, Indicates the first j The position vector of each electrode in the sensor coordinate system Indicates the first j The pressure value of each electrode.

[0013] In one feasible implementation, acquiring multiple joint information of the robotic arm and the three-finger dexterous hand to obtain joint sensing data includes: acquiring multiple joint poses and multiple joint velocities of the robotic arm and the three-finger dexterous hand to obtain joint sensing data, wherein the joint sensing data is:

[0014] ,in, This represents the joint sensing data output by the joint sensors of the second robot. These represent the joint motion poses output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the joint movement speeds output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the left and right arms of the second robot, respectively. This represents the q-th joint sensor at time t.

[0015] In one feasible implementation, the visual sensor is a red-green-blue depth camera, and the acquisition of visual sensing data includes: acquiring an RGB image and a depth image, and outputting visual sensing data based on the RGB image and the depth image, wherein the visual sensing data is:

[0016] ,in, This represents the visual sensing data output by the visual sensor on the three-finger dexterity hand. Represents an RGB image. Represents the depth image. These represent the left and right arms of the second robot, respectively. This represents the c-th visual sensor at time t.

[0017] In one feasible implementation, the fusion based on the multiple modal data includes: extracting kinematic features from the joint sensing data; and extracting image features from the visual sensing data, wherein the formula for image feature extraction using a convolutional neural network is: Where VF represents the extracted visual features, and CNN represents a convolutional neural network. It is image data acquired by a visual sensor; force features are extracted from the force sensing data; and the kinematic features, image features, and force features are fused together using the following formula: MMD represents the final fused multimodal data; In this context, i = 1, 2, and 3 represent kinematic features, image features, and force-sensory features, respectively. In this context, i = 1, 2, and 3 represent the weighting coefficients for the corresponding features, and .

[0018] In one feasible implementation, the fusion based on the kinematic features, the image features, and the force features includes: normalizing the kinematic features, the image features, and the force features; inputting the normalized kinematic features, the image features, and the force features into corresponding processing models to obtain prediction results for each modality; and integrating the prediction results of each modality using a preset fusion strategy to obtain multimodal data.

[0019] In one feasible implementation, the data processing platform is further used to analyze the performance and behavior of the second robot based on the multimodal data, obtain analysis results, and use formulas. The robot's performance is evaluated using PI, which is the overall performance evaluation index for the second robot. Accuracy represents the accuracy with which the robot performs the task, MaxAccuracy is the maximum accuracy, Speed ​​represents the design speed, and MaxSpeed ​​is the maximum design speed. Indicates stability index, It is the optimal steady-state index value. , , It is a weighting coefficient, and + + =1; and send the analysis results to the first robot.

[0020] In one feasible implementation, the movement based on the operation command includes: using a joint space mapping method, converting the operation command into motion parameters in its own joint space through a mapping formula, and executing the corresponding movement. The mapping formula is:

[0021] J=

[0022] Where J represents the converted joint space motion parameters. These are operation instructions. This represents a preset set of joint mapping parameters. This represents the conversion function.

[0023] The technical solution provided by this invention includes a data processing system comprising: a first robot, a second robot, and a data processing platform. The first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is communicatively connected to the second robot. The first robot generates and sends operation commands. The second robot receives and moves based on the operation commands, collecting and sending multiple modal data during the movement. These multiple modal data include joint sensing data, visual sensing data, and force sensing data. The data processing platform receives the multiple modal data sent by the second robot and fuses the multiple modal data to obtain multimodal data. In this embodiment of the invention, by introducing a design where the first robot controls the second robot based on a preset joint mapping relationship, and combining this with a data processing platform, the collection of multiple modal data, including joint sensing data, visual sensing data, and force sensing data, during robot movement is achieved, thereby improving the comprehensiveness and accuracy of data collection and fusion. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of one embodiment of the robot-based data processing system of the present invention;

[0025] Figure 2 This is a schematic diagram of another embodiment of the robot-based data processing system of the present invention;

[0026] Figure 3 This is a schematic diagram of one embodiment of an arm of the second robot in this invention. Detailed Implementation

[0027] This invention provides a robot-based data processing system that improves the comprehensiveness and accuracy of data acquisition and fusion by collecting multiple modal data, including joint sensing data, visual sensing data, and force sensing data.

[0028] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Please see Figure 1 One embodiment of the robot-based data processing system in this invention includes:

[0030] The system comprises a first robot 101, a second robot 102, and a data processing platform 103. The first robot controls the second robot based on a preset joint mapping relationship. The data processing platform is communicatively connected to the first and second robots. The first robot generates and sends operation commands, while the second robot receives and moves based on the operation commands. During the movement, the robot collects and sends multiple modal data, including joint sensing data, visual sensing data, and force sensing data. The data processing platform receives the multiple modal data sent by the second robot and fuses the multiple modal data to obtain multimodal data.

[0031] The operator inputs control commands through the first robot's user interface or by directly operating its robotic arm. The first robot can be equipped with an intuitive user interface, such as a touchscreen, buttons, and joysticks, allowing the operator to input commands. Besides the user interface, the operator can also wear specialized devices to operate the main dual-arm robot's robotic arms. These devices can be data gloves or exoskeletons, enabling more intuitive and natural control. The first robot incorporates high-precision sensors that capture its joint angles, positions, and speeds in real time. These motion states are converted into a series of digital signals containing the actions the operator wants the second robot to perform. These digital signals are further formatted into operational commands that the second robot can understand and execute. This process can include coordinate transformations and scaling to ensure the second robot accurately mimics the first robot's movements.

[0032] The first robot and the second robot interact through a predefined communication protocol. This protocol specifies the format of instructions, transmission methods, and error handling mechanisms. Operation instructions can be sent from the first robot to the second robot wirelessly (such as Wi-Fi or Bluetooth) or wiredly (such as Ethernet or USB). Wireless transmission provides greater flexibility and convenience, while wired transmission may provide more stable and faster data transmission.

[0033] The second robot integrates joint sensors, vision sensors, and force sensors. Joint sensors acquire joint sensing data, which mainly reflects the robot's kinematic parameters such as joint angles, angular velocities, and angular accelerations. These kinematic parameters allow us to understand the robot's dynamic characteristics, such as pose and velocity. Vision sensors acquire visual sensing data, including multimedia information such as images and videos. This information provides an intuitive view of the robot's working environment, aiding in environmental perception and obstacle detection. Through visual sensing data, the robot can identify target objects, locate its own position, and navigate and avoid obstacles in complex environments. Furthermore, visual sensing data can be used to evaluate the robot's work performance, such as checking whether a plate has been successfully wiped clean. Force sensors acquire force sensing data, which reflects the forces and torques experienced by the robot during its interaction with the environment. For example, during plate wiping, force sensing data helps the robot control the pressure of the sponge on the plate, preventing excessive pressure from damaging the plate or insufficient pressure from effectively removing stains. Additionally, force sensing data can be used to achieve compliant robot control, improving the robot's flexibility and adaptability.

[0034] During movement, the second robot collects multimodal data in real time through various sensors, including joint sensing data, visual sensing data, and force sensing data. After packaging the multiple modalities into a format suitable for transmission, the data is sent from the second robot to the data processing platform through a predefined communication protocol. The data processing platform receives the multimodal data sent from the second robot and performs further fusion, analysis, and processing. Through data fusion technology, the data from different modalities are integrated to form a comprehensive understanding of the robot's motion state and environment.

[0035] In this embodiment of the invention, a first robot generates and sends operation commands to a second robot. The second robot receives and moves based on the operation commands, collecting and sending multiple modal data during the movement. The multiple modal data includes joint sensing data, visual sensing data, and force sensing data. The data processing platform receives the multiple modal data sent by the second robot and fuses the multiple modal data to obtain multimodal data. This realizes the collection of multiple modal data, including joint sensing data, visual sensing data, and force sensing data, during robot movement, thereby improving the comprehensiveness and accuracy of data collection and fusion.

[0036] Please see Figure 2 Another embodiment of the robot-based data processing system in this invention includes:

[0037] The system comprises a first robot 201, a second robot 202, and a data processing platform 203. The first robot controls the second robot based on a preset joint mapping relationship. The data processing platform is communicatively connected to the first and second robots. The first robot generates and sends operation commands, while the second robot receives and moves based on the operation commands. During the movement, the robot collects and sends multiple modal data, including joint sensing data, visual sensing data, and force sensing data. The data processing platform receives the multiple modal data sent by the second robot and fuses the multiple modal data to obtain multimodal data.

[0038] The second robot includes a robotic arm 2021 and a three-finger dexterous hand 2022. Both the robotic arm and the three-finger dexterous hand of the second robot are equipped with multiple joint sensors. The three-finger dexterous hand of the second robot is also equipped with a vision sensor and at least one force sensor.

[0039] Both the robotic arm and the three-finger dexterous hand have multiple degrees of freedom. Based on these multiple degrees of freedom, multiple joints of the robotic arm and the three-finger dexterous hand can be controlled to move. Each joint can be controlled using a Dynamixel servo motor, which also serves as a joint sensor. Figure 3 As shown, Figure 3A schematic diagram of an embodiment of a robotic arm of the second robot shows the robotic arm comprising a base 301, a 6-DOF robotic arm 302, and a 10-DOF three-finger dexterous hand 303, which is equipped with joint sensors, a vision sensor, and a force sensor. The second robot utilizes a joint space mapping method, converting operation commands into motion parameters in its own joint space using a mapping formula, and then executes the corresponding movements. The mapping formula is as follows:

[0040] J=

[0041] Where J represents the converted joint space motion parameters. These are operation instructions. This represents a preset set of joint mapping parameters. This represents the conversion function.

[0042] The second robot receives operation instructions from the first robot. These instructions specify the target position, posture, and motion requirements of the second robot. The second robot uses an inverse kinematics algorithm to map the operation instructions from Cartesian space to joint space to obtain the motion parameters of each joint. These motion parameters include the angles, velocities, and accelerations that each joint should achieve. Based on these motion parameters, the second robot controls each joint to operate through joint actuators.

[0043] Multiple joint sensors are used to acquire information from multiple joints of the robotic arm and the three-finger dexterous hand to obtain joint sensing data; a vision sensor is used to acquire visual sensing data; and at least one force sensor is used to acquire at least one force information from the three-finger dexterous hand to obtain force sensing data.

[0044] The force sensor installed on the three-finger dexterity hand includes a first force sensor and a second force sensor. The first force sensor is used to acquire first force information from the palm of the three-finger dexterity hand; the second force sensor is used to acquire second force information from the fingers of the three-finger dexterity hand. Multiple force sensors can be installed on the palm and fingers of the three-finger dexterity hand to improve the accuracy of pressure measurement.

[0045] The force sensor can be a piezoresistive force sensor with a distributed array of electrodes. For each piezoresistive force sensor with a distributed array of electrodes, the method for acquiring force information is as follows: acquiring the resistance value of each electrode in the distributed array; determining the pressure value corresponding to each electrode based on its resistance value; and calculating the corresponding force information based on the pressure value corresponding to each electrode. In the distributed array of electrodes, each electrode can independently measure the resistance value at its location. A pre-established lookup table is used to find the pressure value corresponding to each resistance value, and force information is calculated from the pressure values ​​of each electrode. Force information includes the direction, magnitude, and distribution of the force. By comparing the pressure magnitudes on different electrodes, the direction of the force can be inferred; by calculating the sum or average of the pressures on all electrodes, the magnitude of the force can be obtained; and by analyzing the pressure distribution pattern on the electrodes, the force distribution characteristics can be understood. The force information calculation method is as follows:

[0046]

[0047] Among them, FSD is the final force sensing data obtained. It is the pressure value corresponding to the j-th electrode in the distributed electrode arrangement of the piezoresistive force sensor. is the weighting coefficient associated with the j-th electrode, n is the total number of electrodes, and DF is the force direction factor calculated based on the pressure distribution of each electrode. The formula for calculating DF is:

[0048]

[0049] in, Indicates the first j The position vector of each electrode in the sensor coordinate system Indicates the first j The pressure value of each electrode.

[0050] Acquire information from multiple joints of the robotic arm and the three-finger dexterous hand to obtain joint sensing data, including: acquiring the pose and velocity of multiple joints in the robotic arm and the three-finger dexterous hand to obtain joint sensing data. The joint sensing data is as follows:

[0051]

[0052] in, This represents the joint sensing data output by the joint sensors of the second robot. These represent the joint motion poses output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the joint movement speeds output by the joint sensors of the second robot's robotic arm and three-finger dexterous hand, respectively. These represent the left and right arms of the second robot, respectively. This represents the q-th joint sensor at time t.

[0053] If the visual sensor is a red-green-blue depth camera, then acquiring visual sensing data includes: acquiring RGB images and depth images, and outputting visual sensing data based on the RGB images and depth images. The visual sensing data is as follows:

[0054]

[0055] in, This represents the visual sensing data output by the vision sensor in a three-finger dexterity hand. Represents an RGB image. Represents the depth image. These represent the left and right arms of the second robot, respectively. This represents the c-th visual sensor at time t.

[0056] Before fusing multiple modal data, preprocessing of the data is also required. Specifically, for text data, processing such as correcting spelling errors, standardizing naming conventions, removing stop words, or stemming is required. For image data, processing such as image enhancement, denoising, cropping, or normalization is required.

[0057] This involves fusing data from multiple modalities, including: extracting kinematic features from joint sensing data; and extracting image features from visual sensing data. The formula for image feature extraction using a convolutional neural network is as follows:

[0058]

[0059] Where VF represents the extracted visual features, and CNN represents a convolutional neural network. It is image data acquired by a vision sensor; force features are extracted from force sensing data; and fusion is performed based on kinematic features, image features, and force features, using the following fusion formula:

[0060]

[0061] MMD represents the final fused multimodal data; In this context, i = 1, 2, and 3 represent kinematic features, image features, and force-sensory features, respectively. In this context, i = 1, 2, and 3 represent the weighting coefficients for the corresponding features, and .

[0062] Kinematic features such as joint angles, velocities, and accelerations are extracted from joint sensor data, describing the object's motion state and trajectory. Image data is acquired from vision sensors, and image features, including edges, corners, and textures, are extracted using image processing techniques, reflecting the object's shape, position, and posture. Force sensors provide information about the forces and pressures acting on the object, from which force features such as magnitude, direction, and point of application can be extracted. Based on these kinematic, image, and force features extracted from different sensors, feature fusion techniques, such as feature stitching, feature weighting, or deep learning methods, are used to fuse these features into a unified feature representation, providing a more comprehensive description and identification of the object's state and behavior, thereby improving the accuracy and robustness of robot perception and decision-making.

[0063] The fusion of kinematic features, image features, and force features includes: normalizing the kinematic features, image features, and force features; inputting the normalized kinematic features, image features, and force features into the corresponding processing models to obtain the prediction results for each modality; and integrating the prediction results of each modality using a preset fusion strategy to obtain multimodal data.

[0064] To eliminate dimensional and numerical range differences among different features, these features need to be normalized so that they can be compared and fused on the same scale. The normalized kinematic features, image features, and force-sensory features are then input into their respective processing models, which can be machine learning models or deep learning models. These models can learn and predict based on their respective features to obtain prediction results for each modality. Using a pre-defined fusion strategy, the prediction results of each modality are integrated to obtain multimodal data that combines kinematic, image, and force-sensory information. Fusion strategies include weighted averaging, voting mechanisms, or feature fusion layers in deep learning. This approach can improve the accuracy and robustness of the overall prediction or decision-making.

[0065] The data processing platform is also used to analyze the performance and behavior of the second robot based on multimodal data, obtain analysis results, and use formulas. The robot's performance is evaluated using PI, which is the overall performance evaluation index for the second robot. Accuracy represents the accuracy with which the robot performs the task, MaxAccuracy is the maximum accuracy, Speed ​​represents the design speed, and MaxSpeed ​​is the maximum design speed. Indicates stability index, It is the optimal steady-state index value. , , It is a weighting coefficient, and + + =1; and send the analysis results to the first robot, which is also used to receive and adjust the joint space mapping parameters with the second robot based on the analysis results.

[0066] Feature information is extracted from multimodal data. Based on pre-defined domain knowledge and feature correlation analysis, the corresponding feature data is mapped to an analysis model. Based on the corresponding analysis model, the performance and behavior analysis results of the second robot are output. For example, based on domain knowledge, the performance evaluation of the second robot typically focuses on its motion capabilities, task execution efficiency, and accuracy. Based on this domain knowledge, a series of relevant features can be initially identified, such as speed, acceleration, positional accuracy, and task completion time. In feature correlation analysis, the correlation between the features identified through domain knowledge and the target performance indicators is further analyzed. For example, through correlation coefficients or feature importance assessment methods, it is found that the second robot's maximum speed and positional accuracy are highly correlated with task execution efficiency. Therefore, these two features can be prioritized for input into the performance evaluation model. Behavior prediction aims to predict the robot's behavior based on its historical behavior. To predict the second robot's future actions, based on domain knowledge, its motion trajectory, speed changes, and environmental perception data can be used as relevant features for predicting behavior. Time series analysis or pattern recognition techniques can reveal the inherent patterns and correlations in the second robot's behavior. For example, under specific environmental conditions, the second robot may take a series of specific obstacle avoidance actions. Therefore, the second robot's historical motion trajectory and environmental perception data can be used as input features into the behavior evaluation model to predict its behavior when encountering similar environments in the future. This approach yields analysis results of the second robot's performance and behavior, which can be presented in numerical, graphical, or report formats. The results include performance indicators such as the second robot's motion speed, accuracy, and stability; behavioral characteristics such as path tracking ability and obstacle avoidance response; and the diagnosis of any anomalies or potential problems. By comprehensively analyzing this data, the first robot can gain a comprehensive understanding of the second robot's current state. The first robot needs to evaluate the impact of current joint space mapping parameters on the second robot's performance and behavior, including how parameter settings affect the second robot's motion accuracy, stability, and response speed. By comparing the analysis results with the performance under the current parameter settings, the first robot can identify which parameters may need adjustment to optimize the second robot's performance.

[0067] In this embodiment of the invention, a first robot generates and sends operation commands to a second robot. The second robot receives and moves based on the operation commands, collecting and sending multiple modal data during the movement. The multiple modal data includes joint sensing data, visual sensing data, and force sensing data. The data processing platform receives the multiple modal data sent by the second robot and fuses them to obtain multimodal data. Based on the multimodal data, the performance and behavior of the second robot are analyzed to obtain analysis results, which are then sent to the first robot. The first robot is also used to receive and adjust the joint space mapping parameters with the second robot based on the analysis results. This enables the collection of multiple modal data, including joint sensing data, visual sensing data, and force sensing data, during robot movement, thereby improving the comprehensiveness and accuracy of data collection and fusion, optimizing the collaborative performance and adaptability between robots, and effectively enhancing the overall system performance and intelligence level.

[0068] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-based data processing system, characterized in that The data processing system comprises a first robot, a second robot and a data processing platform, wherein the first robot controls the second robot based on a preset joint mapping relationship, and the data processing platform is in communication connection with the second robot; The first robot is configured to generate and send operation instructions; The second robot is configured to receive the operation instructions and move based on the operation instructions, collect and send multiple modal data during the movement, and the multiple modal data comprises joint sensing data, visual sensing data and force sensing data; The data processing platform is configured to receive the multiple modal data sent by the second robot, and fuse the multiple modal data to obtain multi-modal data; The second robot comprises a mechanical arm, a three-fingered hand, multiple joint sensors, a visual sensor and at least one force sensor, the multiple joint sensors are arranged on the mechanical arm and the three-fingered hand, and the visual sensor and the at least one force sensor are arranged on the three-fingered hand; the multiple joint sensors are configured to obtain multiple joint information of the mechanical arm and the three-fingered hand to obtain joint sensing data; the visual sensor is configured to obtain the visual sensing data; and the at least one force sensor is configured to obtain at least one force information of the three-fingered hand to obtain force sensing data; The method for obtaining the at least one force information of the three-fingered hand comprises: obtaining first force information of a palm of the three-fingered hand; and obtaining second force information of a finger of the three-fingered hand; If the force sensor is a piezoresistive force sensor comprising distributedly arranged electrodes, the manner for each force sensor to obtain force information comprises: obtaining resistance values of each electrode in the distributedly arranged electrodes; determining pressure values corresponding to each electrode based on the resistance values of each electrode; and calculating corresponding force information based on the pressure values corresponding to each electrode, and a force information calculation formula is: Wherein, FSD is the force sense data finally obtained, is the pressure value corresponding to the jth electrode pair in the distributed arrangement of the piezoresistive force sense sensor, is the weight coefficient related to the jth electrode, n is the total number of electrodes, and DF is the force direction factor calculated according to the pressure distribution of each electrode, wherein the calculation formula of DF is: wherein, represents the position vector of the j th electrode in the sensor coordinate system, represents the pressure value of the j th electrode.

2. The machine-based data processing system of claim 1, wherein, The method for obtaining the multiple joint information of the mechanical arm and the three-fingered hand to obtain joint sensing data comprises: obtaining multiple joint action poses and multiple joint action speeds on the mechanical arm and the three-fingered hand to obtain joint sensing data, and the joint sensing data is: wherein, represents joint sensor data outputted by a joint sensor of the second robot, respectively represent joint action poses outputted by joint sensors of a robot arm and a three-fingered hand of the second robot, respectively represent joint action velocities outputted by joint sensors of a robot arm and a three-fingered hand of the second robot, respectively represent left and right arms of the second robot, represents the qth joint sensor inside at time t.

3. The machine-based data processing system of claim 1, wherein, The visual sensor is a red-green-blue-depth camera, and the method for obtaining visual sensing data comprises: obtaining an RGB image and a Depth depth image, and outputting visual sensing data based on the RGB image and the Depth depth image, and the visual sensing data is: wherein, represents visual sensing data outputted by a visual sensor on the three-fingered hand, represents an RGB image, represents a Depth image, respectively represent a left arm and a right arm of the second robot, represents a cth visual sensor inside at time t.

4. The machine-based data processing system of claim 1, wherein, The method for fusing based on the multiple modal data comprises: extracting kinematic features from the joint sensing data; extracting image features from the visual sensing data, and a formula for image feature extraction using a convolutional neural network is: wherein VF represents the extracted visual features, CNN represents a convolutional neural network, is image data acquired by a vision sensor; extracting force features from the force sensing data; fusing based on the kinematic features, the image features and the force features, and a fusion formula is: wherein MMD represents the multi-modal data obtained by final fusion; i=1, 2, 3, respectively represent kinematics features, image features, and force sensation features, i=1, 2, 3, are weighting coefficients of the corresponding features, and .

5. The machine-based data processing system of claim 4, wherein, The method for fusing based on the kinematic features, the image features and the force features comprises: performing normalization processing on the kinematic features, the image features and the force features; and The normalized kinematic feature, the image feature and the force feature are respectively input into corresponding processing models to obtain prediction results of each modality; The prediction results of each modality are integrated by using a preset fusion strategy to obtain multi-modal data.

6. The machine-based data processing system according to any one of claims 1-5, wherein, The data processing platform is further configured to analyze performance and behavior of the second robot based on the multi-modal data, obtain an analysis result, and adopt a formula evaluate the performance of the robot, wherein PI is a comprehensive performance evaluation index of the second robot, Accuracy represents the accuracy of the robot in performing a task, MaxAccuracy is the maximum accuracy, Speed represents the design speed, MaxSpeed is the maximum design speed, represents a stability index, is the optimal stability index value, 、 、 is a weight coefficient, and + + = 1; and send the analysis result to the first robot.

7. The machine-based data processing system of claim 1, wherein, The movement based on the operation instruction comprises: The operation instruction is converted into a movement parameter in a joint space of the robot by using a joint space mapping method and a mapping formula, and corresponding movement is performed, the mapping formula being: J= wherein J represents the converted joint space motion parameter, is an operation instruction, represents a preset joint mapping relationship parameter set, represents a conversion function.

Images