Heterogeneous robot collaborative processing method, device and equipment based on multi-modal perception

Abstract

The application provides a multi-modal perception-based heterogeneous robot cooperative processing method, device and equipment, relating to the technical field of mechanical arm control. In the above method, the visual spatial pose of the material to be carried is obtained, and the visual spatial pose is mapped through coordinate conversion to determine the cooperative contact pose. According to the proportional relationship between the first bearing capacity and the second bearing capacity, the first target load and the second target load are determined. The heavy-load mechanical arm and the light-load mechanical arm are moved to the cooperative contact pose, and the heavy-load mechanical arm and the light-load mechanical arm jointly contact and carry the material to be carried, and the attitude deviation between the current attitude and the target attitude is calculated. The first actual load and the second actual load are obtained, and the load deviation of the light-load mechanical arm is calculated using the second actual load and the second target load. According to the load deviation and the attitude deviation, the pose correction amount and the contact force correction amount of the light-load mechanical arm are generated, and the light-load mechanical arm is compensated in real time according to the pose correction amount and the contact force correction amount while carrying, so that stable cooperative carrying is realized.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, specifically to a method, apparatus, and device for heterogeneous robot collaborative processing based on multimodal perception. Background Technology

[0002] In logistics warehousing and production line transfer scenarios, some materials to be transported are too large or have uneven weight distribution, exceeding the carrying capacity of a single robotic arm. Therefore, two robotic arms are needed simultaneously to collaboratively transport the same material. In actual production lines, heterogeneous robotic arm combinations with significantly different carrying capacities are often deployed, such as a heavy-duty industrial robotic arm and a lightweight collaborative robotic arm working together to perform transport tasks.

[0003] Existing multi-robotic arm collaborative handling solutions mainly employ geometric trajectory synchronization, which involves planning matching end-effector trajectories for each robotic arm in space, enabling them to complete the handling actions in the same time sequence. This solution implicitly assumes that when each robotic arm executes the same trajectory command, the actual load will naturally tend to be balanced.

[0004] However, due to differences in their mechanical design, heterogeneous robotic arms exhibit significant differences in structural stiffness. Heavy-duty robotic arms typically employ large-section links and high-stiffness reducers, resulting in high overall structural stiffness and minimal elastic deformation under load. Lightweight robotic arms, prioritizing weight reduction, utilize thin-walled links and compliant joint modules, leading to significantly lower structural stiffness and much greater elastic deformation under the same external force. Even with mathematically precise matching of the end-effector poses of two robotic arms under synchronized geometric trajectory control, the actual end-effector poses will deviate due to differences in the direction and magnitude of elastic deformation under load. When both robotic arms are simultaneously rigidly constrained to the same material, this differential deformation cannot resolve itself but transforms into uncontrollable internal forces transmitted through the material, posing a risk of load distribution deviating from expectations and material attitude drift. Summary of the Invention

[0005] This application provides a method, apparatus, and device for collaborative processing of heterogeneous robots based on multimodal perception, which can achieve controllable load distribution and avoid material attitude drift when collaboratively handling the same material.

[0006] The technical solution of this application embodiment is as follows: In a first aspect, embodiments of this application provide a heterogeneous robot collaborative processing method based on multimodal perception, applied to a heterogeneous robot system including a heavy-duty robotic arm and a light-duty robotic arm, for handling the same material to be transported, wherein the first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm, and the method includes: The visual spatial pose of the material to be transported is obtained, and the visual spatial pose is mapped to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm, respectively, to determine the cooperative contact pose. Based on the ratio between the first load-bearing capacity and the second load-bearing capacity, determine the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm; The heavy-duty robotic arm and the light-duty robotic arm are moved to the cooperative contact pose to jointly contact and transport the material to be transported, and the attitude deviation between the current attitude and the target attitude of the material to be transported is calculated. The heavy-duty robotic arm moves according to the pre-planned main motion trajectory, and the light-duty robotic arm follows the movement of the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material. Obtain the first actual load of the heavy-duty robotic arm along the load-bearing direction and the second actual load of the light-duty robotic arm along the load-bearing direction, and calculate the load deviation of the light-duty robotic arm using the second actual load and the second target load; Based on the load deviation and the attitude deviation, the pose correction amount and contact force correction amount of the light-load robotic arm are generated, and the light-load robotic arm is controlled to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-load robotic arm in handling.

[0007] In the above technical solution, firstly, the coordinate mapping of visual pose provides a unified spatial target reference for the two heterogeneous robotic arms; secondly, the target load allocation based on the load-bearing capacity ratio establishes a clear deviation criterion for force feedback, enabling the system to distinguish between normal load and parasitic internal forces caused by stiffness differences; thirdly, the master-slave architecture allows the heavy-duty robotic arm to dominate the motion reference due to its high structural stiffness, while simultaneously granting the light-duty robotic arm with lower structural stiffness a correction degree of freedom independent of the master motion; fourthly, the deviation between the actual load obtained by force perception and the target load directly drives the light-duty robotic arm to perform pose or contact force correction, so that the differential deformation caused by the stiffness difference between the two arms is actively absorbed by the light-duty robotic arm in each control cycle, rather than passively accumulating into uncontrollable internal forces. By establishing a target load allocation based on the load-bearing capacity ratio and giving the light-duty robotic arm independent force compensation capability, the differential elastic deformation caused by the structural stiffness difference between the heterogeneous robotic arms can be perceived and actively compensated in real time, keeping the parasitic internal forces between the two arms within a controllable range, ensuring the rationality of load distribution and the stability of material posture during handling.

[0008] Secondly, embodiments of this application provide a heterogeneous robot collaborative processing device based on multimodal perception, applied to a heterogeneous robot system including a heavy-duty robotic arm and a light-duty robotic arm, for handling the same material to be transported. The first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm. The device includes: The data acquisition module is used to acquire the visual spatial pose of the material to be transported, and map the visual spatial pose to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm respectively, so as to determine the cooperative contact pose. The load division module is used to determine the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm based on the ratio between the first load capacity and the second load capacity. The attitude deviation calculation module is used to move the heavy-duty robotic arm and the light-duty robotic arm to the cooperative contact pose, so that they can jointly contact and transport the material to be transported, and calculate the attitude deviation between the current attitude and the target attitude of the material to be transported. The heavy-duty robotic arm moves according to the pre-planned main motion trajectory, and the light-duty robotic arm follows the movement of the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material. The load deviation calculation module is used to obtain the first actual load of the heavy-duty robotic arm along the bearing direction and the second actual load of the light-duty robotic arm along the bearing direction, and to calculate the load deviation of the light-duty robotic arm using the second actual load and the second target load. The assistance compensation module is used to generate the pose correction amount and contact force correction amount of the light-load robotic arm based on the load deviation and the posture deviation, and control the light-load robotic arm to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-load robotic arm in handling.

[0009] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a user interface, a communication bus, and a network interface. The processor, the memory, the user interface, and the network interface are respectively connected to the communication bus. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as provided in any one of the first aspects.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, perform the method described in any one of the methods provided in the first aspect above.

[0011] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By employing coordinate mapping based on the initial visual pose, a unified spatial target benchmark is provided for the two heterogeneous robotic arms. Target load allocation based on the load-bearing capacity ratio establishes a clear deviation criterion for force feedback, enabling the system to distinguish between normal load and parasitic internal forces caused by stiffness differences. The master-slave architecture allows the heavy-duty robotic arm to dominate the motion benchmark due to its high structural stiffness, while granting the lighter-duty robotic arm with lower structural stiffness independent correction degrees of freedom from the master motion. The deviation between the actual load obtained from force perception and the target load directly drives the lighter-duty robotic arm to correct its pose or contact force, ensuring that the differential deformation caused by the stiffness difference between the two arms is actively absorbed by the lighter-duty robotic arm in each control cycle, rather than passively accumulating into uncontrollable internal forces. By establishing a target load allocation based on the load-bearing capacity ratio and granting the lighter-duty robotic arm independent force compensation capabilities, the differential elastic deformation caused by the structural stiffness difference between the heterogeneous robotic arms can be perceived and actively compensated in real time, maintaining the parasitic internal forces between the two arms within a controllable range and ensuring the rationality of load distribution and the stability of material posture during handling. Therefore, it effectively solves the problem of uncontrollable load distribution and material attitude drift caused by the inability of geometric trajectory synchronization to compensate for the differential elastic deformation of each arm when heterogeneous robotic arms with different load-bearing capacity and structural stiffness collaboratively transport the same material.

[0012] 2. In abnormal conditions, the lightly loaded robotic arm first adjusts the support force or end-effector posture; if it does not recover, then control the two robotic arms to stop in coordination, hold the material or place it safely, thus forming an executable abnormal closed loop. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a heterogeneous robot collaborative processing method based on multimodal perception, provided in one embodiment of this application. Figure 2 This is a schematic diagram of a system module for a heterogeneous robot collaborative processing method based on multimodal perception, provided in one embodiment of this application. Figure 3 This is a coordinate transformation diagram of a heterogeneous robot collaborative processing method based on multimodal perception provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a heterogeneous robot collaborative processing device based on multimodal perception provided in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0015] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0016] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0017] In related technologies, relying solely on visual information can obtain the spatial position and orientation of materials, but it is difficult to reflect the actual load-bearing state after the two end effectors come into contact with the materials; relying solely on force information makes it difficult to achieve spatial alignment between the materials, the camera, and the two robotic arms. Therefore, it is necessary to technically correlate the visual pose with the force information of the two robotic arms within the same coordinates and control cycle, and to form a master-slave cooperative control relationship based on differences in load-bearing capacity. In addition, the system in related technologies is caught in a technical contradiction: if absolute trajectory synchronization is guaranteed, it cannot absorb the parasitic internal forces caused by differences in absolute stiffness, which can easily damage the materials; if rigid synchronization is deactivated and compliant control is adopted, it can easily lead to material attitude drift. It is difficult to balance internal force unloading and attitude control.

[0018] Based on this, embodiments of this application provide a heterogeneous robot collaborative processing method, device, electronic device, and readable storage medium based on multimodal perception. This heterogeneous robot collaborative processing method first provides a unified spatial target reference for two heterogeneous robotic arms through visual pose coordinate mapping; secondly, it establishes a clear deviation criterion for force feedback based on the target load allocation according to the load-bearing capacity ratio, enabling the system to distinguish between normal load and parasitic internal forces caused by stiffness differences; thirdly, the master-slave architecture allows the heavy-duty robotic arm to dominate the motion reference due to its high structural stiffness, while simultaneously granting the light-duty robotic arm with lower structural stiffness a correction degree of freedom independent of the master motion; fourthly, the deviation between the actual load obtained from force perception and the target load directly drives the light-duty robotic arm to perform pose or contact force correction, ensuring that the differential deformation caused by the stiffness difference between the two arms is actively absorbed by the light-duty robotic arm in each control cycle, rather than passively accumulating into uncontrollable internal forces. By establishing a target load allocation based on the load-bearing capacity ratio and giving the light-load robotic arm independent force compensation capability, the differential elastic deformation caused by the structural stiffness difference of the heterogeneous robotic arms can be sensed and actively compensated in real time, keeping the parasitic internal force between the two arms within a controllable range, and ensuring the rationality of load distribution and the stability of material posture during the handling process.

[0019] It should be noted that this heterogeneous robot collaborative processing method based on multimodal perception can be used in scenarios such as logistics warehousing, production line transfer, loading and unloading, and stacking, and can achieve stable material handling.

[0020] The technical solutions provided in the embodiments of this application will be further described below with reference to the accompanying drawings.

[0021] Reference Figure 1 , Figure 1 This is a flowchart illustrating the heterogeneous robot collaborative processing method based on multimodal perception provided in this application embodiment. The heterogeneous robot collaborative processing method based on multimodal perception is applied to a heterogeneous robot system including a heavy-duty robotic arm and a light-duty robotic arm to handle the same material. The first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm.

[0022] like Figure 2As shown, the heterogeneous robot system also includes a vision perception module, a unified workstation coordinate system, a collaborative controller, a heavy-duty robotic arm, a lightweight robotic arm, a first end effector, a second end effector, and a first force perception module and a second force perception module. The vision perception module is positioned to observe the material to be transported and the work area. Its output includes at least the position and orientation of the material to be transported relative to the camera coordinate system. This vision perception module can be positioned over the top area of ​​the work area or over the robotic arm to sense its real-time orientation. The force perception module acquires force information representing the interaction between the two robotic arms and the material to be transported. It can use a six-dimensional force / torque sensor installed between the end effector and the robotic arm flange, or it can estimate the end force based on joint torque, drive current, and robotic arm dynamics. The collaborative controller receives the outputs from the vision perception module and the force perception module and generates control commands for the two robotic arms. Here, "heavy-duty" and "lightweight" indicate a relative difference in load-bearing capacity and do not limit specific rated load, brand, number of degrees of freedom, or mechanical structure. The end effectors of the two robotic arms can be in the form of clamping, adsorption, lifting, or a combination thereof, contacting different parts of the same material to be transported, and simultaneously completing lifting, transfer and placement.

[0023] The heterogeneous robot cooperative processing method based on multimodal perception is executed by a processor in an electronic device or a readable storage medium. The heterogeneous robot cooperative processing method based on multimodal perception includes steps S100, S200, S300, S400 and S500.

[0024] Step S100: Obtain the visual spatial pose of the material to be transported, and map the visual spatial pose to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm respectively, so as to determine the cooperative contact pose.

[0025] In one embodiment, visual spatial pose refers to the six-degree-of-freedom position and orientation information of the material to be transported in space, including the translational coordinates of the material's centroid or feature points in three-dimensional space and the rotational orientation angle of the material relative to a reference coordinate system. This visual spatial pose is obtained by a visual perception module installed above or to the side of the work area through image acquisition and pose estimation algorithms. The visual perception module can employ equipment such as industrial structured light cameras, binocular stereo cameras, or depth cameras, combined with deep learning-based object pose estimation algorithms or point cloud registration-based pose recognition algorithms, to output a pose description of the material's coordinate system relative to the camera coordinate system.

[0026] Cooperative contact pose refers to the spatial pose that the end effectors of the heavy-duty and light-duty robotic arms should achieve when contacting the material to be transported. This pose simultaneously determines the contact position and posture of the end effectors relative to the material. Since the two robotic arms each have their own independent base coordinate system, and there is a spatial offset and rotation relationship between the camera coordinate system and the base coordinate systems of the two robotic arms, coordinate transformation is needed to uniformly map the material pose obtained by the vision perception module in the camera coordinate system to the respective base coordinate systems of the two robotic arms. This allows the two robotic arms to establish a consistent spatial target reference around the same material. By establishing a unified coordinate mapping relationship, the problem of inconsistent expression and use of the pose of the same material to be transported when the camera, heavy-duty robotic arm, and light-duty robotic arm are in different coordinate systems is solved, providing a unified spatial reference for subsequent cooperative motion control.

[0027] like Figure 3 As shown, the visual spatial pose is mapped to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm, respectively, to determine the cooperative contact pose, including but not limited to the following steps: Step S110: Obtain the first pose transformation matrix of the coordinate system of the material to be transported relative to the camera coordinate system, output by the visual perception module.

[0028] Specifically, the material coordinate system refers to a local coordinate system fixed on the material to be transported. Its origin can be set at the geometric center of the material or a predefined feature point, and the coordinate axis directions are determined according to the shape characteristics of the material. The camera coordinate system is a coordinate system constructed with the optical center of the vision perception module as the origin, the horizontal and vertical directions of the imaging plane as the X and Y axes, and the optical axis as the Z axis. The first pose transformation matrix is ​​a homogeneous transformation matrix of four rows and four columns. The three rows and three columns of the submatrix in the upper left corner represent the rotation relationship of the material coordinate system relative to the camera coordinate system, and the top three elements of the rightmost column represent the translation components of the origin of the material coordinate system in the camera coordinate system.

[0029] The first pose transformation matrix is ​​calculated and output in real time by the pose estimation algorithm of the visual perception module. Specifically, the visual perception module acquires color and depth images of the material to be transported, and uses a pre-trained deep learning pose estimation network or a registration algorithm based on point cloud features to match the acquired visual features of the material with a pre-stored 3D model of the material, thereby determining the six-degree-of-freedom pose of the material coordinate system in the camera coordinate system, and outputs it in the form of a homogeneous transformation matrix, providing the starting transformation node for subsequent coordinate chain operations.

[0030] Step S120: Read the pre-calibrated coordinate system transformation relationship. The coordinate system transformation relationship includes the calibration matrix of the camera coordinate system relative to the global world coordinate system, the second pose transformation matrix of the global world coordinate system relative to the heavy-duty robot arm base coordinate system, and the third pose transformation matrix of the global world coordinate system relative to the light-duty robot arm base coordinate system.

[0031] Specifically, the global coordinate system refers to a unified reference coordinate system established at the workstation. Its origin and coordinate axis directions are fixed after being determined during the system installation and commissioning phase, serving as a bridge connecting the camera coordinate system and the base coordinate systems of the two robotic arms. The base coordinate system of the heavy-duty robotic arm is established with the center of the heavy-duty robotic arm mounting base as its origin, and its coordinate axis directions are determined by the factory settings of the heavy-duty robotic arm. Similarly, the base coordinate system of the light-duty robotic arm is established with the center of the light-duty robotic arm mounting base as its origin.

[0032] The calibration matrices are obtained through a hand-eye calibration method. The specific calibration process is as follows: a calibration board is placed within the camera's field of view and the reach of the robotic arm. The robotic arm's end effector, carrying the calibration board, is controlled to remain in multiple different postures. The pose of the calibration board observed by the camera and the end effector pose calculated using the forward kinematics of the robotic arm are recorded. The rigid body transformation matrix from the camera coordinate system to the global world coordinate system is solved using the Tsai-Lenz hand-eye calibration algorithm or a multi-point calibration method based on least squares. The second and third pose transformation matrices are obtained through a station calibration method. Specifically, a laser tracker or a high-precision total station is used to measure the coordinates of the calibration points on the bases of the two robotic arms in the global world coordinate system. Combined with the base coordinate system definition provided by the robotic arm manufacturer, the transformation matrix from the global world coordinate system to each robotic arm's base coordinate system is calculated. The above calibration matrices are calibrated once after system installation and stored in the collaborative controller as configuration parameters. By establishing a complete coordinate transformation chain in advance, it is possible to reach the base coordinate systems of the two robotic arms from the camera coordinate system via the global world coordinate system, thus providing a mathematical basis for subsequent pose unification mapping.

[0033] Step S130: Perform homogeneous coordinate transformation based on the first pose transformation matrix and calibration matrix to obtain the absolute pose matrix of the material to be transported in the global world coordinate system.

[0034] Specifically, the absolute pose matrix refers to the representation of the coordinate system of the material to be transported in the global world coordinate system. It describes the absolute spatial position and orientation of the material in the unified workstation coordinate system in the form of a four-row, four-column homogeneous transformation matrix. The homogeneous coordinate transformation operation is to multiply the calibration matrix and the first pose transformation matrix according to the matrix left multiplication rule, that is, to multiply the first pose transformation matrix of the material coordinate system relative to the camera coordinate system by the transformation matrix from the global world coordinate system to the camera coordinate system.

[0035] The collaborative controller reads the calibration matrix from memory and performs a 4x4 matrix multiplication with the first pose transformation matrix output by the vision perception module in the current control cycle to obtain the absolute pose matrix of the material coordinate system in the global world coordinate system. This operation is performed once in each vision sampling cycle to ensure that the absolute pose of the material can track its real-time spatial state. Transforming the material pose from the local observation results of the camera to an absolute spatial description at the workstation level eliminates the influence of camera installation position and orientation on subsequent calculations, allowing the pose of the same material to be distributed subsequently using a unified spatial reference.

[0036] Step S140: Multiply the absolute pose matrix with the second pose transformation matrix and the third pose transformation matrix to obtain the heavy-load target pose of the material to be transported in the heavy-load robot arm base coordinate system and the light-load target pose in the light-load robot arm base coordinate system.

[0037] Specifically, the heavy-duty target pose refers to the expression of the coordinate system of the material to be transported in the base coordinate system of the heavy-duty robot arm, describing the spatial position and orientation relationship of the material relative to the mounting base of the heavy-duty robot arm. The light-duty target pose similarly describes the spatial position and orientation relationship of the material relative to the mounting base of the light-duty robot arm. The multiplication operation involves left-multiplying the transformation matrices from the global world coordinate system to each robot arm base coordinate system by the absolute pose matrix.

[0038] The collaborative controller left-multiplies the second and third pose transformation matrices by the absolute pose matrix obtained in step S130. These two matrix multiplications are performed independently, outputting the heavy-load and light-load target poses in the form of 4x4 homogeneous transformation matrices, respectively. Since the second and third pose transformation matrices are pre-calibrated fixed values, this operation only involves the multiplication of constant matrices and real-time variable matrices, resulting in low computational complexity and completion within millisecond-level control cycles. This allows the pose information of the same material to be used in the respective control spaces of the two heterogeneous robotic arms, providing target references in their respective base coordinate systems for each arm to independently plan its end-effector motion, thus avoiding the limitation that the two robotic arms need to share the same coordinate system to collaborate.

[0039] Step S150: Extract the preset heavy-load contact offset matrix and light-load contact offset matrix of the material to be transported, synthesize the heavy-load target pose with the heavy-load contact offset matrix to determine the cooperative contact pose of the heavy-load robotic arm, and synthesize the light-load target pose with the light-load contact offset matrix to determine the cooperative contact pose of the light-load robotic arm.

[0040] Specifically, the heavy-duty contact offset matrix refers to the fixed spatial offset and attitude deflection of the tool center point of the heavy-duty robotic arm's end effector relative to the origin of the material coordinate system. It describes the orientation and angle at which the heavy-duty robotic arm should make contact with the material. The light-duty contact offset matrix has a similar meaning, describing the contact offset relationship of the light-duty robotic arm's end effector relative to the material coordinate system. These two offset matrices are predetermined during the system configuration phase based on the material's dimensions, center of gravity, contact surface characteristics, and the end effector's contact method, and are stored in the collaborative controller as configuration parameters. For example, for a long strip of material, the heavy-duty contact offset matrix can be defined as the position where the origin of the material coordinate system is offset a certain distance along the positive length direction, and the light-duty contact offset matrix is ​​defined as the position where it is offset a corresponding distance along the negative length direction, so that the two robotic arms make contact from both ends of the material. The synthesis operation multiplies the target material pose in each robotic arm's base coordinate system by the corresponding contact offset matrix to obtain the final contact pose that the end effector should reach in the corresponding robotic arm's base coordinate system.

[0041] The collaborative controller right-multiplies the heavy-load target pose matrix by the heavy-load contact bias matrix, outputting the collaborative contact pose of the heavy-load manipulator; it right-multiplies the light-load target pose matrix by the light-load contact bias matrix, outputting the collaborative contact pose of the light-load manipulator. By transforming material-level pose information into specific contact targets at the end effector level, subsequent motion planning can directly use the collaborative contact pose as the target input for solving the inverse kinematics of each manipulator. Through the parameterized design of the contact bias matrix, the same collaborative control logic can adapt to materials of different shapes and sizes; only the bias matrix parameters need to be changed to adapt to new material types.

[0042] Step S200: Determine the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm based on the ratio of the first load capacity to the second load capacity.

[0043] In one embodiment, the first load-bearing capacity refers to the maximum end-effector load that the heavy-duty robotic arm can continuously and stably withstand within its working posture range. This is typically determined by the robotic arm's factory-rated load parameters, or by derating it based on the actual installation posture and workspace range. The second load-bearing capacity refers to the maximum continuous load-bearing value corresponding to the light-duty robotic arm. The first target load and the second target load refer to the force values ​​along the load-bearing direction that the heavy-duty robotic arm and the light-duty robotic arm are expected to bear during collaborative handling, respectively.

[0044] Determining the proportional relationship: Using the sum of the first and second load-bearing capacities as the total load base, calculate the proportion of each robotic arm's load-bearing capacity to the total load base as the target load-sharing coefficient. Specifically, the target load-sharing coefficient for a heavy-duty robotic arm equals the first load-bearing capacity divided by the sum of the first and second load-bearing capacities; the target load-sharing coefficient for a light-duty robotic arm equals the second load-bearing capacity divided by the sum of the first and second load-bearing capacities. The sum of the two coefficients equals one.

[0045] The sum of the actual loads of the two robotic arms along the bearing direction, output in real time by the force sensing module, is used as the estimated total load currently being carried jointly. This estimated total load is then multiplied by the corresponding target load sharing coefficient to obtain the first and second target loads. This method does not require pre-input of the material weight but utilizes the real-time detected total load for allocation, enabling adaptive tracking of the actual material weight and changes in effective load due to posture changes during handling. In special scenarios where the material weight is known, the target load can also be determined directly by multiplying the material weight by each target load sharing coefficient. By allocating the load according to the load-bearing capacity ratio, each robotic arm bears a load proportion matching its own structural capacity, avoiding the problem of overloading lightly loaded robotic arms or wasting the capacity of heavily loaded robotic arms caused by simple average allocation. At the same time, the allocation method based on the load-bearing capacity ratio ensures that the working points of the two robotic arms are close within their respective load margins, which is beneficial to the overall safety and consistency of the system. The target load allocation establishes a clear deviation criterion for subsequent force feedback, enabling the differentiation between normal load and parasitic internal forces caused by stiffness differences.

[0046] In step S300, the heavy-duty robotic arm and the light-duty robotic arm are moved to a cooperative contact pose to jointly contact and transport the material to be transported, and the attitude deviation between the current attitude and the target attitude of the material to be transported is calculated. The heavy-duty robotic arm moves according to the pre-planned main motion trajectory, and the light-duty robotic arm follows the movement of the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material.

[0047] In one embodiment, the main motion trajectory refers to the end-effector spatial motion path planned by the cooperative controller for the heavy-duty robotic arm from the initial gripping position to the target placement position, based on the initial pose of the material and the target placement pose. This trajectory defines the position and attitude of the heavy-duty robotic arm's end effector at each moment during the handling process. The main motion trajectory can be generated using Cartesian linear interpolation, circular interpolation, or spline curve interpolation, and combined with trapezoidal or S-shaped velocity curves for time parameterization, enabling the heavy-duty robotic arm to move at a constant speed or according to a predetermined acceleration / deceleration law along the trajectory. The target attitude refers to the desired attitude direction that the material to be handled should maintain during the handling process. It is usually set as the attitude of the material at the initial gripping moment or the attitude determined according to the target placement requirements, and is pre-specified by the handling task. The attitude deviation refers to the deviation between the material's current actual attitude and the target attitude, characterized by Euler angle difference or rotation vector difference.

[0048] Commands to move to the cooperative contact pose are simultaneously sent to both robotic arms. Each arm independently plans and executes its approach trajectory from its current pose to the contact pose, using the cooperative contact pose in its respective base coordinate system as the inverse kinematics solution objective. Once both arms' end effectors reach their respective cooperative contact poses, the end effectors initiate the contact action. A force sensing module detects whether the load on both arms along the load-bearing direction exceeds a preset contact confirmation threshold (e.g., 5% of the estimated material weight). If both exceed this threshold, it is confirmed that both arms have formed load-bearing contact with the same material. After contact confirmation, the heavy-duty arm begins moving along the main motion trajectory. The basic desired pose of the light-duty arm is calculated and updated in real time based on the current pose of the heavy-duty arm and the relative contact relationship between the two arms, allowing the light-duty arm to follow the heavy-duty arm. Within each control cycle, the material's posture is acquired through a visual sensing module and transformed to obtain the current posture. The difference between the current posture and the target posture is calculated to obtain the current posture deviation.

[0049] Heavy-duty robotic arms, due to their high structural stiffness, exhibit minimal elastic deformation under load, providing a stable motion reference. Light-duty robotic arms, with lower structural stiffness, would directly transmit elastic deformation as material posture deviation if used as the primary motion reference. Therefore, they are defined as secondary motion roles, performing independent corrective actions while following the primary motion. This architecture allows the heavy-duty robotic arm to dominate the motion reference due to its high structural stiffness, while simultaneously granting the less stiff light-duty robotic arm corrective degrees of freedom independent of the primary motion, providing control space for subsequent force compensation.

[0050] Step S400: Obtain the first actual load of the heavy-duty robotic arm along the load-bearing direction and the second actual load of the light-duty robotic arm along the load-bearing direction, and calculate the load deviation of the light-duty robotic arm using the second actual load and the second target load.

[0051] In one embodiment, the bearing direction refers to the direction of the material's gravity during handling, which is vertically downward in most horizontal handling scenarios. In the end-effector coordinate system, this corresponds to the negative Z-axis direction of the end flange or the Z-axis direction of the force sensor in the force sensing module. The first actual load and the second actual load refer to the actual force values ​​detected along the bearing direction by the force sensing modules of the heavy-duty and light-duty manipulators at the current control moment. These can be force values ​​directly measured by a six-dimensional force / torque sensor installed between the manipulator flange and the end effector, or estimated end-effector force values ​​calculated by the collaborative controller based on the joint torque values ​​fed back by the servo drives of each joint, combined with the manipulator's dynamics model and the current joint angle through Jacobian matrix transpose mapping. The load deviation characterizes the deviation between the actual load currently borne by the light-duty manipulator and the expected load.

[0052] Within each control cycle, the force values ​​along the load-bearing direction are read from the force sensing modules of the two robotic arms. Based on the sharing coefficient determined in step S200 and the sum of the current loads of the two robotic arms, the second target load is recalculated. The second target load is then subtracted from the second actual load to obtain the load deviation. A positive load deviation indicates that the actual load of the lightly loaded robotic arm is lower than the target value, requiring an increase in load. A negative load deviation indicates that the actual load of the lightly loaded robotic arm exceeds the target value, posing an overload risk. Obtaining the true load state after contact through force sensing compensates for the limitation of visual perception, which only provides spatial pose and cannot reflect the actual force distribution. The load deviation directly quantifies the load offset caused by the differential elastic deformation resulting from the structural stiffness differences in the heterogeneous robotic arms, providing a clear driving input for subsequent correction of the lightly loaded robotic arm. Since the load deviation is updated within each control cycle, the degree of load imbalance can be sensed and quantified in real time.

[0053] Step S500: Based on the load deviation and attitude deviation, generate the pose correction amount and contact force correction amount of the light-load robotic arm, and control the light-load robotic arm to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-load robotic arm in handling.

[0054] In one embodiment, the pose correction refers to the small pose increment superimposed on the desired pose of the light-load robotic arm base in each control cycle, including correction components along three translational directions and three rotational directions, used to adjust the spatial position and orientation of the end effector of the light-load robotic arm. The contact force correction refers to the force increment superimposed on the desired contact force of the light-load robotic arm base in each control cycle, used to adjust the magnitude of the contact force applied by the end effector to the material. Real-time compensation refers to the continuous execution of compensation actions by the light-load robotic arm with the pose correction and contact force correction as additional correction inputs on top of the base pose following the main motion of the heavy-load robotic arm during collaborative handling, so that the differential deformation caused by the stiffness difference between the two arms is actively absorbed by the light-load robotic arm in each control cycle, rather than passively accumulating into uncontrollable internal forces.

[0055] While assisting the heavy-duty robotic arm in handling materials, the light-load robotic arm performs real-time compensation based on posture and contact force corrections. This compensation achieves a technical linkage between visual perception and force perception within the same control cycle: the visual perception module provides material posture deviation as the driving force for posture correction, while the force perception module provides load deviation as the driving force for contact force correction. Both types of corrections act simultaneously on the light-load robotic arm, forming a closed-loop control system driven by multimodal perception. By establishing a target load allocation based on load-bearing capacity ratio and endowing the light-load robotic arm with independent force compensation capabilities, the differential elastic deformation caused by structural stiffness differences between the heterogeneous robotic arms can be sensed and actively compensated in real time. This maintains the parasitic internal forces between the two arms within a controllable range, ensuring the rationality of load distribution and the stability of material posture during handling.

[0056] In one embodiment, the pose correction amount and contact force correction amount of the lightly loaded robotic arm are generated based on the load deviation and attitude deviation, including but not limited to the following steps: Step S510: Obtain the instantaneous main motion velocity vector of the end effector of the heavy-duty robotic arm at the current moment, and construct a dynamic compliant coordinate system that is updated in real time with the motion state of the heavy-duty robotic arm, with the direction of the instantaneous main motion velocity vector as the first axis direction and the direction of the bearing plane perpendicular to the instantaneous main motion velocity vector as the second axis direction.

[0057] Specifically, the instantaneous main motion velocity vector refers to the linear velocity vector of the center point of the end-effector of the heavy-duty robotic arm in Cartesian space at the current control moment. Its direction indicates the current forward motion direction of the heavy-duty robotic arm, and its magnitude reflects the current motion rate. This vector is calculated by the cooperative controller based on the first derivative of the time parameterized expression of the main motion trajectory of the heavy-duty robotic arm at the current moment, or it is obtained by mapping the joint velocities of the heavy-duty robotic arm to the end-effector linear velocity through the Jacobian matrix. The bearing plane refers to the plane containing the bearing direction and perpendicular to the instantaneous main motion velocity vector, and the second axis direction is the component direction along the bearing direction in this plane. The dynamic compliance coordinate system is an orthogonal coordinate system composed of the first axis direction, the second axis direction, and the third axis direction determined by the right-hand rule, with its origin set at the center point of the end-effector of the light-duty robotic arm.

[0058] Within each control cycle, the collaborative controller calculates and normalizes the instantaneous primary motion velocity vector at the end effector of the heavy-duty robotic arm, using this unit vector as the first axis direction. The load-bearing direction vector (usually the vertical downward direction in the global world coordinate system expressed in the current coordinate system) is cross-productted with the first axis direction to obtain the third axis direction. The third axis direction is then cross-productted with the first axis direction to obtain the second axis direction, thus forming an orthogonal dynamic compliant coordinate system. This coordinate system rotates and updates in real time as the heavy-duty robotic arm's motion direction changes. When the heavy-duty robotic arm is stationary or at extremely low speed, the coordinate system direction at the previous effective moment remains unchanged or is replaced by a preset default direction. During the handling process, the heavy-duty robotic arm may move along a curved trajectory, with its motion direction varying at different times. If the compliance parameters of the light-duty robotic arm are configured in a fixed coordinate system, the light-duty robotic arm will exhibit compliant behavior in some motion directions and stiff behavior in others, failing to maintain optimal following behavior consistently. The dynamic conforming coordinate system ensures that the conforming direction of the light-load robotic arm is always aligned with the actual movement direction of the heavy-load robotic arm. No matter how the heavy-load robotic arm turns, the following strategy of the light-load robotic arm always maintains directional consistency.

[0059] Step S520: Project the load deviation and attitude deviation onto the dynamic compliance coordinate system to obtain the parallel deviation component along the first axis and the orthogonal deviation component along the second axis.

[0060] Specifically, projection refers to transforming the load deviation vector and attitude deviation vector, expressed in other coordinate systems, to the components of the dynamic compliant coordinate system through coordinate rotation transformation. The parallel deviation component is the composite component of the load deviation and attitude deviation along the first axis of the dynamic compliant coordinate system, reflecting the degree of deviation along the direction of motion. The orthogonal deviation component is the composite component of the load deviation and attitude deviation along the second axis of the dynamic compliant coordinate system, reflecting the degree of deviation perpendicular to the direction of motion and located within the bearing plane.

[0061] The collaborative controller converts load deviations into force vectors and attitude deviations into equivalent displacement deviations, combining them into a composite deviation vector using a preset weighting ratio. Then, using a rotation matrix of the dynamic compliant coordinate system, this composite deviation vector is transformed from the light-load manipulator base coordinate system to the dynamic compliant coordinate system. Its component along the first axis is extracted as the parallel deviation component, and its component along the second axis is extracted as the orthogonal deviation component. This projection operation is a coordinate rotation of a three-dimensional vector. This decomposition of multi-source deviation information into meaningful directions aligned with the current motion state allows subsequent differentiated compliant control to adopt different response strategies for deviation characteristics in different directions, avoiding the application of uniform control force in all directions.

[0062] Step S530: In the dynamic compliant coordinate system, configure spatially anisotropic admittance control parameters for the lightly loaded robotic arm, wherein a first compliant stiffness is configured along the first axis direction, and a second sustaining stiffness is configured along the second axis direction, and the first compliant stiffness is less than the second sustaining stiffness.

[0063] Specifically, admittance control parameters refer to the set of parameters in the admittance control model that map external force / torque deviations to end-effector pose corrections, including stiffness parameters, damping parameters, and inertia parameters. This step focuses on configuring stiffness parameters. The first compliance stiffness is a virtual spring stiffness value along the first axis direction (the direction of forward motion). A smaller value means that the lightly loaded robotic arm is compliant in the direction of motion, easily driven by deviations to generate pose corrections, allowing the lightly loaded robotic arm to smoothly follow the movement of the heavy-load robotic arm without generating counterforce. The second maintenance stiffness is a virtual spring stiffness value along the second axis direction (perpendicular to the direction of motion in the bearing plane). A larger value means that the lightly loaded robotic arm is stiff in this direction, effectively resisting the disturbance of material posture by deviations and firmly maintaining the bearing posture of the material. The first compliance stiffness is smaller than the second maintenance stiffness. Spatial anisotropy refers to configuring control parameters with different values ​​in different directions, so that the same controller exhibits different degrees of compliance in different directions. The specific values ​​are determined experimentally based on the material mass, handling speed, and actual stiffness of the robotic arm.

[0064] Furthermore, the first compliance stiffness and the second maintenance stiffness are not fixed values, but rather adaptively change with the current deviation state. Specifically, the first compliance stiffness monotonically decreases as the absolute value of the load deviation increases, meaning that the greater the deviation of the load borne by the light-load robotic arm from the target, the more compliant it becomes in the direction of movement, making the light-load robotic arm easier to move and thus reducing its own additional burden. The second maintenance stiffness monotonically increases as the absolute value of the attitude deviation increases, meaning that the greater the deviation of the material's attitude from the target, the more rigid it becomes in the direction of load maintenance, more forcefully correcting the material's attitude deviation. The above parameter adjustments are achieved through an adaptive adjustment function, which can be implemented using a piecewise linear function or an exponentially decaying function.

[0065] For example, firstly, the basic compliance stiffness value and the basic sustaining stiffness value are obtained. The basic compliance stiffness value is less than the basic sustaining stiffness value. Both are predetermined through offline calibration based on the secondary load-bearing capacity of the light-load robotic arm and the posture accuracy requirements of the handling task. The calculation process for the first compliance stiffness is: Kc = Kc0·e -α|eL| Where eL is the load deviation of the light-load robotic arm, α is a positive compliance attenuation coefficient, Kc0 is the basic compliance stiffness value, and Kc is the first compliance stiffness. When Kc decreases, the light-load robotic arm becomes more compliant in the first axis direction, actively releasing the internal force between the two arms; when the load deviation approaches zero, Kc returns to the basic value. The calculation process of the second maintenance stiffness is: Km=Km0·(1+β|ep|), where ep is the attitude deviation, α is a positive maintenance gain coefficient, Km0 is the basic maintenance stiffness value, and Km is the second maintenance stiffness. When the absolute value of the attitude deviation increases, the scaling factor monotonically increases, Km increases, and the light-load robotic arm applies a stronger corrective torque in the second axis direction; when the attitude deviation approaches zero, Km returns to the basic value to avoid overcorrection causing oscillation.

[0066] Damping parameters matching the stiffness are configured for each of the two axes to ensure that the admittance response in each axis has an appropriate damping ratio. These parameters together constitute the admittance control parameter set for the light-load robotic arm in the dynamic compliant coordinate system. Through the configuration of anisotropic parameters, the balance between the compliant following capability and the attitude maintenance capability of the light-load robotic arm is dynamically adjusted during the handling process. When the deviation in the direction of motion increases, the compliance is automatically increased to prevent the light-load robotic arm from overloading; when the attitude deviation increases, the stiffness is automatically increased to prevent material instability, thus realizing the adaptive optimization of the control strategy to the real-time working conditions.

[0067] Step S540: Vector synthesis is performed based on the impedance mapping of the first compliance stiffness and the parallel deviation component, and the impedance mapping of the second maintenance stiffness and the orthogonal deviation component, respectively. The synthesis result is then inversely transformed to the base coordinate system of the light-load robotic arm to obtain the pose correction amount.

[0068] Specifically, impedance mapping refers to the process of mapping external deviations into end-effector pose corrections through a virtual spring-damping-inertia system in the admittance control model. Specifically, along the first axis, the parallel deviation component is divided by the first compliance stiffness to obtain the pose correction component along that direction; along the second axis, the orthogonal deviation component is divided by the second maintenance stiffness to obtain the pose correction component along that direction. When considering damping and inertia parameters, numerical integration is also required based on the second-order dynamic equations of admittance control to ensure smooth dynamic transitions in the correction values, avoiding vibrations caused by abrupt changes. Vector synthesis combines the correction components along each axis into a three-dimensional correction vector in the dynamic compliance coordinate system. Inverse transformation uses the transpose of the rotation matrix from the dynamic compliance coordinate system to the light-load manipulator base coordinate system to transform the correction vector back from the dynamic compliance coordinate system to the light-load manipulator base coordinate system.

[0069] The pose correction components in the first and second axes are calculated separately in the dynamic compliant coordinate system. These components are then combined into a correction vector in the dynamic compliant coordinate system. This vector is then multiplied by the rotation matrix from the dynamic compliant coordinate system to the light-load manipulator's base coordinate system to obtain the pose correction amount in the light-load manipulator's base coordinate system. This pose correction amount is superimposed on the light-load manipulator's basic following pose to form the final pose command, which is then sent to the light-load manipulator's motion controller for execution. The deviation information is transformed into a specific pose correction action executable by the light-load manipulator through an admittance control model. Inverse coordinate transformation ensures that the correction amount can be directly used within the light-load manipulator's own control space. Because the first compliant stiffness is small while the second maintaining stiffness is large, the same deviation produces a larger correction amount in the motion direction, allowing the light-load manipulator to follow smoothly, and a smaller correction amount in the load-bearing direction, stabilizing the material's posture. This achieves intelligent correction with directional selectivity.

[0070] In one embodiment, the pose correction amount and contact force correction amount of the lightly loaded robotic arm are generated based on the load deviation and attitude deviation, and the process includes, but is not limited to, the following steps: Step S550: In the dynamic compliant coordinate system, extract the normal force deviation component along the clamping normal of the end effector of the light-load robotic arm and the tangential force deviation component along the tangential direction. When the tangential force deviation component exceeds the preset critical friction force, couple the excess tangential force deviation component to obtain the coupled tangential force deviation component.

[0071] Specifically, the clamping normal direction refers to the normal direction of the contact surface between the end effector of a light-load robotic arm and the material, i.e., the direction in which the end effector applies clamping force. For parallel grippers, the clamping normal direction is the direction of the closing motion of the two gripping fingers; for suction-type end effectors, the clamping normal direction is the normal direction of the suction cup plane. The normal force deviation component refers to the projected component of the load deviation along the clamping normal direction, reflecting the deviation between the clamping force and the target clamping force. The tangential force deviation component refers to the projected component of the load deviation in a plane perpendicular to the clamping normal direction, reflecting the shear force deviation generated by the material on the end effector that may lead to slippage. The critical friction force is the product of the current normal contact force and the static friction coefficient of the end effector-material contact interface, representing the maximum tangential force that can maintain static friction and prevent slippage under the current clamping force. The static friction coefficient is determined experimentally based on the combination of the end effector contact surface material and the material surface material.

[0072] The six-dimensional force data at the end of the lightly loaded robotic arm, output from the force sensing module, is projected onto the clamping normal plane and the tangential plane perpendicular to the clamping normal plane to obtain the values ​​of the normal force deviation component and the tangential force deviation component, respectively. The current normal contact force (directly read by the force sensor) is multiplied by a preset static friction coefficient to obtain the critical friction force value. When the absolute value of the tangential force deviation component exceeds the critical friction force, the excess is calculated and used as a coupled tangential force deviation component in subsequent contact force correction calculations. When the tangential force deviation component does not exceed the critical friction force, the coupled tangential force deviation component is zero. Traditional contact force control only focuses on the magnitude of the normal clamping force, neglecting the influence of the tangential force on material stability. By introducing a critical friction force criterion, when the tangential force approaches the sliding limit, it is coupled to the normal force correction, achieving a preventative response to slippage trends. This allows the system to proactively increase the clamping force to raise the upper limit of friction when a slippage trend appears, rather than remedying it after slippage occurs, thus fundamentally suppressing slippage.

[0073] Step S560: Establish the constraint coupling relationship between the normal force deviation component and the contact force correction amount. Within the constraint coupling relationship, calculate the contact force correction amount using the normal force deviation component and the coupled tangential force deviation component.

[0074] Specifically, the constraint coupling relationship refers to the upper and lower limit constraints applied when calculating the contact force correction. The upper limit constraint is the maximum end effector normal force corresponding to the rated output torque of the light-load manipulator, ensuring that the contact force correction does not cause overload of the light-load manipulator joints. This upper limit value is obtained by mapping the rated torque of each joint of the light-load manipulator in the current posture to the end effector normal direction through the Jacobian matrix, and the minimum value of the mapping results for each joint is taken as the upper limit. The lower limit constraint is the minimum normal force required to maintain static friction between the end effector and the material, that is, the value of the current tangential force component divided by the static friction coefficient. This lower limit ensures that effective frictional constraints are always maintained between the end effector and the material under any correction action, without disengagement.

[0075] The expected value of the contact force correction is calculated based on the combined value of the normal force deviation component and the coupled tangential force deviation component. The synthesis method is as follows: the normal force deviation component is multiplied by a preset normal force gain factor as the base correction, and the coupled tangential force deviation component is multiplied by a preset tangential coupling gain factor as an additional correction, resulting in an unlimited contact force correction. This unlimited value is then compared with upper and lower limits. If it exceeds the upper limit, it is truncated to the upper limit; if it is below the lower limit, it is truncated to the lower limit. The final output is the contact force correction within the constraint range. The typical range for the normal force gain factor is 0.5 to 1.5, and the typical range for the tangential coupling gain factor is 0.3 to 1.0. Specific values ​​are determined experimentally based on the end effector type and material surface characteristics. The upper and lower limits ensure the safety of the contact force correction: the upper limit prevents overloading or damage to the material due to excessive clamping by the lightly loaded robotic arm, and the lower limit prevents material detachment due to insufficient clamping force. The normal-tangential coupling mechanism enables the contact force correction to respond simultaneously to both normal bearing deviation and tangential slippage trend, achieving comprehensive optimization of the contact force.

[0076] Step S570: The contact force correction amount and the pose correction amount are superimposed and applied to the light-load robotic arm, wherein the contact force correction amount controls the gripping normal direction and the pose correction amount controls the tangential direction.

[0077] Specifically, the superposition effect refers to the final control command of the light-load robotic arm being dominated by different types of correction quantities in different directions, forming a force-position hybrid control mode. Specifically, along the gripping normal direction, the motion controller of the light-load robotic arm operates in force control mode, using the contact force correction quantity superimposed on the basic contact force setpoint as the force control target, so that the normal contact force tracks the corrected force value; along the tangential direction perpendicular to the gripping normal direction, the motion controller of the light-load robotic arm operates in position control mode, using the pose correction quantity superimposed on the basic following pose as the position control target, so that the end effector accurately tracks the corrected pose in the tangential direction.

[0078] The control modes for each direction are switched in the end effector of the light-duty robotic arm. A force controller is used along the gripping normal direction, whose input is the force error between the current normal contact force and the corrected target normal force. A proportional-integral force control law generates joint torque commands for the corresponding direction. A position controller is used along the direction perpendicular to the gripping normal direction, whose input is the position error between the current end effector pose and the corrected target pose. A proportional-derivative position control law generates joint torque commands for the corresponding direction. The joint torque commands output by the two controllers are superimposed in joint space and uniformly sent to the servo drives of each joint of the light-duty robotic arm for execution. This force-position hybrid control allows the light-duty robotic arm to maintain stable contact force while precisely adjusting its spatial pose. The two control objectives perform their respective functions without interference. Normal force control ensures that the material is always effectively supported or gripped without falling off, while tangential position control ensures that the material's posture is precisely adjusted without accumulating position errors.

[0079] In one embodiment, while performing real-time compensation based on pose correction and contact force correction, the heterogeneous robot cooperative processing method based on multimodal perception also includes a cooperative judgment process for boundary conditions. The cooperative judgment process includes, but is not limited to, the following steps: Step S610: Calculate the current load saturation of the light-load robotic arm and the time rate of change of the load saturation in real time, wherein the load saturation is the ratio of the second actual load to the second bearing capacity.

[0080] Specifically, load saturation is a dimensionless ratio characterizing the degree of occupancy of a lightly loaded robotic arm relative to its ultimate load capacity, ranging from 0 to 1. A load saturation of zero indicates that the lightly loaded robotic arm is not under load, while a load saturation of one indicates that the lightly loaded robotic arm has reached its rated load limit. The rate of change over time is the first derivative of the load saturation over time, reflecting the speed and direction of its change. A positive rate of change over time indicates that the load saturation is increasing, meaning the load condition of the lightly loaded robotic arm is deteriorating; a negative rate of change over time indicates that the load saturation is decreasing, meaning the load condition is improving.

[0081] Within each control cycle, the second actual load is read and divided by the second load-bearing capacity to obtain the load saturation value for the current control cycle. The time rate of change is calculated by dividing the difference between the load saturation of the current control cycle and the load saturation of the previous control cycle by the duration of the control cycle. To reduce the impact of sensor noise on the time rate of change, a moving average filter can be applied to the load saturation data of multiple consecutive control cycles before differential calculation. By simultaneously monitoring the absolute load level and load change trend of the lightly loaded robotic arm, a two-dimensional decision-making basis is provided for subsequent classification judgment. The absolute value of load saturation alone cannot distinguish between two different levels of danger: steady-state high load and rapidly deteriorating conditions. Introducing the time rate of change allows the system to distinguish between slowly fluctuating normal operation and rapidly deteriorating dangerous trends.

[0082] Step S620: When the load saturation is greater than the preset first tolerance threshold but not the preset second tolerance threshold, and the time change rate is less than or equal to the preset change rate threshold, the main motion trajectory and speed of the heavy-duty robotic arm are kept unchanged, and the pose correction amount of the light-duty robotic arm is compensated.

[0083] Specifically, the first tolerance threshold is a safe boundary value indicating that the lightly loaded robotic arm begins to require monitoring of its load-bearing state, typically ranging from 0.6 to 0.7. The second tolerance threshold is a critical boundary value indicating that the lightly loaded robotic arm is approaching a dangerous zone, typically ranging from 0.8 to 0.9. The rate of change threshold is a critical value for distinguishing between normal fluctuations and rapid deterioration in time rate of change, typically ranging from 0.1 to 0.2 per second. These thresholds are determined through calibration or testing based on the robotic arm's load-bearing capacity, the end effector's contact configuration, and safety requirements.

[0084] This process corresponds to the first-level response state of the lightly loaded robotic arm, where it is "somewhat strained but still able to hold on." At this point, the load saturation has exceeded the normal operating range but has not yet reached a dangerous level, and the trend of change is gradual, indicating that the system still has sufficient adjustment margin. Therefore, the heavy-load robotic arm continues to execute the main motion trajectory without being affected at all. Only by increasing the compensation gain of the pose correction of the light-load robotic arm can the light-load robotic arm increase its own correction force, attempting to transfer the excess load back to the heavy-load robotic arm side through pose adjustment.

[0085] First, it is determined that the current load saturation is between the first and second tolerance thresholds, and the rate of change over time does not exceed the rate of change threshold. Then, the stiffness value in the admittance control parameters of the light-load robotic arm in step S530 is adjusted according to a preset gain amplification factor, increasing the magnitude of the pose correction and thus enhancing the correction action of the light-load robotic arm in a direction favorable to load reduction. The typical gain amplification factor is 1.2 to 1.5 times. In most cases of load fluctuations caused by transient fluctuations or slight center of gravity shifts, the enhanced correction capability of the light-load robotic arm itself can eliminate the deviation without interfering with the main motion of the heavy-load robotic arm, ensuring that the handling cycle efficiency is not affected by slight fluctuations.

[0086] In step S630, when the load saturation is greater than the second tolerance threshold, or the load saturation is greater than the first tolerance threshold and the time change rate is greater than the change rate threshold, the main motion trajectory of the heavy-duty robotic arm is kept unchanged, the speed is reduced, and the load deviation of the light-duty robotic arm is recalculated, and real-time compensation is performed based on the updated load deviation.

[0087] Specifically, this process corresponds to the second-level response state of a lightly loaded robotic arm that is "about to collapse." The triggering condition is one of the following two: The first is that the load saturation directly exceeds the second tolerance threshold, indicating that the lightly loaded robotic arm is approaching its limit, and measures must be taken regardless of the trend of change; the second is that although the load saturation only exceeds the first tolerance threshold, the rate of change over time exceeds the rate of change threshold, indicating that although the current absolute value is not extreme, it is deteriorating rapidly, and will soon break through the limit if no intervention is provided.

[0088] In this scenario, a coordinated deceleration command is sent to the heavy-duty robotic arm. The heavy-duty robotic arm maintains its current main motion trajectory and spatial path direction, but its motion speed and acceleration amplitude are reduced by a deceleration ratio proportional to the amount by which the load saturation exceeds the first tolerance threshold. The deceleration ratio is calculated as follows: the difference between the current load saturation and the first tolerance threshold is divided by the difference between the second tolerance threshold and the first tolerance threshold, yielding a scaling factor between 0 and 1. This scaling factor is then multiplied by the current speed to obtain the target speed after deceleration. When the load saturation reaches the second tolerance threshold, the speed is reduced to a preset minimum maintenance ratio for normal speed. After the heavy-duty robotic arm decelerates, the dynamic load component caused by inertial force decreases due to the reduced motion acceleration, thus alleviating the actual load on the light-duty robotic arm. The second target load and load deviation of the light-duty robotic arm are recalculated using the new load state after deceleration, and the light-duty robotic arm is continuously driven to perform real-time compensation based on the updated load deviation. By reducing the movement speed of the heavy-duty robotic arm, dynamic load margins are released, alleviating the transient load pressure on the light-duty robotic arm. At the same time, the trajectory direction is not changed, allowing the handling task to continue in a decelerated state rather than being completely interrupted. The design that the deceleration magnitude is proportional to the degree of exceeding the limit ensures a gradual system response, avoiding an overreaction caused by slight fluctuations leading to a large deceleration.

[0089] Step S640: When the load saturation is less than the first tolerance threshold within the preset control cycle, the speed of the heavy-duty robotic arm is gradually restored.

[0090] Specifically, the preset number of control cycles is the length of the observation window for determining whether the load state has stabilized and returned to normal. Gradual recovery means that the speed is not immediately changed from the deceleration value to the normal value, but rather gradually and linearly increased according to the preset speed recovery slope.

[0091] By continuously monitoring load saturation, the load state is considered to have stabilized and returned to normal when the load saturation remains below the first tolerance threshold for a preset number of control cycles. Subsequently, a speed recovery process is initiated, gradually increasing the movement speed of the heavy-duty robotic arm at a preset recovery acceleration (e.g., 10% increase per second over normal speed) until it returns to normal operating speed. During the recovery process, load saturation is continuously monitored; if it exceeds the first tolerance threshold again, recovery is immediately paused and the current speed is maintained or the speed is reduced again. This gradual recovery strategy avoids the inertial force impact caused by sudden speed changes and prevents repeated oscillations of deceleration and recovery caused by a sudden increase in acceleration during speed recovery, ensuring the smoothness and continuity of the handling process.

[0092] In one embodiment, during the real-time compensation process based on updated load deviations, the heterogeneous robot cooperative processing method based on multimodal perception further includes, but is not limited to, the following steps: Step S650: Obtain the force rate of change characterizing the second actual load over time and the displacement rate of change characterizing the end effector motion of the lightly loaded robotic arm.

[0093] Specifically, the force change rate refers to the first derivative of the second actual load in the time dimension, reflecting the rate of change of the contact force on the side of the light-load robotic arm. This value is obtained by numerically differentiating the second actual load data within a continuous control cycle and applying a low-pass filter. The displacement change rate refers to the movement rate of the tool center point at the end of the light-load robotic arm in Cartesian space, reflecting the actual displacement change of the end of the light-load robotic arm due to the execution of correction actions. This value is obtained by calculating the end position from the joint encoder data of the light-load robotic arm using forward kinematics, and then differentiating the end position data within a continuous control cycle.

[0094] Within each control cycle, the force rate of change is obtained by subtracting the current value of the second actual load from the previous value and dividing by the control cycle duration. Similarly, the displacement rate of change is obtained by taking the norm of the difference between the current value of the light-load robotic arm's end-effector position and the previous value and dividing by the control cycle duration. To improve signal quality, second-order Butterworth low-pass digital filters with cutoff frequencies ranging from 5 Hz to 10 Hz are applied to both the raw force and position data before differential calculations. Synchronously acquiring dynamic change information from both the force and displacement channels provides a time-domain data foundation for subsequent multi-signal cross-validation.

[0095] Step S660: Within a preset time window, perform cross-correlation calculations on the rate of change of force perception and the rate of change of displacement to obtain the force-displacement cross-correlation coefficient.

[0096] Specifically, the preset time window refers to the time span of the sliding observation window used to calculate the cross-correlation coefficient, corresponding to a data length of twenty to fifty control periods. Cross-correlation is a statistical method for measuring the degree of linear correlation between two time series. The force-potential cross-correlation coefficient ranges from negative one to positive one; the closer the absolute value is to one, the more consistent the trends of the two signals, while the closer the absolute value is to zero, the less linear the relationship between the two signals.

[0097] Force rate of change and displacement rate of change sequences are collected within a preset time window. Both sequences are then mean-reduced, and their zero-delay cross-correlation coefficient is calculated. The calculation method is as follows: corresponding elements of the two mean-reduced sequences are multiplied one by one, summed, and then divided by the product of the root mean square values ​​of the two sequences. This operation is updated in a sliding window manner within each control cycle. Under normal collaborative handling conditions, there is a clear causal relationship between displacement changes and force changes at the end effector of the lightly loaded robotic arm; that is, end effector movement leads to force changes or force changes drive end effector correction, and the two exhibit a high cross-correlation coefficient. By quantifying the temporal correlation between force and displacement signals, a key statistical criterion for identifying material slippage is provided.

[0098] Step S670: If the absolute value of the force change rate is greater than the preset first abnormal threshold, the absolute value of the displacement change rate is less than the preset second abnormal threshold, and the force-displacement cross-correlation coefficient is lower than the preset lower limit of correlation, then the first abnormal working condition is determined to be triggered, the handling is suspended and the local self-correction state is entered to reduce the weight of the pose correction amount and increase the contact force correction amount.

[0099] Specifically, the first anomaly threshold is the critical value for judging anomalies in the absolute value of the force perception rate of change. When the force perception rate of change exceeds this threshold, it indicates that the force signal is changing rapidly. The second anomaly threshold is the critical value for judging anomalies in the absolute value of the displacement rate of change. When the displacement rate of change is below this threshold, it indicates that the end effector of the lightly loaded robotic arm is hardly moving. The lower limit of correlation is the critical value for judging anomalies in the force-displacement cross-correlation coefficient. The above parameters characterize a situation where the force is changing rapidly (large force perception rate of change) but the end effector of the robotic arm does not move significantly (small displacement rate of change), and the normal correspondence between the force change and the displacement change is lost (low cross-correlation coefficient). This signal characteristic pattern precisely matches the physical phenomenon of relative slippage of material on the surface of the end effector—the material sliding causes a change in the contact force distribution, resulting in a sudden change in the force signal, but the robotic arm itself does not move actively.

[0100] Furthermore, by combining the visual pose changes of the material to be transported detected by the visual perception module within a preset sampling interval, the existence of slippage is further verified when the relative displacement component of the material relative to the end effector of the lightly loaded robotic arm exceeds a preset visual offset threshold. Only when the above-mentioned force, displacement, and visual conditions are simultaneously met is the first abnormal condition triggered. This multi-signal cross-verification mechanism effectively avoids misjudgments caused by sensor noise, external vibration, or single-channel signal anomalies.

[0101] Local self-correction mode refers to pausing the collaborative handling progress (the heavy-duty robotic arm stops advancing its main motion trajectory) but not releasing the contact between the two robotic arms and the material, instead focusing on a specific correction mode to eliminate slippage. In local self-correction mode, the weight of the pose correction is reduced to a preset minimum hold value (e.g., 10% to 20% of the normal weight) to avoid exacerbating slippage by continuing to perform deviation-based pose correction when the material has already shifted position. Simultaneously, using the slippage characterization quantity as feedback input, the contact force correction of the light-duty robotic arm is gradually increased according to a preset anti-slip force increment strategy. The anti-slip force increment strategy is as follows: every preset increment interval (e.g., five control cycles), the contact force correction is increased by a preset increment step size (e.g., 5% of the current normal contact force). After each increment, the force-position cross-correlation coefficient is recalculated. If the cross-correlation coefficient recovers to above the lower limit of correlation and the slippage characterization quantity decreases to a safe range, the slippage is determined to be eliminated, and normal collaborative handling is resumed. Through a multi-signal fusion slippage identification method, the reliability of slippage detection is significantly improved under vibration, collision, and sensor noise conditions in real-world environments. The gradual force correction strategy avoids excessive clamping in one step, which could damage materials or overload the robotic arm. It eliminates slippage with the minimum necessary increment and immediately resumes normal handling, ensuring handling efficiency and material integrity.

[0102] In one embodiment, after entering a local self-correction state, the heterogeneous robot cooperative processing method based on multimodal perception further includes, but is not limited to, the following steps: Step S680: If the first abnormal condition exists after the preset correction time limit, or if the second abnormal condition is triggered by a sudden change in the second actual load, then a coordinated stop command is sent to the heavy-load robotic arm and the light-load robotic arm simultaneously. Specifically, the preset correction time limit is the maximum allowed duration of the system in a local self-correction state, typically set to 1 to 3 seconds. This time limit is set based on the following: if the gradual force application strategy fails to eliminate slippage within this time range, it indicates that the current abnormal state exceeds the unilateral correction capability of the lightly loaded robotic arm, and continued attempts may lead to more serious consequences. A sudden change refers to an event where the change in the second actual load exceeds a preset sudden change threshold within a single control cycle. The sudden change threshold is typically set to 20% to 30% of the second load capacity. Load sudden changes usually characterize extreme conditions such as material breakage, sudden failure of the end effector in contact with the material, or external collision. The triggering of the second abnormal condition is independent of the correction process of the first abnormal condition; that is, regardless of whether a correction operation is currently being performed, the second abnormal condition is directly triggered once a load sudden change is detected. The coordinated stop command is an emergency state command issued simultaneously to the motion controllers of both robotic arms. This command has the highest priority and can override any currently executing motion command. An upgrade mechanism for corrective failures and a direct response channel for extreme conditions have been established to ensure that the system can immediately enter a safety protection mode when its self-healing capabilities are exhausted or when unpredictable extreme events occur.

[0103] Step S690: In response to the coordinated stop command, control the heavy-duty robotic arm and the light-duty robotic arm to stop their spatial translation along the current motion trajectory and maintain the current contact force intensity; Specifically, stopping the spatial translation along the current trajectory means that both robotic arms immediately freeze the spatial translational motion of their ends, and the servo controllers of each joint switch to position-holding mode, using the joint angles at the moment of stopping as the holding target. Maintaining the current contact force intensity means locking the contact force values ​​applied to the material by each robotic arm at the moment of stopping as the continuous output target. The instantaneous normal contact force values ​​read by the force sensors of both robotic arms at the moment the stop command is received are recorded, and this value is continuously maintained as the constant target value of the force controller, ensuring that the material will not fall or shift due to sudden changes in contact force while stopped. This avoids two dangerous situations that may occur in traditional emergency stop schemes: first, if the clamping force is released simultaneously during an emergency stop, the material will fall under the action of gravity, which may cause material damage or personal injury; second, if the contact force changes suddenly during an emergency stop, due to the slight difference in the control response time of the two robotic arms, the instantaneous force imbalance may cause the material to bounce or flip. By locking the contact force, the material remains within the stable constraints of the two robotic arms after stopping.

[0104] Step S6100: While maintaining the current contact force intensity, acquire surrounding environmental information, plan the placement trajectory of the material to be transported to the safe bearing surface based on the environmental information, and control the heavy-duty robotic arm and the light-duty robotic arm to move the material to be transported onto the safe bearing surface based on the placement trajectory.

[0105] Specifically, environmental information refers to the surrounding spatial scene data collected by the visual perception module within the current work area, including the spatial position, orientation, and size information of each surface within the work area. The safe bearing surface refers to a horizontal or near-horizontal plane capable of bearing the weight of the material to be transported within the reachable spatial intersection of the two robotic arms, such as a worktable, conveyor belt surface, ground, or a dedicated buffer area.

[0106] The visual perception module acquires depth images or point cloud data of the current scene. A plane segmentation algorithm extracts horizontal plane regions from the scene, selecting planes with an area larger than the material's bottom area, closest to the current material position, and located within the intersection of the reachable spaces of the two robotic arms as candidate safe bearing surfaces. The closest candidate is chosen as the target safe bearing surface. Starting with the current material pose and aiming for the material's bottom surface to contact the safe bearing surface, a safe placement trajectory is planned, where the two robotic arms work together to translate the material along the shortest feasible path to above the safe bearing surface and then vertically descend. During trajectory execution, both robotic arms maintain a contact force no less than a preset percentage (e.g., 80%) of the lock value to prevent the material from falling off during movement.

[0107] Once the material reaches the safe bearing surface, the force sensing module detects whether the actual load on both robotic arms along the bearing direction continuously decreases to below the no-load threshold (e.g., 5% of the second bearing capacity) within a preset confirmation time (e.g., 0.5 to 1 second). If this condition is met, it is determined that the material has been fully supported by the safe bearing surface, and its weight is now borne by the bearing surface rather than the robotic arms. Subsequently, the two robotic arms are controlled to release contact with the material in the order of light-load robotic arm first, followed by heavy-load robotic arm. The reason for releasing the light-load robotic arm first is that the light-load robotic arm has a smaller load, and the transient force disturbance generated when it is released first is also smaller, with the least impact on the stability of the material; then the heavy-load robotic arm with a larger load is released, at which point the material is completely supported by the bearing surface, and the release of the heavy-load robotic arm will not cause material displacement. A complete closed-loop processing chain is formed from the occurrence of an anomaly to the safe landing of the material. It is not simply a matter of stopping and waiting for manual intervention, but rather autonomously completing the placement and release actions while ensuring the safety of the material, reducing reliance on manual intervention and ensuring the safety of materials, equipment, and personnel in abnormal situations. An orderly disengagement strategy further avoids the risk of transient instability that may occur when releasing two robotic arms at the same time due to incomplete synchronization of control timing.

[0108] This application provides a unified spatial target reference for two heterogeneous robotic arms through coordinate mapping of visual pose. Target load allocation based on load-bearing capacity ratio establishes a clear deviation criterion for force feedback, enabling the system to distinguish between normal load and parasitic internal forces caused by stiffness differences. The master-slave architecture allows the heavy-duty robotic arm to dominate the motion reference due to its high structural stiffness, while simultaneously granting the light-duty robotic arm with lower structural stiffness independent correction degrees of freedom from the master motion. The deviation between the actual load obtained from force perception and the target load directly drives the light-duty robotic arm to perform pose or contact force correction, ensuring that the differential deformation caused by the stiffness difference between the two arms is actively absorbed by the light-duty robotic arm in each control cycle, rather than passively accumulating into uncontrollable internal forces. Dynamically conforming coordinate system and adaptive admittance parameters ensure that the following strategy of the light-duty robotic arm remains optimal in any motion direction. Normal-tangential coupling contact force correction integrates anti-slip logic into normal control, reducing the frequency of abnormal triggering. Multi-level gradient response achieves a balance between safety and efficiency. Multi-signal fusion slip identification improves judgment reliability. Safe stopping and orderly placement closed-loop meet industrial safety standards.

[0109] like Figure 4 As shown, this application embodiment provides a heterogeneous robot collaborative processing device 100 based on multimodal perception, applied to a heterogeneous robot system including a heavy-duty robotic arm and a light-duty robotic arm, for handling the same material to be transported. The first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm. The heterogeneous robot collaborative processing device 100 based on multimodal perception acquires the visual spatial pose of the material to be transported through a data acquisition module 110, and maps the visual spatial pose to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm respectively to determine the collaborative contact pose; using a load division module 120, based on the ratio of the first load-bearing capacity and the second load-bearing capacity, it determines the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm; and using an attitude deviation calculation module 130 to... The heavy-duty robotic arm and the light-duty robotic arm move to a cooperative contact pose, jointly contacting and transporting the material to be transported. The attitude deviation between the current attitude and the target attitude of the material to be transported is calculated. The heavy-duty robotic arm moves according to a pre-planned main motion trajectory, and the light-duty robotic arm follows the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material. The load deviation calculation module 140 obtains the first actual load of the heavy-duty robotic arm along the load-bearing direction and the second actual load of the light-duty robotic arm along the load-bearing direction. The load deviation of the light-duty robotic arm is calculated using the second actual load and the second target load. Then, the assistance compensation module 150 generates the pose correction amount and contact force correction amount of the light-duty robotic arm based on the load deviation and attitude deviation. The light-duty robotic arm is controlled to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-duty robotic arm in transporting the material.

[0110] It should be noted that the data acquisition module 110 is connected to the load partitioning module 120, the load partitioning module 120 is connected to the attitude deviation calculation module 130, the attitude deviation calculation module 130 is connected to the load deviation calculation module 140, and the load deviation calculation module 140 is connected to the assistance compensation module 150. The aforementioned heterogeneous robot collaborative processing device based on multimodal perception is used to execute the aforementioned heterogeneous robot collaborative processing device based on multimodal perception.

[0111] It should also be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0112] This application also discloses an electronic device. (See reference...) Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0113] The communication bus 502 is used to enable communication between these components.

[0114] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0115] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0116] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 501.

[0117] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 5 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program based on a heterogeneous robot collaborative processing method using multimodal perception.

[0118] exist Figure 5In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and acquire user input data; while the processor 501 can be used to call an application program stored in the memory 505 for a heterogeneous robot collaborative processing method based on multimodal perception. When executed by one or more processors 501, the electronic device 500 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0119] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0120] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0124] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practical disclosure.

[0125] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A heterogeneous robot collaborative processing method based on multimodal perception, characterized in that, A method for handling the same material in a heterogeneous robotic system comprising a heavy-duty robotic arm and a light-duty robotic arm, wherein the first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm, comprises: The visual spatial pose of the material to be transported is obtained, and the visual spatial pose is mapped to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm, respectively, to determine the cooperative contact pose. Based on the ratio between the first load-bearing capacity and the second load-bearing capacity, determine the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm; The heavy-duty robotic arm and the light-duty robotic arm are moved to the cooperative contact pose to jointly contact and transport the material to be transported, and the attitude deviation between the current attitude and the target attitude of the material to be transported is calculated. The heavy-duty robotic arm moves according to the pre-planned main motion trajectory, and the light-duty robotic arm follows the movement of the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material. Obtain the first actual load of the heavy-duty robotic arm along the load-bearing direction and the second actual load of the light-duty robotic arm along the load-bearing direction, and calculate the load deviation of the light-duty robotic arm using the second actual load and the second target load; Based on the load deviation and the attitude deviation, the pose correction amount and contact force correction amount of the light-load robotic arm are generated, and the light-load robotic arm is controlled to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-load robotic arm in handling.

2. The method according to claim 1, characterized in that, The step of generating the pose correction amount and contact force correction amount of the light-load robotic arm based on the load deviation and the attitude deviation includes: The instantaneous main motion velocity vector of the end effector of the heavy-duty robotic arm at the current moment is obtained, and a dynamic compliant coordinate system that is updated in real time with the motion state of the heavy-duty robotic arm is constructed with the direction of the instantaneous main motion velocity vector as the first axis direction and the direction of the bearing plane perpendicular to the instantaneous main motion velocity vector as the second axis direction. The load deviation and the attitude deviation are projected onto the dynamic compliance coordinate system to obtain the parallel deviation component along the first axis and the orthogonal deviation component along the second axis. In the dynamic compliant coordinate system, spatially anisotropic admittance control parameters are configured for the lightly loaded robotic arm, wherein a first compliant stiffness is configured along the first axis direction, and a second sustaining stiffness is configured along the second axis direction, and the first compliant stiffness is less than the second sustaining stiffness. Vector synthesis is performed based on the impedance mapping between the first compliance stiffness and the parallel deviation component, and the impedance mapping between the second maintenance stiffness and the orthogonal deviation component, respectively. The synthesis result is then inversely transformed to the base coordinate system of the lightly loaded robotic arm to obtain the pose correction amount.

3. The method according to claim 2, characterized in that, The step of generating the pose correction amount and contact force correction amount of the light-load robotic arm based on the load deviation and the attitude deviation further includes: In the dynamic compliant coordinate system, the normal force deviation component along the clamping normal of the end effector of the light-load robotic arm and the tangential force deviation component along the tangential direction are extracted from the load deviation. When the tangential force deviation component exceeds the preset critical friction force, the excess tangential force deviation component is coupled to obtain the coupled tangential force deviation component. Establish a constraint coupling relationship between the normal force deviation component and the contact force correction amount. Within the constraint coupling relationship, calculate the contact force correction amount using the normal force deviation component and the coupled tangential force deviation component. The contact force correction amount and the pose correction amount are superimposed on the light-load robotic arm, wherein the contact force correction amount controls the gripping normal direction, and the pose correction amount controls the tangential direction.

4. The method according to claim 1, characterized in that, While performing real-time compensation based on the pose correction amount and the contact force correction amount, the method also includes a collaborative judgment process for boundary conditions, which includes: The current load saturation of the light-load robotic arm and the time change rate of the load saturation are calculated in real time, wherein the load saturation is the ratio of the second actual load to the second bearing capacity; When the load saturation is greater than the preset first tolerance threshold but not the preset second tolerance threshold, and the time change rate is less than or equal to the preset change rate threshold, the main motion trajectory and speed of the heavy-duty robotic arm are kept unchanged, and the pose correction amount of the light-duty robotic arm is compensated. When the load saturation is greater than the second tolerance threshold, or the load saturation is greater than the first tolerance threshold and the time change rate is greater than the change rate threshold, the main motion trajectory of the heavy-duty robotic arm is kept unchanged, the speed is reduced, and the load deviation of the light-duty robotic arm is recalculated, and real-time compensation is performed based on the updated load deviation. Within a preset control period, when the load saturation is less than the first tolerance threshold, the speed of the heavy-duty robotic arm is gradually restored.

5. The method according to claim 4, characterized in that, In the process of real-time compensation based on updated load deviations, the method further includes: Obtain the rate of change of force perception characterizing the second actual load over time, and the rate of change of displacement characterizing the end effector motion of the lightly loaded robotic arm; Within a preset time window, cross-correlation calculations are performed on the force-displacement cross-correlation coefficient to obtain the force-displacement cross-correlation coefficient. If the absolute value of the force change rate is greater than a preset first abnormal threshold, the absolute value of the displacement change rate is less than a preset second abnormal threshold, and the force-displacement cross-correlation coefficient is lower than a preset lower correlation limit, then the first abnormal working condition is determined to be triggered, the handling is suspended and a local self-correction state is entered, so as to reduce the weight of the pose correction amount and increase the contact force correction amount.

6. The method according to claim 5, characterized in that, After entering the local self-correction state, the method further includes: If the first abnormal condition exists after the preset correction time limit, or if the second abnormal condition is triggered by a sudden change in the second actual load, a coordinated stop command is sent synchronously to the heavy-load robotic arm and the light-load robotic arm. In response to the coordinated stop command, the heavy-duty robotic arm and the light-duty robotic arm are controlled to stop their spatial translation along the current motion trajectory and maintain the current contact force intensity; While maintaining the current contact force intensity, acquire surrounding environmental information, plan the placement trajectory of the material to be transported to the safe bearing surface based on the environmental information, and control the heavy-duty robotic arm and the light-duty robotic arm to move the material to be transported onto the safe bearing surface based on the placement trajectory.

7. The method according to claim 1, characterized in that, The step of mapping the visual spatial pose to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm respectively to determine the cooperative contact pose includes: Obtain the first pose transformation matrix of the coordinate system of the material to be transported relative to the camera coordinate system, output by the visual perception module; Read the pre-calibrated coordinate system transformation relationship, which includes the calibration matrix of the camera coordinate system relative to the global world coordinate system, the second pose transformation matrix of the global world coordinate system relative to the heavy-duty manipulator base coordinate system, and the third pose transformation matrix of the global world coordinate system relative to the light-duty manipulator base coordinate system. Based on the first pose transformation matrix and the calibration matrix, a homogeneous coordinate transformation operation is performed to obtain the absolute pose matrix of the material to be transported in the global world coordinate system. The absolute pose matrix is ​​multiplied by the second pose transformation matrix and the third pose transformation matrix to obtain the heavy-load target pose of the material to be transported in the heavy-load robot arm base coordinate system and the light-load target pose in the light-load robot arm base coordinate system. Extract the preset heavy-load contact offset matrix and light-load contact offset matrix of the material to be transported, synthesize the heavy-load target pose with the heavy-load contact offset matrix to determine the cooperative contact pose of the heavy-load robotic arm, and synthesize the light-load target pose with the light-load contact offset matrix to determine the cooperative contact pose of the light-load robotic arm.

8. A heterogeneous robot collaborative processing device based on multimodal perception, characterized in that, An apparatus for use in heterogeneous robotic systems comprising heavy-duty and light-duty robotic arms to handle the same material, wherein the first load-bearing capacity of the heavy-duty robotic arm is greater than the second load-bearing capacity of the light-duty robotic arm, the apparatus comprising: The data acquisition module is used to acquire the visual spatial pose of the material to be transported, and map the visual spatial pose to the base coordinate system of the heavy-duty robotic arm and the base coordinate system of the light-duty robotic arm respectively, so as to determine the cooperative contact pose. The load division module is used to determine the first target load corresponding to the heavy-duty robotic arm and the second target load corresponding to the light-duty robotic arm based on the ratio between the first load capacity and the second load capacity. The attitude deviation calculation module is used to move the heavy-duty robotic arm and the light-duty robotic arm to the cooperative contact pose, so that they can jointly contact and transport the material to be transported, and calculate the attitude deviation between the current attitude and the target attitude of the material to be transported. The heavy-duty robotic arm moves according to the pre-planned main motion trajectory, and the light-duty robotic arm follows the movement of the heavy-duty robotic arm to assist the heavy-duty robotic arm in transporting the material. The load deviation calculation module is used to obtain the first actual load of the heavy-duty robotic arm along the bearing direction and the second actual load of the light-duty robotic arm along the bearing direction, and to calculate the load deviation of the light-duty robotic arm using the second actual load and the second target load. The assistance compensation module is used to generate the pose correction amount and contact force correction amount of the light-load robotic arm based on the load deviation and the posture deviation, and control the light-load robotic arm to perform real-time compensation based on the pose correction amount and contact force correction amount while assisting the heavy-load robotic arm in handling.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, a communication bus, and a network interface. The processor, the memory, the user interface, and the network interface are respectively connected to the communication bus. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images