Dual-arm robot wiring control method and system based on active perception

Abstract

The present invention proposes a dual-arm robot wiring control method and system based on active perception, which relates to the field of robot control technology. It includes constructing a high-level primitive selection imitation learning network, identifying the primitive that needs to be executed currently based on the local image, the global image and the position of the end effector of the operating robot arm; inputting the local image and the position of the end effector of the operating robot arm into the low-level primitive imitation learning network model, obtaining the next action of the operating robot arm, and updating the position of the end effector of the operating robot arm; inputting the global image, the position of the end effector of the perceived robot arm and the cable point cloud data into the active perception network model, obtaining the next action of the perceived robot arm, and updating the position of the end effector of the perceived robot arm; and looping the above steps based on the updated information to complete the entire wiring process. The present invention adopts an active perception method to reduce the self-occlusion rate of the robot operating deformable objects and improve the robot's autonomous wiring capability.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a dual-arm robot wiring control method and system based on active perception. Background Art

[0002] The routing of flexible cables is a crucial operation in the 3C and automotive manufacturing sectors. However, due to the challenges of deformable objects, such as complex deformation modeling, difficulty in detecting deformation, and the unlimited degrees of freedom of deformation, most wiring harness operations, such as wiring harness arrangement in the automotive industry, are still performed manually.

[0003] When a robot operates flexible cable wiring, the linear object is easily blocked by the robot's robotic arm and the surrounding environment. This results in a high visual self-occlusion rate and limited information, which affects the perception and operation of deformable linear objects.

[0004] Currently, existing robotic systems for manipulating linear objects mostly compensate for the high rate of visual self-occlusion by increasing the number of cameras. This increases system cost and limits the amount of supplementary visual information. Furthermore, the impact of visual self-occlusion on robotic operation remains. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a dual-arm robot wiring control method and system based on active perception. Based on the imitation learning hierarchical framework, the active perception method is adopted to reduce the self-occlusion rate of the robot when operating deformable objects and improve the robot's autonomous wiring capability.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a dual-arm robot wiring control method based on active perception.

[0008] A dual-arm robot wiring control method based on active perception, wherein the dual-arm robot includes a perception manipulator arm and an operation manipulator arm, includes the following steps:

[0009] Step 1: Acquire a local image through the perception manipulator, and obtain the perception manipulator end effector pose, the operation manipulator end effector pose, the global image, and the cable point cloud data representing the cable status;

[0010] Step 2: Build a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator. The primitives are the multiple subtasks that the wiring process is divided into.

[0011] Step 3: Construct low-level primitive imitation learning network model and active perception network model;

[0012] Step 4: Input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model to perform feature extraction and vector splicing to obtain the next action of the manipulator and update the end-effector pose of the manipulator.

[0013] Step 5: Input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model for feature extraction and vector concatenation to obtain the next action of the perception manipulator and update the perception manipulator end-effector pose.

[0014] Step 6: Based on the updated perceived robot arm end-effector pose, update the local image; based on the updated local image, global image and the operating robot arm end-effector pose, loop the above step 2 to obtain the primitives to be executed in the next step; based on the updated local image, the operating robot arm end-effector pose, global image, perceived robot arm end-effector pose and cable point cloud data, loop the above steps 4 to 5 until the entire wiring process is completed.

[0015] A second aspect of the present invention provides a dual-arm robot wiring control system based on active perception.

[0016] The dual-arm robot wiring control system based on active perception includes:

[0017] The data acquisition module is configured to: acquire a local image through the sensing manipulator, and acquire a position of an end effector of the sensing manipulator, a position of an end effector of the operating manipulator, a global image, and cable point cloud data representing a state of the cable;

[0018] The primitive recognition and division module is configured to: construct a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator, wherein the primitives are the multiple subtasks that the wiring process is divided into;

[0019] The model building module is configured to: build a low-level primitive imitation learning network model and an active perception network model;

[0020] The action module is configured to: input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model, perform feature extraction and vector splicing, obtain the next action of the manipulator, and update the end-effector pose of the manipulator;

[0021] The active perception module is configured to: input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model, perform feature extraction and vector splicing, obtain the next action of the perception manipulator, and update the perception manipulator end-effector pose;

[0022] The loop module is configured to: update the local image based on the updated perception robot arm end effector posture; loop the above step 2 based on the updated local image, the global image and the operation robot arm end effector posture to obtain the primitive to be executed in the next step; loop the above steps 4 to 5 based on the updated local image, the operation robot arm end effector posture, the global image, the perception robot arm end effector posture and the cable point cloud data until the entire wiring process is completed.

[0023] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the dual-arm robot wiring control method based on active perception as described in the first aspect of the present invention.

[0024] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the dual-arm robot wiring control method based on active perception as described in the first aspect of the present invention are implemented.

[0025] One or more of the above technical solutions have the following beneficial effects:

[0026] 1. The present invention provides a dual-arm robot wiring control method and system based on active perception, which applies the active perception method to cable wiring tasks with high visual occlusion rates. For a dual-arm robot system including a perception manipulator, an operating manipulator, a local camera and a global camera, wherein the local camera is deployed on the perception manipulator, the present invention constructs a low-level primitive imitation learning network model and an active perception network model. The next action of the operating manipulator is obtained through the low-level primitive imitation learning network model, and the next action of the perception manipulator is obtained through the active perception network model. Then, the perception manipulator drives the local camera to obtain a new local image, guides the execution action of the operating manipulator, enables the manipulator to continuously seek a perspective with richer information, reduces the self-occlusion rate of the robot operating deformable objects, and improves the robot's autonomous wiring capability.

[0027] 2. The present invention also designs a hierarchical framework based on imitation learning, constructing a high-level primitive selection imitation learning network to identify and classify primitives. The high-level primitive selection imitation learning network serves as the high-level strategy, while the low-level primitive imitation learning network model serves as the low-level strategy. The high-level strategy uses imitation learning to obtain primitive segmentation data and active perception of the robotic arm posture data through remote robot wiring, and trains the active perception network and the high-level network to output high-level primitive information. The low-level strategy uses imitation learning or heuristic algorithms to implement the operation of each primitive, thereby completing the robot wiring skill. Overall, the success rate of high-level classification and low-level operation is improved.

[0028] 3. This invention combines a hierarchical imitation learning strategy with a solution to the long-term problem of robotic cable routing by breaking down the routing task into distinct primitives. For the grasping, moving, spooling, winding, and inserting primitives, a heuristic algorithm is designed using data collected by force or tactile sensors on the manipulator arm. For the routing primitive, an imitation learning network model of the aforementioned low-level primitives is constructed to implement the solution. This reduces overall system cost and addresses the high visual self-occlusion rate and complex modeling issues associated with deformable object manipulation.

[0029] 4. The robot of the present invention can adapt to changes in the environment by learning and mastering the primitive skills in the wiring process, and has good generalization ability.

[0030] 5. In the process of obtaining the next action of the operating robot arm through the low-level primitive imitation learning network model, the local image obtained by the local camera and the position of the end effector of the operating robot arm are respectively input into the ResNet18 network and linear projection layer set in parallel in the low-level primitive imitation learning network model, and feature extraction is performed respectively. After splicing, the extracted features are input into the fully connected layer to obtain the next action of the operating robot arm.

[0031] 6. In the process of obtaining the next action of the perceived robotic arm through the active perception network model, the global image, the perceived robotic arm end effector posture and the cable point cloud data are respectively input into the ResNet18 network and two linear projection layers set in parallel in the active perception network model, and feature extraction is performed respectively. The extracted features are then input into the multi-layer MLP layer to obtain the next action of the perceived robotic arm.

[0032] 7. In the process of identifying and classifying primitives through the high-level primitive selection imitation learning network, the historical primitives are first constructed, and the global image and local image are respectively input into the ResNet18 network set up in parallel with the high-level primitive selection imitation learning network. The end effector posture of the operating robot arm and the historical primitives are linearly projected, and the features are extracted respectively. The extracted features are spliced ​​and input into the fully connected layer to complete the recognition and classification of the primitives.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 This is a flow chart of the method of the first embodiment.

[0036] Figure 2 Simplified structure diagram of the YOLOv10 network.

[0037] Figure 3 Flowchart for data processing of low-level primitive imitation learning network model.

[0038] Figure 4 This is the data processing flow chart of the active perception network model.

[0039] Figure 5 Flowchart of data processing for imitation learning network selection for high-level primitives.

[0040] Figure 6 This is a schematic diagram of a robot wiring control system according to embodiment 2 of the present invention.

[0041] Figure 7 This is a schematic diagram of a robot wiring system according to embodiment 4 of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0044] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0045] Example 1

[0046] As mentioned above, when existing robots operate flexible cable wiring, the linear objects are easily blocked by the robot's robotic arm and the surrounding environment. This results in a high visual self-occlusion rate and limited information, which affects the perception and operation of deformable linear objects. Increasing the number of cameras to compensate for the high visual self-occlusion rate will increase system costs and limit the amount of supplementary visual information.

[0047] In order to reduce system costs and solve the problems of high visual self-occlusion rate and complex modeling in the operation of deformable objects, the embodiment of the present invention is based on an imitation learning hierarchical framework and adopts an active perception method to propose a dual-arm robot wiring control method based on active perception. This method can reduce the self-occlusion rate of the robot when operating deformable objects and improve the robot's autonomous wiring capability.

[0048] like Figure 1 As shown, a dual-arm robot wiring control method based on active perception is provided, wherein the dual-arm robot includes a perception manipulator and an operation manipulator, wherein a local camera is deployed on the perception manipulator to obtain a local image, and a global camera is deployed in the wiring operation environment to obtain a global image and the status of the cable;

[0049] The following steps are involved:

[0050] A dual-arm robot wiring control method based on active perception is characterized in that the dual-arm robot includes a perception manipulator arm and an operation manipulator arm, and includes the following steps:

[0051] Step 1: Acquire a local image through the perception manipulator, and obtain the perception manipulator end effector pose, the operation manipulator end effector pose, the global image, and the cable point cloud data representing the cable status;

[0052] Step 2: Build a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator. The primitives are the multiple subtasks that the wiring process is divided into.

[0053] Step 3: Construct low-level primitive imitation learning network model and active perception network model;

[0054] Step 4: Input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model to perform feature extraction and vector splicing to obtain the next action of the manipulator and update the end-effector pose of the manipulator.

[0055] Step 5: Input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model for feature extraction and vector concatenation to obtain the next action of the perception manipulator and update the perception manipulator end-effector pose.

[0056] Step 6: Based on the updated perceived robot arm end-effector pose, update the local image; based on the updated local image, global image and the operating robot arm end-effector pose, loop the above step 2 to obtain the primitives to be executed in the next step; based on the updated local image, the operating robot arm end-effector pose, global image, perceived robot arm end-effector pose and cable point cloud data, loop the above steps 4 to 5 until the entire wiring process is completed.

[0057] It can be understood that during the real-time operation of the robotic arm, the high-level primitive selection imitation learning network outputs in real time what primitive should be executed at this time based on the current environmental information. The specific primitive operation is then implemented through the low-level primitive imitation learning network model, and the output of the high-level primitive selection imitation learning network also changes with the changes in the environment caused by the operation of the low-level primitive imitation learning network model, thereby guiding the system to enter the next primitive or continue to execute the current primitive.

[0058] The high-level primitive selection imitation learning network is used for the system to make primitive decisions, breaking down long sequence wiring tasks into primitives. The high-level strategy outputs the primitive sequence number that should be executed at this time in real time according to the divided primitive type, so that the system can better complete the task and improve the task success rate.

[0059] The high-level primitive selection imitation learning network outputs the current primitive number according to the environmental changes, executes the primitive through the low-level operation strategy, and then the high-level primitive will continue to output the primitive number according to the current environment, guiding the system to continue to complete the operation of the next primitive, and loop execution to the last primitive to complete the entire wiring task.

[0060] Through the above scheme, this embodiment applies the active perception method to cable wiring tasks with high visual occlusion rates. For a dual-arm robot system including a perception manipulator, an operating manipulator, a local camera, and a global camera, a low-level primitive imitation learning network model and an active perception network model are constructed. The next action of the operating manipulator is obtained through the low-level primitive imitation learning network model, and the next action of the perception manipulator is obtained through the active perception network model. The perception manipulator then drives the local camera to obtain a new local image, guides the execution action of the operating manipulator, enables the manipulator to continuously seek a perspective with richer information, reduces the self-occlusion rate of the robot when operating deformable objects, and improves the robot's autonomous wiring capability.

[0061] like Figure 1As shown, the embodiment of the present invention also designs a hierarchical framework based on imitation learning, integrating active perception methods to realize robot cable routing. In the network learning process, it is divided into high-level strategies and low-level strategies. The high-level strategy adopts the imitation learning method to obtain primitive division data by remotely controlling the robot routing; at the same time, the active perception network actively perceives the robot arm posture data, trains the active perception network, adjusts the perception of the robot arm posture, and then obtains more visual data to solve the problem of visual occlusion; at the same time, the high-level primitive selection imitation learning network is trained to identify primitives based on primitive division data to output high-level primitive information;

[0062] The low-level strategy uses imitation learning or heuristic algorithms to implement the operation of each primitive, thereby completing the robot's wiring skills.

[0063] In some embodiments, further comprising:

[0064] The routing process is divided into multiple subtasks, including grabbing, moving, spooling, winding, inserting and routing, with each subtask being a primitive;

[0065] If the primitive type identified by the imitation learning network is grasping, moving, spooling, winding, and inserting primitives, a heuristic algorithm is designed to implement the above primitives using data collected by the force sensor or tactile sensor set on the manipulator;

[0066] If the high-level primitive type selected by the imitation learning network is a wiring primitive, the above steps 3 to 6 are performed.

[0067] Next, the method of this embodiment will be explained in detail with reference to the accompanying drawings. Figure 2 As shown, the specific steps are as follows:

[0068] (1) Initialize the robotic arm.

[0069] The positions of the perception robot arm of the handheld camera and the end effector of the operating robot arm that performs wiring are defined as follows:

[0070] s1=[x1,y1,z1,α1,β1,γ1], s2=[x2,y2,z2,α2,β2,γ2];

[0071] The next steps of the sensing and operating robotic arms are:

[0072] a t1 =[Δx1,Δy1,Δz1,Δα1,Δβ1,Δγ1], a t2=[Δx2,Δy2,Δz2,Δα2,Δβ2,Δγ2]; the global image acquired by the global camera is represented as I1, the local image acquired by the local camera is represented as I2, the force sensor information is F, the tactile sensor information is T, and the state of the cable at time t is represented as a series of point clouds

[0073]

[0074] It can be understood that the above-mentioned force sensors and tactile sensors are both deployed on the operating robot arm to obtain force sensor information and tactile sensor information.

[0075] (2) Low-level primitive implementation.

[0076] Different routing scenarios have different primitives in their high-level strategies. Here, the primitives are set to six: grab, move, draw, wind, route, and insert. As you can see, the entire routing task is composed of different components, and the routing methods for different components are also different. The components in most routing platforms basically include the following:

[0077] For the three components of "wiring clip, winding pile, and socket", the three primitives of "wiring, winding, and insertion" are divided respectively;

[0078] The three primitives of "grabbing, moving, and straightening" are established to better complete the entire wiring task. That is, the primitive division is obtained through the experience of human remote operation in the early stage of completing the task. Such primitive division helps to complete the entire wiring process.

[0079] Here are the implementations of each primitive:

[0080] 1) Use a global camera to capture image data of each component to build an image dataset of wiring clips, wiring piles, and sockets, and use it to train YOLOv10. The specific process is as follows:

[0081] 1. Collect image data for each component and annotate its location in the image and the component category it belongs to: wiring clip, winding pile, or socket.

[0082] 2. Use the collected data to train YOLOv10, so that the network inputs the global image I1 and outputs the location and category of the components in the image. The simplified structure diagram of the YOLOv10 network is as follows: Figure 2 As shown, it includes Backbone layer, Neck layer and Head layer.

[0083] 3. The positions and directions of each component outputted are formed into a sequential list from the starting end to the ending end of the wiring for subsequent sequential wiring information extraction.

[0084] Specifically, after the low-level strategy completes the wiring of a primitive, it moves the robotic arm to a suitable position close to the next component by moving the primitive and adjusts the direction of the gripper to start the primitive operation of the next component. The position and direction of the movement can be extracted in sequence from the position and direction list output by YOLO.

[0085] 2) For the five primitives of grasping, moving, straightening, winding, and inserting, no learning approach is adopted. Instead, the simple primitives are implemented by designing heuristic algorithms based on the detected component information and using force or tactile sensors.

[0086] 3) For wiring primitives, due to the high degree of freedom and modeling complexity of cables, it is impossible to design them through a single heuristic algorithm. Therefore, an imitation learning network is constructed, that is, a low-level primitive imitation learning network model. The network structure is as follows Figure 3 As shown, it includes a ResNet18 network and a linear projection layer set in parallel, as well as two fully connected layers (fully connected layer 1 and fully connected layer 2).

[0087] The image data is passed through the Resnet network to obtain a feature vector, which is then concatenated with the vector obtained after linear mapping of the end pose and input into the fully connected layer to obtain the output.

[0088] Going further:

[0089] 1. Collect low-level wiring data, remotely imitate learning primitive trajectories, and construct the data set D1 = {(I2, s2, a2)}.

[0090] 2. The network strategy is π θ =(a2|I2,s2), the goal is to find a parameterized policy π θ , maximize the likelihood estimate of the current data set D1:

[0091]

[0092] 3. Train the network using the Adam optimizer.

[0093] (3) Construct an active perception network based on supervised learning. The network structure is as follows Figure 4 As shown, it includes a ResNet18 network and two linear projection layers set in parallel, followed by a 3-layer MLP layer.

[0094] Going further:

[0095] 1) Use a global camera to capture the environment image and use the non-rigid registration method (SPR) to obtain the point cloud data X during the robot cable operation process t .

[0096] 2) Remotely operate two robotic arms to collect active perception network data. During the operation of the wiring operation robotic arm, the perception robotic arm actively adjusts its posture to find a more informative perspective to form a data set D2 = {(X t ,I1,s1,a1)}.

[0097] 3) Network strategy is The goal is to optimize the strategy Use the Adam optimizer to train the network to maximize the likelihood estimate of the current data set D2:

[0098]

[0099] (4) Construct a high-level primitive selection imitation learning network. The network structure is as follows Figure 5 As shown, it includes two ResNet18 networks set up in parallel, two linear projection layers, and then two fully connected layers.

[0100] Furthermore, it also includes collecting high-level primitive data sets, manually dividing primitives, and labeling data sets for training high-level primitive selection imitation learning networks:

[0101] 1) Construct historical primitives: Use teleoperation to control the dual-arm robot to collect the entire wiring trajectory data. According to the specific wiring scenario, the wiring trajectory of the human teleoperation is divided into primitives and labeled according to experience, and the primitive selections at time t and the previous few times are saved as a sequence p t =[p1,p2,…,p n ], p n The sequence numbers corresponding to different primitives are 1, 2...6, and n moments ago p t Set to [0,0,…,0], save the high-level strategy data set as D2={(I1,I2,s2,p n )}.

[0102] The global image and local image are respectively input into the ResNet18 network set up in parallel with the high-level primitive selection imitation learning network to extract features;

[0103] Perform linear projection on the end effector pose of the manipulator and the historical primitives to extract features;

[0104] The extracted features are concatenated and input into the fully connected layer to identify the serial number corresponding to the primitive, thus completing the recognition and classification of the primitive.

[0105] 2) The network strategy is π Φ =(p|I1,I2,s2,p t ), since the network is used for classification, the cross entropy loss is used to update the network parameters:

[0106]

[0107] where y ij Indicates whether sample i belongs to the jth category, 1 means it belongs, 0 means it does not belong, It represents the probability that sample i output by the model belongs to the jth class, and the network is trained using the Adam optimizer.

[0108] (5) Execute the framework until the entire wiring task is completed.

[0109] The embodiment of the present invention designs a hierarchical framework based on imitation learning, constructs a high-level primitive selection imitation learning network, and identifies and classifies primitives through the high-level primitive selection imitation learning network; the high-level primitive selection imitation learning network is used as a high-level strategy, and the low-level primitive imitation learning network model is used as a low-level strategy; the high-level strategy adopts the imitation learning method, obtains primitive division data and active perception robot arm posture data through remote control robot wiring, trains the active perception network and the high-level network, and outputs high-level primitive information; the low-level strategy adopts imitation learning or heuristic algorithms to implement the operation of each primitive, thereby completing the robot wiring skills.

[0110] Example 2

[0111] This embodiment discloses a dual-arm robot wiring control system based on active perception.

[0112] like Figure 6 As shown in FIG, the dual-arm robot wiring control system based on active perception includes:

[0113] The data acquisition module is configured to: acquire a local image through the sensing manipulator, and acquire a position of an end effector of the sensing manipulator, a position of an end effector of the operating manipulator, a global image, and cable point cloud data representing a state of the cable;

[0114] The primitive recognition and division module is configured to: construct a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator, wherein the primitives are the multiple subtasks that the wiring process is divided into;

[0115] The model building module is configured to: build a low-level primitive imitation learning network model and an active perception network model;

[0116] The action module is configured to: input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model, perform feature extraction and vector splicing, obtain the next action of the manipulator, and update the end-effector pose of the manipulator;

[0117] The active perception module is configured to: input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model, perform feature extraction and vector splicing, obtain the next action of the perception manipulator, and update the perception manipulator end-effector pose;

[0118] The loop module is configured to: update the local image based on the updated perception robot arm end effector posture; loop the above step 2 based on the updated local image, the global image and the operation robot arm end effector posture to obtain the primitive to be executed in the next step; loop the above steps 4 to 5 based on the updated local image, the operation robot arm end effector posture, the global image, the perception robot arm end effector posture and the cable point cloud data until the entire wiring process is completed.

[0119] Also includes primitive partitioning module:

[0120] The routing process is divided into multiple subtasks, including grabbing, moving, spooling, winding, inserting and routing, and each subtask is regarded as a primitive.

[0121] It also includes primitive judgment module:

[0122] If the primitive type identified by the imitation learning network is grasping, moving, spooling, winding, and inserting primitives, a heuristic algorithm is designed to implement the above primitives using data collected by the force sensor or tactile sensor set on the manipulator;

[0123] If the high-level primitive type selected by the imitation learning network is a wiring primitive, the above steps 3 to 6 are performed.

[0124] More specifically, the above-mentioned action module is used to:

[0125] The local image and the pose of the end effector of the operating robot arm are respectively input into the ResNet18 network and linear projection layer set in parallel in the low-level primitive imitation learning network model, and feature extraction is performed respectively. After splicing, the extracted features are input into the fully connected layer to obtain the next action of the operating robot arm.

[0126] The above-mentioned active perception module is specifically used for:

[0127] The global image, the perceived robotic arm end-effector pose, and the cable point cloud data are respectively input into the ResNet18 network and two linear projection layers set in parallel in the active perception network model for feature extraction. The extracted features are then input into the multi-layer MLP layer to obtain the next action of the perceived robotic arm.

[0128] The network strategy of the low-level primitive imitation learning network model is represented by π θ=(a2|I2,s2), the goal is to find a parameterized policy π θ , maximize the likelihood estimate of data set D1:

[0129]

[0130] Among them, the data set D1 = {(I2, s2, a2)}; I2 is the local image; s2 is the position of the end effector of the operating robot arm; a2 is the next action of the operating robot arm.

[0131] The network strategy of the active sensing network model is expressed as The goal is to optimize the strategy Maximize the likelihood estimate of data set D2:

[0132]

[0133] Where D2={(X t ,I1,s1,a1)};X t is the cable point cloud data representing the cable status at time t; I1 is the global image; s1 is the position of the end effector of the perceived robotic arm; a1 is the next action of the perceived robotic arm.

[0134] It also includes a component recognition module, which is configured to train a YOLOv10 network model based on the acquired global image:

[0135] The global image is input into the YOLOv10 network model, which passes through the Backbone layer, Neck layer, and Head layer in sequence to obtain the location and category of components in the global image. The components include wiring clips, winding piles, and sockets.

[0136] The positions and directions of each component are obtained to form a sequential list according to the starting end to the ending end of the wiring; the sequential list is used to provide a reference for the moving position and direction of the operating robot arm after completing a primitive and before starting the primitive operation of the next component.

[0137] Furthermore, it also includes:

[0138] The high-level primitive selection imitation learning network primitive recognition module is configured as follows:

[0139] Construct historical primitives: divide the wiring trajectory of human teleoperation into primitives and annotate the image according to experience, and save the primitive selection at time t and before as a sequence p t =[p1,p2,…,p n ], where p n are the serial numbers corresponding to different primitives;

[0140] The global image and local image are respectively input into the ResNet18 network set up in parallel with the high-level primitive selection imitation learning network to extract features;

[0141] Perform linear projection on the end effector pose of the manipulator and the historical primitives to extract features;

[0142] The extracted features are concatenated and input into the fully connected layer to identify the serial number corresponding to the primitive, thus completing the recognition and classification of the primitive.

[0143] In the above primitive recognition module:

[0144] The cross entropy loss is used to update the network parameters of the high-level primitive selection imitation learning network:

[0145]

[0146] Among them, y ij Indicates whether sample i belongs to the jth category, 1 means it does, and 0 means it does not; It represents the probability that sample i output by the model belongs to the jth class.

[0147] Example 3

[0148] The purpose of this embodiment is to provide a computer-readable storage medium.

[0149] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the dual-arm robot wiring control method based on active perception as described in Example 1 of the present disclosure.

[0150] Example 4

[0151] The purpose of this embodiment is to provide an electronic device.

[0152] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the dual-arm robot wiring control method based on active perception as described in Example 1 of the present disclosure are implemented.

[0153] like Figure 7 As shown, this embodiment provides a hierarchical dual-arm robot wiring system based on active perception and imitation learning. The system hardware includes three parts: a perception module, a decision module, and an execution module. The decision module includes the electronic equipment in this embodiment.

[0154] The perception module includes a global camera, a local camera, a tactile sensor, and a force sensor.

[0155] The global camera is used to identify environmental information including the location of wiring clips, wiring piles, and sockets, the status of cables, and grab points;

[0156] The local camera is used by a robotic arm to actively perceive during the wiring process;

[0157] Tactile sensors and force sensors are used for the implementation of low-level wiring primitives.

[0158] The role of the decision module is to input the information collected by the perception module into the high-level imitation learning network and output operation primitives.

[0159] The execution module consists of two robotic arms: a perception arm and a manipulation arm. The manipulation arm implements the manipulation of specific primitives, while the perception arm adjusts the local camera position. The perception and manipulation arms feed environmental information back to the perception module until the last routing primitive is realized, completing the entire routing operation.

[0160] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0161] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0162] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A dual-arm robot wiring control method based on active perception, characterized in that: The dual-arm robot includes a sensing manipulator arm and an operating manipulator arm, and includes the following steps: Step 1: Acquire a local image through the perception manipulator, and obtain the perception manipulator end effector pose, the operation manipulator end effector pose, the global image, and the cable point cloud data representing the cable status; Step 2: Build a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator. The primitives are the multiple subtasks that the wiring process is divided into. Step 3: Construct low-level primitive imitation learning network model and active perception network model; Step 4: Input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model to perform feature extraction and vector splicing to obtain the next action of the manipulator and update the end-effector pose of the manipulator. Step 5: Input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model for feature extraction and vector concatenation to obtain the next action of the perception manipulator and update the perception manipulator end-effector pose. Step 6: Based on the updated perceived robot arm end-effector pose, update the local image; based on the updated local image, global image and the operating robot arm end-effector pose, loop the above step 2 to obtain the primitives to be executed in the next step; based on the updated local image, the operating robot arm end-effector pose, global image, perceived robot arm end-effector pose and cable point cloud data, loop the above steps 4 to 5 until the entire wiring process is completed.

2. The dual-arm robot wiring control method based on active perception according to claim 1, characterized in that: Also includes: The routing process is divided into multiple subtasks, including grabbing, moving, spooling, winding, inserting and routing, with each subtask being a primitive; If the primitive type identified by the imitation learning network is grasping, moving, spooling, winding, and inserting primitives, a heuristic algorithm is designed to implement the above primitives using data collected by the force sensor or tactile sensor set on the manipulator; If the high-level primitive type selected by the imitation learning network is a wiring primitive, the above steps 3 to 6 are performed.

3. The dual-arm robot wiring control method based on active perception according to claim 1, characterized in that: The network strategy of the low-level primitive imitation learning network model is represented by π θ =(a2|I2,s2), the goal is to find a parameterized policy π θ , maximize the likelihood estimate of data set D1: Among them, the data set D1 = {(I2, s2, a2)}; I2 is the local image; s2 is the position of the end effector of the manipulator; a2 is the next action of the manipulator; or, The network strategy of the active sensing network model is expressed as The goal is to optimize the strategy Maximize the likelihood estimate of data set D2: Where D2={(X t ,I1,s1,a1)};X t is the cable point cloud data representing the cable status at time t; I1 is the global image; s1 is the position of the end effector of the perceived robotic arm; a1 is the next action of the perceived robotic arm.

4. The dual-arm robot wiring control method based on active perception according to claim 1, characterized in that: The fourth step specifically includes: The local image and the end-effector pose of the manipulator are input into the ResNet18 network and linear projection layer set in parallel in the low-level primitive imitation learning network model respectively for feature extraction. The extracted features are concatenated and input into the fully connected layer to obtain the next action of the manipulator. or, The global image, the perceived robotic arm end-effector pose, and the cable point cloud data are respectively input into the ResNet18 network and two linear projection layers set in parallel in the active perception network model for feature extraction. The extracted features are then input into the multi-layer MLP layer to obtain the next action of the perceived robotic arm.

5. The dual-arm robot wiring control method based on active perception according to claim 1, characterized in that: It also includes training the YOLOv10 network model based on the acquired global image: The global image is input into the YOLOv10 network model, which passes through the Backbone layer, Neck layer, and Head layer in sequence to obtain the location and category of components in the global image. The components include wiring clips, winding piles, and sockets. The positions and directions of each component are obtained to form a sequential list according to the starting end to the ending end of the wiring; the sequential list is used to provide a reference for the moving position and direction of the operating robot arm after completing a primitive and before starting the primitive operation of the next component.

6. The dual-arm robot wiring control method based on active perception according to claim 1, characterized in that: It also includes training of high-level primitive selection imitation learning networks, including: Construct historical primitives: divide the wiring trajectory of human teleoperation into primitives and annotate the image according to experience, and save the primitive selection at time t and before as a sequence p t =[p1,p2,…,p n ], where p n are the serial numbers corresponding to different primitives; The global image and local image are respectively input into the ResNet18 network set up in parallel with the high-level primitive selection imitation learning network to extract features; Perform linear projection on the end effector pose of the manipulator and the historical primitives to extract features; The extracted features are concatenated and input into the fully connected layer to identify the serial number corresponding to the primitive, thus completing the recognition and classification of the primitive.

7. The dual-arm robot wiring control method based on active perception according to claim 6, characterized in that: The cross entropy loss is used to update the network parameters of the high-level primitive selection imitation learning network: Among them, y ij Indicates whether sample i belongs to the jth category, 1 means it does, and 0 means it does not; It represents the probability that sample i output by the model belongs to the jth class.

8. The dual-arm robot wiring control system based on active perception is characterized by: include: The data acquisition module is configured to: acquire a local image through the sensing manipulator, and acquire a position of an end effector of the sensing manipulator, a position of an end effector of the operating manipulator, a global image, and cable point cloud data representing a state of the cable; The primitive recognition and division module is configured to: construct a high-level primitive selection imitation learning network to identify the primitives that need to be executed based on the local image, the global image, and the position of the end effector of the manipulator, wherein the primitives are the multiple subtasks that the wiring process is divided into; The model building module is configured to: build a low-level primitive imitation learning network model and an active perception network model; The action module is configured to: input the local image and the end-effector pose of the manipulator into the low-level primitive imitation learning network model, perform feature extraction and vector splicing, obtain the next action of the manipulator, and update the end-effector pose of the manipulator; The active perception module is configured to: input the global image, the perception manipulator end-effector pose, and the cable point cloud data into the active perception network model, perform feature extraction and vector splicing, obtain the next action of the perception manipulator, and update the perception manipulator end-effector pose; The loop module is configured to: update the local image based on the updated perception robot arm end effector posture; loop the above step 2 based on the updated local image, the global image and the operation robot arm end effector posture to obtain the primitive to be executed in the next step; loop the above steps 4 to 5 based on the updated local image, the operation robot arm end effector posture, the global image, the perception robot arm end effector posture and the cable point cloud data until the entire wiring process is completed.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the dual-arm robot wiring control method based on active perception as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the dual-arm robot wiring control method based on active perception as described in any one of claims 1 to 7 are implemented.

Images