A dual-arm cooperation assembly method, device, system and medium of a robot

Abstract

The application relates to the technical field of robots, in particular to a dual-arm cooperation assembly method, device and system of a robot and a medium, which are used for solving the problems of low assembly effect and low assembly efficiency in the related art. The method comprises the following steps: obtaining assembly data of a product to be assembled from assembly data of multiple products based on a first image containing the product to be assembled in a robot operation area; each assembly data is input through a man-machine interaction interface and comprises process images and assembly demonstration data of multiple assembly processes of the product; performing feature comparison on the first image and the process images of each assembly process in the obtained assembly data respectively; determining an assembly task of the product to be assembled, wherein the assembly task comprises at least one subtask corresponding to an assembly process to be operated; and controlling dual arms of the robot to cooperatively complete assembly operation on the product to be assembled based on the assembly task and the assembly demonstration data in the obtained assembly data. The assembly efficiency is improved, and the assembly precision is ensured.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus, system and medium for collaborative assembly of two arms of a robot. Background Technology

[0002] In related technologies, for unstructured scenarios, it is often necessary to customize specific tooling to assist robotic arms in completing assembly tasks, which increases costs. Therefore, manual assembly is usually used. However, this manual assembly method has certain hidden risks. From a safety perspective, during manual assembly, workers inevitably encounter relatively dangerous assembly objects or parts, resulting in potential safety hazards and low safety. From an efficiency perspective, as assembly time increases and the assembly product models and tools change continuously, workers gradually become fatigued, leading to a decrease in assembly efficiency and accuracy. Reduced human attention also contributes to a decline in assembly quality and efficiency. Summary of the Invention

[0003] This application provides a robot dual-arm collaborative assembly method, device, system, and medium to improve assembly efficiency and accuracy, and to cope with assembly tasks in different scenarios, production lines, and products, effectively improving the robot's adaptability in tooling scenarios.

[0004] The specific technical solutions provided in this application are as follows:

[0005] In a first aspect, embodiments of this application provide a method for collaborative assembly of a robot's two arms, comprising:

[0006] Acquire a first image containing the product to be assembled within the robot's operating area;

[0007] Based on the first image, the assembly data of the product to be assembled is obtained from the assembly data of multiple products. Each assembly data is entered through a human-computer interaction interface and includes process images and assembly demonstration data of multiple assembly processes of the product.

[0008] The process images of each assembly process in the first image and the obtained assembly data are compared by feature comparison to determine the current assembly process of the product to be assembled.

[0009] Based on the current assembly process and each assembly process, the assembly task of the product to be assembled is determined, and the assembly task includes at least one sub-task corresponding to the assembly process to be operated.

[0010] Based on the assembly task and the assembly demonstration data obtained from the assembly data, the robot's two arms are controlled to work together to complete the assembly operation for the product to be assembled.

[0011] Using the above method, assembly data of various products can be obtained through images based on the assembly data of multiple products entered through the human-computer interaction interface. Then, based on the process images of each assembly process in the obtained assembly data, the current assembly process of the product to be assembled can be determined, and the assembly task of the product to be assembled can be determined. Then, based on the assembly task and the assembly demonstration data in the obtained assembly data, the robot's two arms can be controlled to complete the assembly operation collaboratively. This method does not require specific tooling to assist the robotic arm in completing the assembly task, has stronger scenario versatility, can be compatible with and understand multiple processes on the production line, and can realize the robot's autonomous understanding and generalizable operation. It can cope with assembly tasks of different scenarios, different production lines, and different products. In particular, it can effectively cope with production line changeovers and assembly tasks of different products, improve assembly efficiency, and at the same time ensure assembly accuracy, improve overall product quality and production stability.

[0012] In one possible implementation, each assembly data set further includes a main image and identification information of the product. Then, obtaining the assembly data of the product to be assembled from the assembly data of multiple products based on the first image includes:

[0013] Target recognition is performed on the first image;

[0014] If the icon of the product serial number of the product to be assembled is detected, the icon is scanned and identified to obtain the identification information of the product to be assembled.

[0015] If the main body of the product to be assembled is identified, the main body image in the assembly data of the multiple products is respectively matched with the target image of the area where the main body of the product is located in the first image. The identification information corresponding to the main body image that matches the target image is determined as the identification information of the product to be assembled.

[0016] Based on the identification information, the assembly data corresponding to the identification information is obtained from the assembly data of the various products.

[0017] Using the above method, the identification information of the product to be assembled can be determined by comparing the features of the image, thereby obtaining assembly data that can guide subsequent assembly operations based on the identification information.

[0018] In one possible implementation, if the current assembly process is not the last assembly process among all assembly processes, then determining the assembly task of the product to be assembled based on the current assembly process and all assembly processes includes:

[0019] From each of the assembly processes, obtain each subsequent assembly process following the current assembly process;

[0020] According to the process sequence of each assembly process, sub-tasks for the product to be assembled are generated sequentially based on the current assembly process and each subsequent assembly process.

[0021] Each generated subtask is identified as an assembly task for the product to be assembled.

[0022] Using the above method, the assembly task of the product to be assembled can be subdivided into multiple sub-tasks, which can be compatible with and understand multiple processes on the production line, realize the robot's autonomous understanding and generalizable operation, and improve the robot's intelligence.

[0023] In one possible implementation, controlling the robot's two arms to collaboratively complete the assembly operation for the product to be assembled, based on the assembly task and the assembly demonstration data obtained from the assembly data, includes:

[0024] According to the sequence of assembly processes corresponding to each subtask in the assembly task, the robot's two arms are controlled to work together to complete the assembly operation for the product to be assembled. Specifically, the following operations are performed for each subtask in the assembly task:

[0025] For the current subtask in the assembly task, obtain the assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data;

[0026] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product of the current sub-task, and the robot's right arm is controlled to perform assembly operations on the target product.

[0027] Wherein, if the current subtask is the first task in the assembly task, then the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, then the target product is an intermediate product obtained after assembly by the previous assembly process.

[0028] Using the above method, the robot's two arms can be guided to cooperate based on the obtained assembly demonstration data to complete specific assembly operations.

[0029] In one possible implementation, the method further includes:

[0030] If the obtained assembly demonstration data includes images of parts, then the current subtask is determined to be a parts assembly task;

[0031] The robot's right arm is controlled to retrieve a first target part from the loading system that matches the part in the part image;

[0032] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product, and the right arm is controlled to assemble the first target component onto the target product.

[0033] Using the above methods, the robot can autonomously determine the task type and clarify the assembly content. Based on the obtained assembly demonstration data, the robot can be guided to perform dual-arm collaboration to complete the assembly of parts, thereby improving assembly efficiency and enhancing the robot's intelligence. This enables the robot to autonomously understand and generalize its tasks.

[0034] In one possible implementation, the method further includes:

[0035] Obtain a second image of the target product;

[0036] Identify the hole positions of the first target component from the second image to obtain the first position of the first target component;

[0037] Based on the hole positions in the obtained assembly demonstration data and the first position, the offset is determined;

[0038] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target component onto the target product.

[0039] Using the above method, when there is a misalignment between the actual hole position (i.e., the first position identified by the second image) and the standard hole position in the assembly demonstration data, the position can be adjusted in time to improve the assembly effect, ensure the assembly quality, and reduce the product defect rate.

[0040] In one possible implementation, the method further includes:

[0041] If the obtained assembly demonstration data does not include accessory images, then the current subtask is determined to be the accessory fastening task;

[0042] Obtain a third image of the target product;

[0043] Based on the operation object in the obtained assembly demonstration data, the second target part corresponding to the operation object is identified in the third image;

[0044] Control the left arm to fix the target product, and control the right arm to perform a fastening operation on the second target component according to the torque requirements associated with the operation object in the obtained assembly demonstration data, thereby completing the current sub-task.

[0045] Using the above methods, the robot can autonomously determine the task type and clarify the assembly content. Based on the obtained assembly demonstration data, the robot can be guided to autonomously tighten the parts, enabling the robot's two arms to work together to complete the part tightening, thereby improving assembly efficiency and enhancing the robot's intelligence.

[0046] Secondly, embodiments of this application provide a robot's dual-arm collaborative assembly device, comprising:

[0047] The acquisition module is used to acquire a first image containing the product to be assembled within the robot's operating area;

[0048] The filtering module is used to obtain the assembly data of the product to be assembled from the assembly data of multiple products based on the first image. Each assembly data is entered through a human-computer interaction interface and includes process images and assembly demonstration data of multiple assembly processes of the product.

[0049] The process determination module is used to compare the process images of each assembly process in the first image and the obtained assembly data to determine the current assembly process of the product to be assembled.

[0050] The task determination module is used to determine the assembly task of the product to be assembled based on the current assembly process and each assembly process. The assembly task includes at least one sub-task corresponding to the assembly process to be operated.

[0051] An assembly module is used to control the robot's two arms to work together to complete the assembly operation for the product to be assembled, based on the assembly task and the assembly demonstration data obtained from the assembly data.

[0052] In one possible implementation, each assembly data also includes a main image and identification information of the product, in which case the filtering module is specifically used for:

[0053] Target recognition is performed on the first image;

[0054] If the icon of the product serial number of the product to be assembled is detected, the icon is scanned and identified to obtain the identification information of the product to be assembled.

[0055] If the main body of the product to be assembled is identified, the main body image in the assembly data of the multiple products is respectively matched with the target image of the area where the main body of the product is located in the first image. The identification information corresponding to the main body image that matches the target image is determined as the identification information of the product to be assembled.

[0056] Based on the identification information, the assembly data corresponding to the identification information is obtained from the assembly data of the various products.

[0057] In one possible implementation, if the current assembly process is not the last assembly process among all assembly processes, then the task determination module is specifically used for:

[0058] From each of the assembly processes, obtain each subsequent assembly process following the current assembly process;

[0059] According to the process sequence of each assembly process, sub-tasks for the product to be assembled are generated sequentially based on the current assembly process and each subsequent assembly process.

[0060] Each generated subtask is identified as an assembly task for the product to be assembled.

[0061] In one possible implementation, the assembly module is specifically used for:

[0062] According to the sequence of assembly processes corresponding to each subtask in the assembly task, the robot's two arms are controlled to work together to complete the assembly operation for the product to be assembled. Specifically, the following operations are performed for each subtask in the assembly task:

[0063] For the current subtask in the assembly task, obtain the assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data;

[0064] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product of the current sub-task, and the robot's right arm is controlled to perform assembly operations on the target product.

[0065] Wherein, if the current subtask is the first task in the assembly task, then the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, then the target product is an intermediate product obtained after assembly by the previous assembly process.

[0066] In one possible implementation, the assembly module is further configured to:

[0067] If the obtained assembly demonstration data includes images of parts, then the current subtask is determined to be a parts assembly task;

[0068] The robot's right arm is controlled to retrieve a first target part from the loading system that matches the part in the part image;

[0069] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product, and the right arm is controlled to assemble the first target component onto the target product.

[0070] In one possible implementation, the assembly module is further configured to:

[0071] Obtain a second image of the target product;

[0072] Identify the hole positions of the first target component from the second image to obtain the first position of the first target component;

[0073] Based on the hole positions in the obtained assembly demonstration data and the first position, the offset is determined;

[0074] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target component onto the target product.

[0075] In one possible implementation, the assembly module is further configured to:

[0076] If the obtained assembly demonstration data does not include accessory images, then the current subtask is determined to be the accessory fastening task;

[0077] Obtain a third image of the target product;

[0078] Based on the operation object in the obtained assembly demonstration data, the second target part corresponding to the operation object is identified in the third image;

[0079] Control the left arm to fix the target product, and control the right arm to perform a fastening operation on the second target component according to the torque requirements associated with the operation object in the obtained assembly demonstration data, thereby completing the current sub-task.

[0080] Thirdly, embodiments of this application provide a robotic dual-arm collaborative assembly system, comprising:

[0081] The input host is used to display the human-computer interaction interface and collect assembly data of various products entered by the user through the human-computer interaction interface. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product.

[0082] A feeding system is used to provide the robot with parts to be assembled;

[0083] The robot includes a memory for storing computer programs or instructions, and a processor for executing the computer programs or instructions in the memory, such that the methods described in any of the first aspects above are performed.

[0084] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.

[0085] Fifthly, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the architecture of an optional robot dual-arm collaborative assembly system in an embodiment of this application;

[0087] Figure 2 This is a schematic diagram of a human-computer interaction interface for entering product assembly data in an embodiment of this application;

[0088] Figure 3 This is a flowchart illustrating a robot's dual-arm collaborative assembly method according to an embodiment of this application.

[0089] Figure 4 This is a flowchart illustrating a method for filtering assembly data of products to be assembled from assembly data of multiple products according to an embodiment of this application.

[0090] Figure 5 This is a schematic diagram illustrating the process of determining the identification information of a product to be assembled in an embodiment of this application;

[0091] Figure 6 This is a schematic diagram of the architecture of an image detection and recognition network in an embodiment of this application;

[0092] Figure 7 This is a schematic diagram illustrating the process of determining the identification information of another product to be assembled in an embodiment of this application;

[0093] Figure 8 This is a schematic diagram illustrating a process for determining the current assembly step in an embodiment of this application;

[0094] Figure 9 This is a flowchart illustrating a method for determining the assembly task of a product to be assembled, as described in an embodiment of this application.

[0095] Figure 10 This is a schematic diagram illustrating the specific process of dual-arm collaborative assembly of a robot according to an embodiment of this application;

[0096] Figure 11 This is a schematic diagram of the processing flow of an accessory assembly task in an embodiment of this application;

[0097] Figure 12 This is a schematic diagram illustrating the processing of an accessory assembly task in an embodiment of this application;

[0098] Figure 13 This is a schematic diagram of a hole misalignment in an embodiment of this application;

[0099] Figure 14 This is a schematic diagram of the processing flow for a component fastening task in an embodiment of this application;

[0100] Figure 15 This is a schematic diagram illustrating the process of fastening an accessory in one embodiment of this application.

[0101] Figure 16 This is a schematic diagram of the logical architecture of a robot's dual-arm collaborative assembly device according to an embodiment of this application;

[0102] Figure 17 This is a schematic diagram of the physical architecture of the electronic device in the embodiments of this application. Detailed Implementation

[0103] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0104] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0105] The design concept of the embodiments of this application will be briefly introduced below.

[0106] This application relates to the field of robotics technology, and mainly to a method, apparatus, system and medium for collaborative assembly of a robot's two arms.

[0107] In related technologies, industrial robotic arm assembly is usually applicable to structured, standard, and general industrial scenarios. However, for unstructured scenarios, it is often necessary to customize specific tooling to assist the robotic arm in completing the assembly task, which increases costs. Therefore, manual assembly is usually used to complete the assembly task.

[0108] However, this manual assembly method has certain hidden risks. From a safety perspective, during the manual assembly process, it is inevitable to encounter relatively dangerous assembly objects or parts, which makes the assembly process potentially unsafe and less secure. From an efficiency perspective, as the assembly time increases and the assembly product models and tools change continuously, the workers will gradually become fatigued, leading to a decrease in the efficiency and accuracy of manual assembly. Reduced worker attention will also lead to a decrease in assembly effect and efficiency.

[0109] In view of this, in order to solve the problems of low assembly effect and low assembly efficiency in related technologies, this application provides a robot dual-arm collaborative assembly method. In this application embodiment, based on the first image of the robot operating area containing the product to be assembled, the assembly data of the product to be assembled is obtained from the assembly data of multiple products. Each assembly data is entered through a human-machine interface and includes process images and assembly demonstration data of multiple assembly processes of the product. The first image and the process images of each assembly process in the obtained assembly data are compared by feature comparison to determine the current assembly process of the product to be assembled, thereby determining the assembly task of the product to be assembled. The assembly task includes at least one sub-task corresponding to the assembly process to be operated. Then, based on the assembly task and the assembly demonstration data in the obtained assembly data, the robot's dual arms are controlled to collaboratively complete the assembly operation for the product to be assembled.

[0110] The above method is applicable to assembly tasks in unstructured scenarios. It does not require specific tooling to assist the robotic arm in completing assembly tasks, and has stronger scenario versatility. It can be compatible with and understand multiple processes on the production line, enabling the robot to autonomously understand and generalize its operations. It can handle assembly tasks in different scenarios, on different production lines, and for different products. In particular, it can effectively handle assembly tasks for production line changes and for different products, improving assembly efficiency while ensuring assembly accuracy, thus enhancing overall product quality and production stability.

[0111] The dual-arm collaborative assembly method for robots provided in this application is applicable to robots, or includes robot assembly systems, etc. The robot can be an intelligent device, a fixed-base robotic arm, a movable robotic arm, a multi-legged robot, a mobile robot, a humanoid / humanoid robot, a bipedal robot, a bionic human, etc.

[0112] In some embodiments, the aforementioned assembly system including the robot can be a dual-arm collaborative assembly system of the robot, see [reference]. Figure 1As shown, the system includes an input host, a feeding system, and a robot. The input host is used to display the human-machine interface and collect assembly data of various products entered by the user through the human-machine interface. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product. The feeding system is used to provide the robot with the parts to be assembled. The robot is used to execute a dual-arm collaborative assembly method of a robot according to an embodiment of this application to realize automated assembly operations.

[0113] in, Figure 1 The system shown may also include an image acquisition device installed on the production line to acquire images on the production line and transmit the images to the robot on the production line, so that the robot can perform the aforementioned robot dual-arm cooperative assembly method based on the acquired images, thereby realizing automatic robot assembly on the production line. At this time, the robot may or may not have image acquisition function, and this application does not limit it.

[0114] In some embodiments of this application, the assembly data to be entered may include the main image of the product, the product serial number, process images and assembly demonstration data of each assembly process, accessory images, three-dimensional (3D) structural data, user remarks, etc., wherein the main image of the product is used for product identification; accessory images are used for accessory identification; the product serial number includes the product model (such as ID, which can be called identification information) and other information; the assembly demonstration data includes assembly videos, assembly images and action sequences, the hole positions, dimensions, and quantities of accessories in the corresponding process images, the specifications, models, applicability, torque requirements of accessories, and quality inspection data, etc., used to guide the robot to perform specific assembly operations and select accessories, etc.; the assembly videos and action sequences can be pre-collected through teleoperation, customized remote operation interfaces, etc., and are assembly process data of left and right arm collaboration from the robot's perspective, and are robot operation process data, including robotic arm end data, robot perception image data, robot joint angle data, etc.; the quality inspection data is used to detect the assembly quality.

[0115] So, see Figure 2 As shown, the aforementioned human-computer interaction interface may include multiple data entry buttons (or touch areas), such as main image entry button, accessory image entry button, hole position entry button, product serial number entry button, accessory data entry button, etc., as well as thumbnails of the main image and accessory images, etc.

[0116] In this application, the entered assembly data can be used for information retrieval during the product assembly process. This allows for the detection of necessary component data, such as screw / clip data, simply by looking at images. This facilitates the robot's selection of tooling components and timely selection of the required components for the assembly task. By pre-entering the assembly data for each product, accurate component matching can be ensured for the robot or operator, optimizing the assembly process. For example, if the entered assembly data does not include component images, it can be understood as only a component fastening task, such as screw tightening. This effectively addresses the selection of tooling components when different product models are produced in combination, avoiding production errors, improving production efficiency, and ensuring assembly quality and product reliability.

[0117] The preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0118] See Figure 3 As shown in the figure, this application provides a method for collaborative assembly of a robot's two arms. The specific process of this method may include, but is not limited to, the following:

[0119] Step 300: Obtain the first image containing the product to be assembled within the robot's operating area.

[0120] In some embodiments of this application, the robot's image acquisition device can be used to acquire a first image containing the product to be assembled within the robot's operating area.

[0121] In other embodiments, an image acquisition device can be installed on the production line to acquire a first image containing the product to be assembled. This first image is then transmitted to a target robot via internal communication. The product to be assembled is located within the target robot's operating area. For example, the product to be assembled is... Figure 5 The washing machine shown.

[0122] To ensure the accuracy of subsequent image analysis, a high-resolution camera can be used to capture images of the robot's operating area containing the products to be assembled, ensuring image clarity. In some embodiments, the acquired first image can also be preprocessed, including adjusting brightness and contrast, removing noise, and performing possible image enhancements, to improve the accuracy of subsequent image analysis.

[0123] Step 310: Based on the first image, obtain the assembly data of the product to be assembled from the assembly data of multiple products. Each assembly data is entered through a human-computer interaction interface and includes process images and assembly demonstration data of multiple assembly processes of the product.

[0124] In some embodiments of this application, each assembly data also includes a main image of the product and identification information. Therefore, when performing step 310, refer to... Figure 4 As shown, the specific steps that can be performed include, but are not limited to, the following:

[0125] Step 3101: Perform target recognition on the first image.

[0126] In this embodiment of the application, when performing step 3101, an existing target recognition algorithm can be invoked to identify the target in the acquired first image, and subsequent steps 3102 or 3103 can be performed according to the different identified objects.

[0127] Step 3102: If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled.

[0128] In some embodiments, if the icon of the product serial number of the product to be assembled is identified from the first image, since the product serial number includes the product model, i.e., identification information, then step 3102 is executed to scan and identify the identified product serial number icon to obtain the product serial number of the product to be assembled, and the model of the product to be assembled is selected from the product serial number to obtain the identification information of the product to be assembled. The above process is as follows: Figure 5 As shown.

[0129] Step 3103: If the main body of the product to be assembled is identified, the main body images in the assembly data of multiple products are respectively matched with the target image of the area where the main body of the product is located in the first image. The identification information corresponding to the main body image that matches the target image is determined as the identification information of the product to be assembled.

[0130] In other embodiments, if the main body of the product to be assembled is identified from the first image, step 3103 is executed, in which the main body images from the first image and the assembly data of multiple products are input into a deep learning model. The key features of the main body in each image are extracted by the deep learning model, and the key features of the first image are matched with the key features of each main body image. The main body image that matches the target image is selected from the matching results, such as the main body image with the highest matching rate. The identification information corresponding to the selected main body image is determined as the identification information of the product to be assembled.

[0131] The aforementioned deep learning model can be any image recognition algorithm, such as the YOLO series of image detection and recognition networks. However, it should be noted that this application does not limit the image recognition algorithm; any image recognition algorithm can be used to complete the above feature extraction and comparison processing flow, filtering out the subject image that matches the target image. As a specific embodiment, the classic architecture of the aforementioned image detection and recognition network adopts the YOLOv8 network architecture, which is the latest iteration of the YOLO series network, capable of efficiently and accurately identifying and locating multiple objects in an image.

[0132] See Figure 6 As shown, the network architecture of the aforementioned image detection and recognition network can be divided into three main components: the backbone network, the neck layer (also known as the feature enhancement layer), and the head layer. The backbone network is responsible for extracting rich features from the original image, and may include modules such as CBS (Convolutions Bn SiL), C2f, and SPPF. The introduction of the C2f module, in particular, enhances feature representation capabilities while maintaining a lightweight model by parallelizing more gradient flow branches. The neck layer fuses features from different levels through a feature pyramid network (such as a Path Aggregation Network (PANet)), enhancing the model's ability to detect multi-scale targets. The head layer directly outputs the detection results, including the location of each predicted bounding box and the confidence score of its class. This network, with its depthwise separable convolutions, fully convolutional modules, and optimized training strategies, such as improved loss functions (e.g., VFL and DFL), significantly improves detection accuracy while maintaining high-speed inference.

[0133] During implementation, please refer to Figure 7 As shown, the acquired first image is input into the aforementioned image detection and recognition network. Assume the dimensions of the first image are H×W×C (for RGB images, H and W are the height and width of the image, respectively, and C is the number of channels, i.e., a value of 3). The feature sub-network in the image detection and recognition network can be based on the CSP (Cross Stage Partial) architecture, utilizing components such as CSPBlock and SPPF layers. This architecture reduces computational redundancy while extracting multi-level features from the input image. Then, through a Neck layer, such as PANet, feature fusion and multi-scale information integration can be performed, ensuring the network has good adaptability to targets of different sizes. Finally, through operations such as global average pooling, the feature map obtained after this series of processing is compressed to a 1x1x256 dimension, thereby condensing the rich information of the entire image into a 1*256 feature vector. This feature vector is rich in high-level semantic information of the image for subsequent feature matching.

[0134] The key features of the main body of the product to be assembled, extracted from the first image, are compared with the main body images in the pre-recorded assembly data of various products (such as...). Figure 7 Feature matching is performed on the main images of refrigerators, washing machines, televisions, etc. shown. Similarly, the main images in the assembly data of various products also use the aforementioned image detection and recognition network to extract their respective features. In some embodiments, the features of multiple main images can be extracted in advance through the aforementioned detection and recognition network and their respective features can be saved to facilitate feature matching during implementation and improve processing speed.

[0135] After feature matching of the features of the product to be assembled in the first image with the features of various subject images, the image detection and recognition network outputs the identification information corresponding to the subject image that matches the target image, thus obtaining the identification information of the product to be assembled.

[0136] Step 3104: Based on the identification information, obtain the assembly data corresponding to the identification information from the assembly data of various products.

[0137] In some embodiments of this application, when performing step 3104, based on the identification information of the product to be assembled output by the deep learning model, the assembly data corresponding to the identification information of the product to be assembled is filtered from the assembly data of various products, thus obtaining the assembly data of the product to be assembled. The assembly data includes process images of multiple assembly processes of the product to be assembled and assembly demonstration data. The assembly demonstration data can guide the robot to perform specific assembly operations, enabling the robot to imitate and complete the assembly operation of the product to be assembled based on the assembly demonstration data.

[0138] Step 320: Compare the features of the first image and the process images of each assembly process in the obtained assembly data to determine the current assembly process of the product to be assembled.

[0139] In some embodiments, when performing step 320, the key features of the first image can be compared with the features of the process images of each assembly process in the obtained assembly data, and the assembly process corresponding to the process image with the highest matching rate can be determined as the current assembly process of the product to be assembled.

[0140] For example, such as Figure 8As shown, assuming the washing machine assembly data includes process images of six assembly steps, namely: installing the lower counterweight, installing the upper counterweight, supplying and securing the front and rear drum covers, installing and securing the rear cover, installing and securing the top cover, and securing the soap dish screws. Then, by comparing the key features of the first image with the process images of the aforementioned six assembly steps, the current assembly step of the product to be assembled (the washing machine) can be obtained. Furthermore, assuming the first image has the highest feature matching rate with the process image of the fourth assembly step, the fourth assembly step is determined as the current assembly step of the product to be assembled (the washing machine), namely, the installation and securing of the rear cover.

[0141] Step 330: Based on the current assembly process and each assembly process, determine the assembly task of the product to be assembled. The assembly task includes at least one sub-task corresponding to the assembly process to be operated.

[0142] In some embodiments, if the current assembly process is the last assembly process in the obtained assembly data, then when performing step 330, the assembly task of the process to be assembled is the task corresponding to the last assembly process.

[0143] In other embodiments, if the determined current assembly process is not the last assembly process among the aforementioned assembly processes, then when executing step 330, refer to... Figure 9 As shown, the specifics may include, but are not limited to, the following:

[0144] Step 3301: Obtain the subsequent assembly processes after the current assembly process from each assembly process.

[0145] For example, see still Figure 8 As shown, assuming that the current assembly process of the product to be assembled (washing machine) is the 4th assembly process, in the specific implementation, when executing step 3301, based on the aforementioned 6 assembly processes and the current assembly process of the product to be assembled, namely the 4th assembly process, two subsequent assembly processes after the current assembly process can be obtained, namely the 5th assembly process and the 6th assembly process: top cover installation and fastening and soap box screw fastening.

[0146] Step 3302: According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, generate sub-tasks for the product to be assembled in sequence.

[0147] For example, see still Figure 8As shown, assuming the current assembly process of the product to be assembled is the 4th assembly process, and the obtained subsequent assembly processes are the 5th and 6th assembly processes, then when executing step 3302, a subtask can be generated based on the current assembly process, i.e., the 4th assembly process, denoted as subtask 1. Subtasks can also be generated based on the obtained subsequent assembly processes, i.e., the 5th and 6th assembly processes, denoted as subtask 2 and subtask 3 respectively.

[0148] Step 3303: Identify each generated subtask as an assembly task for the product to be assembled.

[0149] For example, see still Figure 8 As shown, assuming that the subtasks generated in sequence are subtask 1, subtask 2 and subtask 3, then when executing step 3303, subtask 1, subtask 2 and subtask 3 are determined as the assembly tasks of the product to be assembled (washing machine). That is, the assembly tasks for the product to be assembled include three subtasks: subtask 1, subtask 2 and subtask 3.

[0150] Thus, according to the method of this application, the overall assembly task for the product to be assembled can be automatically broken down into various sub-tasks. Subsequent assembly is then performed on each sub-task, enabling the robot to understand and perform multiple processes on the production line, achieving autonomous and generalizable robot operation, while ensuring assembly quality and product reliability. It should be noted that each assembly process can also correspond to multiple sub-tasks. For example, the fourth assembly process mentioned above, namely the installation and fastening of the rear cover, can be divided into two sub-tasks: rear cover installation and fastening. In other words, this application can specifically and meticulously break down the assembly task of the product to be assembled based on the actual scenario.

[0151] Step 340: Based on the assembly task and the assembly demonstration data obtained from the assembly data, control the robot's two arms to work together to complete the assembly operation for the product to be assembled.

[0152] In some embodiments of this application, when performing step 340, refer to Figure 10 As shown, the assembly operation for the product to be assembled can be completed by performing some or all of the following steps:

[0153] Step 3401: Following the sequence of assembly processes corresponding to each subtask in the assembly task, control the robot's two arms to work together to complete the assembly operation for the product to be assembled. Specifically, for each subtask in the assembly task, execute the following steps:

[0154] Step 34011: For the current subtask in the assembly task, obtain the assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data;

[0155] Step 34012: Based on the obtained assembly demonstration data, control the robot's left arm to fix the target product of the current subtask, and control the robot's right arm to perform assembly operations on the target product.

[0156] If the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly by the previous assembly process.

[0157] In related technologies, assembly tasks include component assembly tasks and component fastening tasks, for example, Figure 8 The six assembly steps shown can include the first, second, third, fourth and fifth assembly steps for assembling the parts, and the sixth assembly step for fastening the parts.

[0158] In this application, the specific task content of the current subtask is determined based on whether the assembly demonstration data of the assembly process corresponding to the current subtask obtained from the obtained assembly data includes accessory images.

[0159] In some embodiments, when performing step 34012, refer to Figure 11 As shown, the specific steps can be performed as follows:

[0160] Step 1100: If the obtained assembly demonstration data includes part images, then the current subtask is determined to be the part assembly task.

[0161] For example, such as Figure 12 As shown, the acquired assembly demonstration data includes accessory images, which include accessories and screws; therefore, the current subtask is determined to be the accessory assembly task.

[0162] Step 1110: Control the robot's right arm to retrieve the first target part that matches the part in the part image from the feeding system.

[0163] In some embodiments of this application, when performing step 1110, a first target accessory matching the accessory in the accessory image can be detected from the loading system by a target detection method based on the accessory in the accessory image. Alternatively, the area where the accessories are stored in the loading system can be pre-divided, and a type of accessory can be fixedly placed in the divided area. The information of the accessories stored on site is obtained by the robot in advance through a three-dimensional terrain construction method. In this way, the robot can quickly locate and obtain the accessories.

[0164] In practice, the acquisition of parts can be achieved through the robot's right-hand auxiliary tooling or through an end effector with magnetic attraction. Taking screws as the first target part as an example: First, the robot's end effector is typically equipped with magnetic, vacuum-holding, or custom-made screw clamps, tailored to the screw's material characteristics. For example, a strong magnetic head is used for metal screws to achieve stable and non-destructive pickup. Based on existing intelligent feeding systems, screws are rapidly identified and screened for quality before reaching the pickup position. This is similar to the aforementioned target detection method that identifies parts in a parts library, and screws matching the parts in the part image are placed on the conveyor belt of the feeding system. This ensures that the feeding system only delivers screws that meet the specifications to the robot.

[0165] After the robot identifies the required screw from the feeding system, it can control its right arm to precisely position itself above the screw through the operation of a high-precision servo system. The actuator of the right arm slowly descends and activates the suction mechanism at the appropriate time to steadily capture the screw.

[0166] Step 1120: Based on the obtained assembly demonstration data, control the robot's left arm to fix the target product, and control the right arm to assemble the first target component onto the target product.

[0167] In some embodiments, when performing step 1120, based on the assembly process data in the obtained assembly demonstration data, such as assembly videos, assembly images, or part or all of the action sequences, an open-source robotic arm (A Low-cost Open-source Hardware System for Bimanual Teleoperation, ALOHA) is invoked to control the robot's left arm to fix the target product and control the right arm to move according to the action sequence in the assembly process data. ALOHA is a low-cost, open-source, two-handed remote-controlled hardware system. Its algorithm, Action Chunking with Transformers (ACT), uses the Transformers neural network model and has imitation learning capabilities. Based on 15 minutes of demonstration data, the open-source robotic arm can learn an action, enabling end-to-end imitation learning directly from real demonstrations.

[0168] In some embodiments of this application, an optimal path from the gripping point of the first target component to the assembly position (i.e., the installation position of the first target component on the target product) can be calculated based on algorithms such as A* or RRT (Fast Random Tree Search) to enable the right arm to assemble the first target component onto the target product. The optimal path not only considers the shortest path but also comprehensively avoids obstacles in the working environment to ensure the safety and efficiency of the robot's actions.

[0169] During movement, high-precision encoders and closed-loop feedback control systems (such as PID control) ensure millimeter-level position and attitude control accuracy for the robot, guaranteeing stable operation even under complex working conditions. Furthermore, because the robot moves smoothly along a preset trajectory, relying on high-precision encoders and closed-loop feedback control systems during movement, it ensures that the first target component can be accurately assembled onto the target product.

[0170] In real-world scenarios, pre-assembled components on the target product, such as... Figure 12 The cover plate shown may have a certain gap or hole misalignment with the main body of the target product. Pressing the target product with the left arm of the robot can ensure a seamless connection between the assembled parts and the main body, but it cannot solve the problem of misalignment between the first hole on the assembled parts and the second hole on the main body. Figure 13 As shown, there is an error in the hole position between the circuit board and the housing when installing the circuit board. If the robot's right arm is directly controlled to assemble the parts (screws), the expected assembly effect may not be achieved.

[0171] In some embodiments, after controlling the robot's left arm to fix the target product, the following operations also need to be performed:

[0172] Operation 1: Obtain the second image of the target product.

[0173] To ensure the accuracy of subsequent position adjustments, a high-resolution camera must be used to photograph the target product to ensure that the image is clear and contains all necessary feature information, such as hole positions, edges, and shape. In some embodiments, the acquired second image may also be preprocessed, including adjusting brightness and contrast, removing noise, and performing possible image enhancements, to improve the accuracy of subsequent image analysis.

[0174] Operation 2: Identify the hole positions of the first target component from the second image to obtain the first position of the first target component.

[0175] In some embodiments of this application, when performing operation two, a traditional image detection algorithm, such as the YOLO algorithm, can be used to obtain the first position of the first target accessory, denoted as P. actual =(x actual y actual ).

[0176] Step 3: Determine the offset based on the hole positions and the first position in the obtained assembly demonstration data.

[0177] In some embodiments, if the first image contains multiple holes, an image matching algorithm, such as SIFT, SURF, ORB, etc., can be used to obtain the aforementioned offset based on the hole positions in the obtained assembly demonstration data and the first image.

[0178] For example, suppose the hole positions in the assembly demonstration data are denoted as P. ideal =(x ideal y ideal ), and the first position of the first target component is denoted as P. actual =(x actual y actual Therefore, the determined offset can be represented by a vector as Δ=(Δx,Δy), where Δx=x ideal -x actual Δy represents the distance moved along the x-axis, Δy = y ideal -y actual This represents the distance along the y-axis. A positive value indicates that the left arm needs to move to the right (positive x-axis direction) or upward (positive y-axis direction); a negative value indicates the opposite, that is, to move to the left (negative x-axis direction) or downward (negative y-axis direction).

[0179] In practical applications, it may also be necessary to consider factors such as the working range of the robotic arm, movement speed, and acceleration limits, and to make necessary adjustments or constraints on the offset to ensure the feasibility of the motion plan.

[0180] It should be noted that the above examples only involve movement along the x and y axes. In real-world scenarios, rotation angles may also be involved.

[0181] Operation 4: Based on the offset, control the left arm to adjust the position of the target product, and after the adjustment is completed, control the right arm to assemble the first target component onto the target product.

[0182] In this embodiment, when performing operation three, the distance that the left arm operating accessory (such as the aforementioned cover plate) needs to move along the x-axis and y-axis, and / or the rotation angle, etc., are calculated based on the offset. Then, when performing operation four, the left arm is controlled to adjust the position of the target product based on the offset so that the actual hole position is aligned with the ideal assembly position, i.e., the hole position position in the obtained assembly demonstration data. Then, after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

[0183] In some embodiments, the assembly demonstration data also includes quality inspection data. After the adjustment is completed, an image of the adjusted image can be obtained and compared with the quality inspection data. For example, feature matching can be performed between the image and the image in the quality inspection data to evaluate the adjustment effect. If the evaluation result meets the preset conditions, such as the deviation value being less than the deviation threshold, the subsequent operation of controlling the right arm to assemble is then executed.

[0184] In some embodiments of this application, in order to move efficiently and safely to the ideal assembly position, i.e. the hole positions in the obtained assembly demonstration data, path planning algorithms such as Dijkstra's algorithm and A* algorithm can be used for path planning. Especially in complex environments with obstacles, by using the above-mentioned path planning algorithms, obstacles can be effectively avoided based on obtaining the optimal path.

[0185] Continuing the previous example, assuming that controlling the robot's left arm to follow a straight path is sufficient for position adjustment, then the offset can be used directly to guide the robot's left arm to move. For example, suppose the actual detected center point of the screw hole (first position) is P. actual = (100, 200), the hole position in the obtained assembly demonstration data is P ideal = (110, 190), then the obtained Δx = x ideal -x actual =110-100=10, Δy=y ideal -y actual =190-200=-10, which means that the left arm needs to move 10 units along the positive x-axis and 10 units downward along the y-axis to complete the hole alignment operation.

[0186] Specifically, when a robot executes its current subtask based on path planning, it involves path parsing and instruction generation, motion control trajectory tracking, environmental perception and obstacle avoidance, and status monitoring and feedback. First, the path points output by the path planning algorithm are converted into motion instructions executable by the robot, ensuring trajectory tracking through precise control. During movement, the robot continuously monitors its surroundings using sensors, and can immediately assess and adjust its path when encountering obstacles. The input instructions for path parsing and instruction generation come from the results of the path planning algorithm, which is a sequential set of path points, each containing coordinates (x, y, z) and possible rotation angles (roll, pitch, yaw). It may also include motion parameters such as speed and acceleration limits. Upon receiving this data, the robot's control system converts it into an instruction format that the robot can understand and execute, such as a joint space path or a Cartesian space path. The robot's internal motion instruction set is then sent to the actuators for execution.

[0187] The input commands for motion control trajectory tracking are specific motion commands generated by the robot control system based on the parsed path commands for each joint or end effector. These commands can include joint angle, velocity, and acceleration commands. Through closed-loop control (such as PID control), the robot can monitor the deviation between its own position and the target path in real time and dynamically adjust the drive signals of each joint to ensure accurate tracking of the predetermined trajectory. The output commands are current or voltage signals sent to the motors or drivers, which are used to drive the robot's joints to move along the predetermined trajectory.

[0188] Environmental perception and obstacle avoidance typically rely on data collected from various sensors, such as LiDAR, cameras, and ultrasonic sensors, to monitor environmental conditions and potential obstacles in real time. Based on this data, or perception data obtained through processing, the robot dynamically assesses the safety of the current path and, if necessary, invokes obstacle avoidance algorithms to replan the path. Based on the obstacle avoidance results, the robot can either return to the path planning module to update the path or directly issue an emergency stop or path adjustment command to the motion control system.

[0189] Continuing the previous example, after the left arm adjusts the position of the target product, completes the alignment of the screw holes, and fixes the arm in place, awaiting the screw-fixing operation of the right arm. Specifically, the end effector of the robot's right arm precisely aligns with the screw hole on the target product, controlling the right arm to perform the screw-placement operation, that is, to place the screw into the screw hole. During this process, the attitude and force of the actuator can be finely adjusted to control the right arm to place the screw smoothly and accurately in the predetermined position. In some embodiments of this application, because the position is adjusted in advance, the positional accuracy is guaranteed, and further ensuring synchronization ensures that the screw can enter smoothly without damaging the workpiece.

[0190] Next, the assembly enters the torque control stage. Specifically, the robot's assembly head typically integrates an electric or pneumatic screwdriver, which has a built-in high-precision torque sensor and intelligent control system. The moment the screw begins to be screwed in, the preset torque and speed parameters are activated. Based on the screw type, material, and torque requirements obtained from the assembly demonstration data, the robot adjusts its control in real time through monitoring and feedback, controlling the right arm to fasten the first target component onto the target product. This ensures the screw is precisely tightened to the specified torque value, effectively avoiding workpiece damage due to overtightening or safety hazards caused by undertightening, thus achieving high efficiency and high standards in assembly.

[0191] As a specific embodiment, regarding the aforementioned current assembly process: rear cover installation and fastening, if the rear cover is not installed, the generated sub-task may include two specific assembly operations: installing the rear cover for the product to be assembled and fastening the rear cover with screws. The obtained assembly demonstration data includes a first component image and a second component image. The first component image corresponds to the first sub-target component, i.e., the rear cover, and the second component image corresponds to the second sub-target component, i.e., the screws.

[0192] After executing step 1110, control the robot's two arms or any one of its robotic arms to retrieve the first sub-target component, i.e., the back cover, that matches the component in the first component image from the loading system. Based on the obtained assembly demonstration data, place the back cover onto the target product. The assembly demonstration data can be an assembly video or an assembly image; the assembly image can be a washing machine assembled with the back cover. Then, control the robot's right arm to retrieve (grab) the second sub-target component, i.e., the screw, that matches the component in the second component image from the loading system. Based on the assembly video in the obtained assembly demonstration data, control the robot's left arm to press the back cover and the target product, and control the right arm to assemble the second sub-target component, i.e., the screw, onto the target product to fix the back cover onto the target product. Figure 12 As shown.

[0193] Among them, after controlling the robot's left arm to press the back cover and the target product, and before controlling the right arm to assemble the second sub-target accessory, i.e., the screw, onto the target product, operations one to four can be performed to adjust the position of the back cover that needs to be adjusted. See the above content for details, which will not be repeated here.

[0194] In other embodiments, when performing step 34012, see [reference needed]. Figure 14 As shown, the following steps can also be performed:

[0195] Step 1400: If the obtained assembly demonstration data does not include part images, then determine the current subtask as the part fastening task;

[0196] Step 1410: Obtain the third image of the target product;

[0197] Step 1420: Based on the operation object in the obtained assembly demonstration data, identify the second target part corresponding to the operation object in the third image;

[0198] Step 1430: Control the left arm to fix the target product, and control the right arm to perform a fastening operation on the second target component according to the torque requirements associated with the operation object in the obtained assembly demonstration data, thus completing the current sub-task.

[0199] To ensure the accuracy of subsequent image analysis, a high-resolution camera can be used to capture images of the target product, ensuring that the images are clear and contain all necessary feature information, such as edges and shapes. In some embodiments, the acquired third images can also be preprocessed, including adjusting brightness and contrast, removing noise, and performing possible image enhancements, to improve the accuracy of subsequent image analysis.

[0200] For example, such as Figure 15As shown, assuming the obtained assembly demonstration data does not include component images, then the current subtask is determined to be the component fastening task. Furthermore, assuming the obtained third image is as follows... Figure 15 As shown, the operation object is based on the obtained assembly demonstration data, i.e. Figure 15 In the left image shown, the screw is identified as a second target component corresponding to the object being manipulated in the third image. The left arm is then controlled to pinch the target products around the second target component, and the right arm is controlled to tighten the second target component according to the torque requirements associated with the object being manipulated in the obtained assembly demonstration data. Figure 15 This illustration only shows the tightening operation for one second target component; the tightening operations for other second target components are similar.

[0201] In some embodiments, multi-dimensional quality inspections can be performed after each assembly and / or tightening of the product to be assembled. Specifically, visual inspection technology can be used to capture images of assembly points, and the captured images can be analyzed based on quality inspection data from assembly demonstration data obtained for the product to be assembled, to verify the assembly and / or tightening of components, such as whether screws are fully embedded, whether there is any tilting or omission, and the overall flatness of the assembly surface. Furthermore, the torque of the tightened screws can be re-checked to ensure that the tightness of the screws meets the requirements.

[0202] In certain scenarios, sealing tests or functional tests may also be conducted to comprehensively evaluate the assembly effect. Through quality inspection, non-compliant assembly points will be marked and reported, triggering reassembly or manual intervention to further ensure product assembly quality.

[0203] Based on the same inventive concept, see [reference] Figure 16 As shown in the figure, this application provides a robot dual-arm collaborative assembly device, including:

[0204] The acquisition module 1610 is used to acquire a first image containing the product to be assembled within the robot's operating area;

[0205] The filtering module 1620 is used to obtain the assembly data of the product to be assembled from the assembly data of multiple products based on the first image. Each assembly data is entered through a human-computer interaction interface and includes process images and assembly demonstration data of multiple assembly processes of the product.

[0206] The process determination module 1630 is used to compare the process images of each assembly process in the first image and the obtained assembly data to determine the current assembly process of the product to be assembled.

[0207] The task determination module 1640 is used to determine the assembly task of the product to be assembled based on the current assembly process and each assembly process, wherein the assembly task includes at least one sub-task corresponding to the assembly process to be operated.

[0208] Assembly module 1650 is used to control the robot's two arms to work together to complete the assembly operation for the product to be assembled, based on the assembly task and the assembly demonstration data in the obtained assembly data.

[0209] In one possible implementation, each assembly data also includes a main image of the product and identification information, in which case the filtering module 1620 is specifically used for:

[0210] Target recognition is performed on the first image;

[0211] If the icon of the product serial number of the product to be assembled is detected, the icon is scanned and identified to obtain the identification information of the product to be assembled.

[0212] If the main body of the product to be assembled is identified, the main body image in the assembly data of the multiple products is respectively matched with the target image of the area where the main body of the product is located in the first image. The identification information corresponding to the main body image that matches the target image is determined as the identification information of the product to be assembled.

[0213] Based on the identification information, the assembly data corresponding to the identification information is obtained from the assembly data of the various products.

[0214] In one possible implementation, if the current assembly process is not the last assembly process among all assembly processes, then the task determination module 1640 is specifically used for:

[0215] From each of the assembly processes, obtain each subsequent assembly process following the current assembly process;

[0216] According to the process sequence of each assembly process, sub-tasks for the product to be assembled are generated sequentially based on the current assembly process and each subsequent assembly process.

[0217] Each generated subtask is identified as an assembly task for the product to be assembled.

[0218] In one possible implementation, the assembly module 1650 is specifically used for:

[0219] According to the sequence of assembly processes corresponding to each subtask in the assembly task, the robot's two arms are controlled to work together to complete the assembly operation for the product to be assembled. Specifically, the following operations are performed for each subtask in the assembly task:

[0220] For the current subtask in the assembly task, obtain the assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data;

[0221] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product of the current sub-task, and the robot's right arm is controlled to perform assembly operations on the target product.

[0222] Wherein, if the current subtask is the first task in the assembly task, then the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, then the target product is an intermediate product obtained after assembly by the previous assembly process.

[0223] In one possible implementation, the assembly module 1650 is further configured to:

[0224] If the obtained assembly demonstration data includes images of parts, then the current subtask is determined to be a parts assembly task;

[0225] The robot's right arm is controlled to retrieve a first target part from the loading system that matches the part in the part image;

[0226] Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product, and the right arm is controlled to assemble the first target component onto the target product.

[0227] In one possible implementation, the assembly module 1650 is further configured to:

[0228] Obtain a second image of the target product;

[0229] Identify the hole positions of the first target component from the second image to obtain the first position of the first target component;

[0230] Based on the hole positions in the obtained assembly demonstration data and the first position, the offset is determined;

[0231] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target component onto the target product.

[0232] In one possible implementation, the assembly module 1650 is further configured to:

[0233] If the obtained assembly demonstration data does not include accessory images, then the current subtask is determined to be the accessory fastening task;

[0234] Obtain a third image of the target product;

[0235] Based on the operation object in the obtained assembly demonstration data, the second target part corresponding to the operation object is identified in the third image;

[0236] Control the left arm to fix the target product, and control the right arm to perform a fastening operation on the second target component according to the torque requirements associated with the operation object in the obtained assembly demonstration data, thereby completing the current sub-task.

[0237] Based on the same inventive concept, this application provides a robot dual-arm collaborative assembly system, comprising:

[0238] The input host is used to display the human-computer interaction interface and collect assembly data of various products entered by the user through the human-computer interaction interface. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product.

[0239] A feeding system is used to provide the robot with parts to be assembled;

[0240] The robot, see reference Figure 17 As shown, it includes:

[0241] Memory 171 is used to store computer programs or instructions;

[0242] Processor 172 is configured to execute computer programs or instructions in memory 171 such that any of the methods described in the above embodiments is performed.

[0243] The processor 172 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 171, such that any of the methods described in the above embodiments are executed.

[0244] It should be noted that the specific connection medium between the memory 171 and the processor 172 is not limited in the embodiments of this application. Figure 17 In this diagram, the memory 171 and processor 172 are connected via bus 173. The connections between other components are merely illustrative and not intended to be limiting. Bus 173 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 17 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0245] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when executed by a processor, enables the processor to perform any of the methods described in the above embodiments. Since the principle by which the computer-readable storage medium solves the problem is similar to a robotic dual-arm collaborative assembly method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0246] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0247] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0248] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.

[0249] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0250] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for collaborative assembly of two robotic arms, characterized in that, include: Acquire a first image containing the product to be assembled within the robot's operating area; Based on the first image, the assembly data of the product to be assembled is obtained from the assembly data of various products pre-entered through the human-computer interaction interface. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product. The process images of each assembly process in the first image and the obtained assembly data are compared by feature comparison to determine the current assembly process of the product to be assembled. Based on the current assembly process and each assembly process, the assembly task of the product to be assembled is determined, and the assembly task includes at least one sub-task corresponding to the assembly process to be operated. According to the sequence of assembly processes corresponding to each subtask in the assembly task, the robot's two arms are controlled to work together to complete the assembly operation for the product to be assembled. Specifically, the following operations are performed for each subtask in the assembly task: For the current subtask in the assembly task, obtain the assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data; Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product of the current sub-task, and the robot's right arm is controlled to perform assembly operations on the target product. Wherein, if the current subtask is the first task in the assembly task, then the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, then the target product is an intermediate product obtained after assembly by the previous assembly process.

2. The method as described in claim 1, characterized in that, Each assembly data set also includes a main image and identification information of the product. Therefore, obtaining the assembly data of the product to be assembled from the assembly data of various products pre-entered through a human-computer interaction interface, based on the first image, includes: Perform target recognition on the first image; If the icon of the product serial number of the product to be assembled is detected, the icon is scanned and identified to obtain the identification information of the product to be assembled. If the main body of the product to be assembled is identified, the main body image in the assembly data of the multiple products is respectively matched with the target image of the area where the main body of the product is located in the first image. The identification information corresponding to the main body image that matches the target image is determined as the identification information of the product to be assembled. Based on the identification information, the assembly data corresponding to the identification information is obtained from the assembly data of the various products.

3. The method as described in claim 1, characterized in that, If the current assembly process is not the last assembly process among all assembly processes, then determining the assembly task of the product to be assembled based on the current assembly process and all assembly processes includes: From each of the assembly processes, obtain each subsequent assembly process following the current assembly process; According to the process sequence of each assembly process, sub-tasks for the product to be assembled are generated sequentially based on the current assembly process and each subsequent assembly process. Each generated subtask is identified as an assembly task for the product to be assembled.

4. The method as described in claim 1, characterized in that, The method further includes: If the obtained assembly demonstration data includes images of parts, then the current subtask is determined to be a parts assembly task; The robot's right arm is controlled to retrieve a first target part from the loading system that matches the part in the part image; Based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product, and the right arm is controlled to assemble the first target component onto the target product.

5. The method as described in claim 4, characterized in that, The method further includes: Obtain a second image of the target product; Identify the hole positions of the first target component from the second image to obtain the first position of the first target component; Based on the hole positions in the obtained assembly demonstration data and the first position, the offset is determined; Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target component onto the target product.

6. The method as described in claim 1, characterized in that, The method further includes: If the obtained assembly demonstration data does not include accessory images, then the current subtask is determined to be the accessory fastening task; Obtain a third image of the target product; Based on the operation object in the obtained assembly demonstration data, the second target part corresponding to the operation object is identified in the third image; Control the left arm to fix the target product, and control the right arm to perform a fastening operation on the second target component according to the torque requirements associated with the operation object in the obtained assembly demonstration data, thereby completing the current sub-task.

7. A robotic dual-arm collaborative assembly device, characterized in that, include: The acquisition module is used to acquire a first image containing the product to be assembled within the robot's operating area; The filtering module is used to obtain the assembly data of the product to be assembled from the assembly data of a variety of products pre-entered through the human-computer interaction interface based on the first image. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product. The process determination module is used to compare the process images of each assembly process in the first image and the obtained assembly data to determine the current assembly process of the product to be assembled. The task determination module is used to determine the assembly task of the product to be assembled based on the current assembly process and each assembly process. The assembly task includes at least one sub-task corresponding to the assembly process to be operated. The assembly module is used to control the robot's two arms to collaboratively complete the assembly operation for the product to be assembled, according to the process sequence of the assembly steps corresponding to each subtask in the assembly task. Specifically, for each subtask in the assembly task, the following operations are performed: for the current subtask, assembly demonstration data of the corresponding assembly step is obtained from the acquired assembly data; based on the obtained assembly demonstration data, the robot's left arm is controlled to fix the target product of the current subtask, and the robot's right arm is controlled to perform the assembly operation on the target product; wherein, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is an intermediate product obtained after assembly by the previous assembly step.

8. A robotic dual-arm collaborative assembly system, characterized in that, include: The input host is used to display the human-computer interaction interface and collect assembly data of various products entered by the user through the human-computer interaction interface. Each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product. A feeding system is used to provide the robot with parts to be assembled; The robot includes a memory for storing computer programs; A processor for executing a computer program in the memory such that the method described in any one of claims 1-6 is performed.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-6.

Images