Double-arm cooperative assembly method, device and system of robot and medium

Abstract

The invention relates to the technical field of robots, in particular to a double-arm cooperative assembly method, device and system of a robot and a medium, and is used for solving the problems of low assembly effect and low assembly efficiency in related technologies. Assembling data of the to-be-assembled product is obtained from the assembling data of the multiple products, and each assembling data is input through a human-computer interaction interface and comprises process images of multiple assembling processes of the product and assembling demonstration data; feature comparison is carried out on the first image and a process image of each assembly process in the obtained assembly data, an assembly task of the to-be-assembled product is determined, and the assembly task comprises at least one sub-task corresponding to the to-be-operated assembly process; based on the assembly task and assembly demonstration data in the obtained assembly data, two arms of the robot are controlled to cooperatively complete assembly operation for the to-be-assembled product; the assembling efficiency is improved, and the assembling precision is ensured.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot dual-arm collaborative assembly method, device, system and medium. Background Art

[0002] In related technologies, for unstructured scenarios, it is often necessary to customize specific tooling to assist the robot arm to complete the assembly task, which increases the cost. Therefore, manual work is usually used to complete the assembly task. However, this manual assembly method has certain hidden risks. From a safety perspective, during the manual assembly process, manual work will inevitably encounter more dangerous assembly objects or accessories, which makes the assembly process have certain safety hazards and is less safe. From an efficiency perspective, as the assembly time increases, the assembly product models, assembly tools, etc. are constantly changing during the assembly process, and manual work will gradually become fatigued, resulting in reduced assembly efficiency and accuracy. Reduced manual attention will also lead to reduced assembly effects and efficiency. Summary of the invention

[0003] The embodiments of the present application provide a robot's dual-arm collaborative assembly method, device, system and medium to improve assembly efficiency, improve assembly accuracy, and be able to cope with assembly tasks in different scenarios, different production lines, and different products, effectively improving the robot's scene adaptability in tooling scenarios.

[0004] The specific technical solutions provided by the embodiments of this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a dual-arm collaborative assembly method of a robot, comprising:

[0006] Acquire a first image containing a product to be assembled in the robot operation area;

[0007] Based on the first image, obtaining assembly data of the product to be assembled from assembly data of multiple products, each assembly data is input through a human-computer interaction interface, including process images and assembly demonstration data of multiple assembly processes of the product;

[0008] Performing feature comparison between the first image and the process images of each assembly process in the obtained assembly data respectively, to determine the current assembly process of the product to be assembled;

[0009] Based on the current assembly process and the various assembly processes, determining an assembly task for the product to be assembled, wherein the assembly task includes at least one subtask corresponding to the assembly process to be operated;

[0010] Based on the assembly task and the assembly demonstration data in the obtained assembly data, the dual arms of the robot are controlled to collaboratively complete the assembly operation on the product to be assembled.

[0011] Through the above method, based on the assembly data of multiple products entered through the human-computer interaction interface, it is possible to obtain the assembly data of the product to be assembled through images, thereby determining the current assembly process of the product to be assembled based on the process images of each assembly process in the obtained assembly data, and then determining the assembly task of the product to be assembled. Then, based on the assembly task and the assembly demonstration data in the obtained assembly data, the robot's dual arms can be controlled to collaborate to complete the assembly operation. This method does not require specific tooling to assist the robot arm to complete the assembly task, has stronger scenario versatility, can be compatible with and understand multiple processes of the production line, and realize the robot's autonomous understanding of generalizable operations. It can cope with assembly tasks of different scenarios, different production lines, and different products, and can especially effectively cope with production line changes and assembly tasks of different products, thereby improving assembly efficiency, while also ensuring assembly accuracy and improving overall product quality and production stability.

[0012] In a possible implementation, each assembly data further includes a main body image and identification information of a product, and obtaining the assembly data of the to-be-assembled product from the assembly data of multiple products based on the first image includes:

[0013] performing target recognition on the first image;

[0014] If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled;

[0015] If the product body of the product to be assembled is identified, feature matching is performed on the main body images in the assembly data of the multiple products and the target images in the first image in the area where the product body is located, and identification information corresponding to the main body image matching the target image is determined as identification information of the product to be assembled;

[0016] Based on the identification information, assembly data corresponding to the identification information is obtained from assembly data of the plurality of products.

[0017] Through the above method, the identification information of the product to be assembled can be determined by comparing the features of the image, so that based on the identification information, assembly data that can guide subsequent assembly operations can be obtained.

[0018] In a possible implementation, if the current assembly process is not the last assembly process among the assembly processes, determining the assembly task of the product to be assembled based on the current assembly process and the assembly processes includes:

[0019] From the assembly processes, obtaining each subsequent assembly process after the current assembly process;

[0020] According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, sequentially generate subtasks for the product to be assembled;

[0021] The generated subtasks are determined as assembly tasks for the product to be assembled.

[0022] Through the above method, the assembly task of the product to be assembled can be subdivided into multiple subtasks, which can be compatible with and understand multiple processes of the production line, enable the robot to autonomously understand generalizable operations, and improve the intelligence of the robot.

[0023] In a possible implementation, based on the assembly task and the assembly demonstration data in the obtained assembly data, controlling the two arms of the robot to collaboratively complete the assembly operation for the product to be assembled includes:

[0024] According to the process sequence of the assembly process corresponding to each subtask in the assembly task, the two arms of the robot are controlled to collaborate to complete the assembly operation for the product to be assembled, wherein the following operations are performed for each subtask in the assembly task:

[0025] For a current subtask in the assembly task, obtaining assembly demonstration data of an assembly process corresponding to the current subtask from the obtained assembly data;

[0026] Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product of the current subtask, and control the right arm of the robot to perform an assembly operation on the target product;

[0027] Among them, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly in the previous assembly process.

[0028] Through the above method, the robot's dual-arm collaboration can be guided based on the obtained assembly demonstration data to complete specific assembly operations.

[0029] In a possible implementation, the method further includes:

[0030] If the obtained assembly demonstration data includes an accessory image, determining that the current subtask is an accessory assembly task;

[0031] Controlling the right arm of the robot to obtain a first target component that matches the component in the component image from a loading system;

[0032] Based on the obtained assembly demonstration data, the left arm of the robot is controlled to fix the target product, and the right arm is controlled to assemble the first target accessory onto the target product.

[0033] Through the above method, the robot can independently determine the task type and clarify the assembly content, and then guide the robot to complete the assembly of accessories through dual-arm collaboration based on the obtained assembly demonstration data, thereby improving assembly efficiency and the intelligence of the robot, and realizing the robot's autonomous understanding of generalizable operations.

[0034] In a possible implementation, the method further includes:

[0035] Acquire a second image of the target product;

[0036] Identify the hole position of the first target accessory from the second image to obtain a first position of the first target accessory;

[0037] Determine an offset based on the hole position in the obtained assembly demonstration data and the first position;

[0038] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

[0039] Through the above method, when the actual hole position (i.e., the first position identified by the second image) is misaligned with the standard hole position in the assembly demonstration data, the position can be adjusted in time to improve the assembly effect, ensure the assembly quality, and reduce the product defective rate.

[0040] In a possible implementation, the method further includes:

[0041] If the obtained assembly demonstration data does not include an accessory image, determining that the current subtask is an accessory fastening task;

[0042] Acquire a third image of the target product;

[0043] Based on the operation object in the obtained assembly demonstration data, identifying a second target accessory corresponding to the operation object in the third image;

[0044] The left arm is controlled to fix the target product, and the right arm is controlled to perform a tightening operation on the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data, so as to complete the current subtask.

[0045] Through the above method, the robot can independently determine the task type and clarify the assembly content, and then guide the robot to autonomously fasten accessories based on the obtained assembly demonstration data, so as to realize the robot's dual-arm collaboration to complete the fastening of accessories, thereby improving the assembly efficiency and the intelligence of the robot.

[0046] In a second aspect, an embodiment of the present application provides a dual-arm collaborative assembly device of a robot, comprising:

[0047] An acquisition module, used for acquiring a first image containing a product to be assembled in the robot operation area;

[0048] A screening module, configured to obtain assembly data of the product to be assembled from assembly data of multiple products based on the first image, each assembly data being input through a human-computer interaction interface and including process images and assembly demonstration data of multiple assembly processes of the product;

[0049] A process determination module, used for comparing the first image with the process images of each assembly process in the obtained assembly data, to determine the current assembly process of the product to be assembled;

[0050] A task determination module, used to determine the assembly task of the product to be assembled based on the current assembly process and the various assembly processes, wherein the assembly task includes at least one subtask corresponding to the assembly process to be operated;

[0051] The assembly module is used to control the two arms of the robot to collaboratively complete the assembly operation for the product to be assembled based on the assembly task and the assembly demonstration data in the obtained assembly data.

[0052] In a possible implementation, each assembly data also includes a main body image and identification information of the product, and the screening module is specifically used to:

[0053] performing target recognition on the first image;

[0054] If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled;

[0055] If the product body of the product to be assembled is identified, feature matching is performed on the main body images in the assembly data of the multiple products and the target images in the first image in the area where the product body is located, and identification information corresponding to the main body image matching the target image is determined as identification information of the product to be assembled;

[0056] Based on the identification information, assembly data corresponding to the identification information is obtained from assembly data of the plurality of products.

[0057] In a possible implementation, if the current assembly process is not the last assembly process among the assembly processes, the task determination module is specifically configured to:

[0058] From the assembly processes, obtaining each subsequent assembly process after the current assembly process;

[0059] According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, sequentially generate subtasks for the product to be assembled;

[0060] The generated subtasks are determined as assembly tasks for the product to be assembled.

[0061] In a possible implementation, the assembly module is specifically used to:

[0062] According to the process sequence of the assembly process corresponding to each subtask in the assembly task, the two arms of the robot are controlled to collaborate to complete the assembly operation for the product to be assembled, wherein the following operations are performed for each subtask in the assembly task:

[0063] For a current subtask in the assembly task, obtaining assembly demonstration data of an assembly process corresponding to the current subtask from the obtained assembly data;

[0064] Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product of the current subtask, and control the right arm of the robot to perform an assembly operation on the target product;

[0065] Among them, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly in the previous assembly process.

[0066] In a possible implementation, the assembly module is further used to:

[0067] If the obtained assembly demonstration data includes an accessory image, determining that the current subtask is an accessory assembly task;

[0068] Controlling the right arm of the robot to obtain a first target component that matches the component in the component image from a loading system;

[0069] Based on the obtained assembly demonstration data, the left arm of the robot is controlled to fix the target product, and the right arm is controlled to assemble the first target accessory onto the target product.

[0070] In a possible implementation, the assembly module is further used to:

[0071] Acquire a second image of the target product;

[0072] Identify the hole position of the first target accessory from the second image to obtain a first position of the first target accessory;

[0073] Determine an offset based on the hole position in the obtained assembly demonstration data and the first position;

[0074] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

[0075] In a possible implementation, the assembly module is further used to:

[0076] If the obtained assembly demonstration data does not include an accessory image, determining that the current subtask is an accessory fastening task;

[0077] Acquire a third image of the target product;

[0078] Based on the operation object in the obtained assembly demonstration data, identifying a second target accessory corresponding to the operation object in the third image;

[0079] The left arm is controlled to fix the target product, and the right arm is controlled to perform a tightening operation on the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data, so as to complete the current subtask.

[0080] In a third aspect, an embodiment of the present application provides a robot dual-arm collaborative assembly system, comprising:

[0081] An input host, used to display a human-computer interaction interface and collect assembly data of multiple products input by a user through the human-computer interaction interface, each assembly data including process images and assembly demonstration data of multiple assembly processes of the product;

[0082] A feeding system, used for providing the robot with accessories to be assembled;

[0083] The robot comprises a memory for storing computer programs or instructions; and a processor for executing the computer programs or instructions in the memory, so that any method in the first aspect described above is executed.

[0084] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods described in the first aspect above.

[0085] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, enables the computer to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a schematic diagram of the architecture of an optional robot dual-arm collaborative assembly system in an embodiment of the present application;

[0087] Figure 2 This is a schematic diagram of a human-computer interaction interface for entering assembly data of a product in an embodiment of the present application;

[0088] Figure 3 A schematic flow chart of a dual-arm collaborative assembly method of a robot in an embodiment of the present application;

[0089] Figure 4 It is a flowchart of a method for screening assembly data of a product to be assembled from assembly data of multiple products in an embodiment of the present application;

[0090] Figure 5 A schematic diagram of a process for determining identification information of a product to be assembled in an embodiment of the present application;

[0091] Figure 6 A schematic diagram of the architecture of an image detection and recognition network in an embodiment of the present application;

[0092] Figure 7 This is a schematic diagram of a process for determining identification information of another product to be assembled in an embodiment of the present application;

[0093] Figure 8 A schematic diagram of a process for determining a current assembly process in an embodiment of the present application;

[0094] Fig. 9 A flowchart of a method for determining an assembly task for a product to be assembled according to an embodiment of the present application;

[0095] Fig.10 A schematic diagram of a specific process of dual-arm collaborative assembly of a robot in an embodiment of the present application;

[0096] Fig.11 A schematic diagram of a processing flow of an accessory assembly task in an embodiment of the present application;

[0097] Fig.12 A schematic diagram of a processing process of an accessory assembly task in an embodiment of the present application;

[0098] Fig.13 A schematic diagram of hole position misalignment in an embodiment of the present application;

[0099] Fig.14 A schematic diagram of a processing flow of an accessory fastening task in an embodiment of the present application;

[0100] Fig.15 A schematic diagram of a processing process of an accessory fastening task in an embodiment of the present application;

[0101] Fig.16 A schematic diagram of the logical architecture of a dual-arm collaborative assembly device of a robot in an embodiment of the present application;

[0102] Fig.17 It is a schematic diagram of the physical structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0103] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0104] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0105] The following is a brief introduction to the design concept of the embodiments of the present application.

[0106] The present application relates to the field of robotics technology, and primarily to a robot's dual-arm collaborative assembly method, device, system, and medium.

[0107] In related technologies, industrial robot arm assembly is usually suitable for structured, standard, and general industrial scenarios. However, in unstructured scenarios, it is often necessary to customize specific tooling to assist the robot arm to complete the assembly task, which increases the cost. Therefore, assembly tasks are usually completed manually.

[0108] However, this manual assembly method has certain hidden risks. From a safety perspective, manual workers will inevitably encounter dangerous assembly objects or accessories during the manual assembly process, which makes the assembly process have certain safety hazards and is relatively unsafe. From an efficiency perspective, as the assembly time increases, the assembly product models, assembly tools, etc. are constantly changing during the assembly process, and workers will gradually become fatigued, resulting in reduced assembly efficiency and accuracy. Reduced human attention will also lead to reduced assembly results and efficiency.

[0109] In view of this, in order to solve the problems of low assembly effect and assembly efficiency in related technologies, an embodiment of the present application provides a robot's dual-arm collaborative assembly method. In the embodiment of the present application, based on a first image of a product to be assembled contained in an acquired robot operation area, assembly data of the product to be assembled is obtained from assembly data of multiple products, and each assembly data is entered through a human-computer interaction interface, including process images and assembly demonstration data of multiple assembly processes of the product; feature comparison is performed on the first image and the process images of each assembly process in the acquired assembly data, respectively, to determine the current assembly process of the product to be assembled, thereby determining the assembly task of the product to be assembled, and the assembly task includes at least one subtask corresponding to the assembly process to be operated; and then based on the assembly task and the assembly demonstration data in the acquired assembly data, the robot's dual arms are controlled to collaborate to complete the assembly operation on the product to be assembled.

[0110] The above method can be applied to assembly tasks in unstructured scenarios. It does not require specific tooling to assist the robotic arm in completing assembly tasks. It has stronger scenario versatility, can be compatible with and understand multiple processes on the production line, and can enable the robot to autonomously understand generalizable operations. It can cope with assembly tasks in different scenarios, different production lines, and different products, and can especially effectively cope with production line changes and assembly tasks of different products, thereby improving assembly efficiency. At the same time, it can also ensure assembly accuracy and improve overall product quality and production stability.

[0111] The dual-arm collaborative assembly method of a robot provided in the embodiment of the present application is applicable to a robot, or an assembly system including a robot, etc. The robot may be an intelligent device, a mechanical arm with a fixed base, a movable mechanical arm, a multi-legged robot, a mobile robot, a humanoid robot, a bipedal robot, a bionic man, etc.

[0112] In some embodiments, the aforementioned assembly system including a robot may be a robot dual-arm collaborative assembly system, see Figure 1As shown, the system includes an input host, a feeding system and a robot, wherein the input host is used to display a human-computer interaction interface and collect assembly data of multiple products entered by the user through the human-computer interaction interface, each assembly data includes process images and assembly demonstration data of multiple assembly processes of the product; the feeding system is used to provide the robot with accessories to be assembled; the robot is used to execute a robot's dual-arm collaborative assembly method in an embodiment of the present application to realize automated assembly operations.

[0113] in, Figure 1 The system shown may also include an image acquisition device installed on the production line, which is used to acquire images on the production line and transmit the images to the robot on the production line, so that the robot executes the aforementioned upper arm collaborative assembly method of the robot based on the acquired images, thereby realizing the robot automatic assembly of the production line. At this time, the robot may or may not have the image acquisition function, and this application does not limit it.

[0114] In some embodiments of the present application, the assembly data to be entered may include the main image of the product, the product serial number, the process images and assembly demonstration data of each assembly process, the accessory image, three-dimensional (3D) structure data, user note information, etc., wherein the main image of the product is used for product identification; the accessory image is used to identify the accessory; the product serial number includes the product model (such as ID, etc., which can be called identification information) and other information; the assembly demonstration data includes assembly video, assembly image and action sequence, the hole position, size, quantity, etc. of the accessory in the corresponding process image, the specification, model, applicability, torque requirement of the accessory, and quality inspection data, etc., which are used to guide the robot to perform specific assembly operations, screen accessories, etc.; the assembly video and action sequence can be pre-collected through remote operation, custom remote operation interface, etc., and are the assembly process data of the left and right arms working together from the perspective of the robot, and are the robot operation process data, including the end data of the robotic arm, the robot perception image data, the robot joint angle data, etc.; the quality inspection data is used to detect the assembly quality.

[0115] Then, see Figure 2 As shown, the aforementioned human-computer interaction interface may include multiple data entry buttons (or touch areas), such as a main body image entry button, an accessory image entry button, a hole position entry button, a product serial number entry button, an accessory data entry button, etc., as well as thumbnails of the main body image and thumbnails of the accessory image.

[0116] In this application, the entered assembly data can be used for information retrieval during the product assembly process, so that the accessory data required for assembly, such as screw / snap data, can be detected through images, which is convenient for the robot to screen tooling accessories and promptly select the accessories required for the assembly task. By pre-entering the assembly data of each product, it can be ensured that accurate accessory matching can be provided to the robot or operator, and the assembly process can be optimized. For example, if the accessory image is not included in the entered assembly data, it can be understood as only an accessory tightening task, such as screw tightening. In this way, it can effectively cope with the screening of tooling accessories during the mixed production of products of different models, avoid production errors, improve production efficiency, and also ensure assembly quality and product reliability.

[0117] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0118] See also Figure 3 As shown, the embodiment of the present application provides a robot dual-arm collaborative assembly method, and the specific process of the method may include but is not limited to the following:

[0119] Step 300: Acquire a first image containing a product to be assembled in an operating area of ​​the robot.

[0120] In some embodiments of the present application, an image acquisition device of the robot may be used to acquire a first image containing the product to be assembled in the operation area of ​​the robot.

[0121] In other embodiments, an image acquisition device may be installed on the production line to acquire a first image containing the product to be assembled through the image acquisition device, and then transmit the first image to the target robot through internal communication, wherein the product to be assembled is located in the operating area of ​​the target robot, such as the product to be assembled is Figure 5 Washing machine shown.

[0122] In order to ensure the accuracy of subsequent image analysis, a high-resolution camera may be used to shoot the products to be assembled in the robot operation area to ensure clear images, etc. In some embodiments, the first image collected may also be pre-processed, including adjusting brightness, contrast, removing noise, and possible image enhancement, etc., to improve the accuracy of subsequent image analysis.

[0123] Step 310: Based on the first image, assembly data of the product to be assembled is obtained from assembly data of multiple products, wherein each assembly data is input through a human-computer interaction interface, including process images and assembly demonstration data of multiple assembly processes of the product.

[0124] In some embodiments of the present application, each assembly data also includes a main image and identification information of the product. Then, when executing step 310, refer to Figure 4 As shown, the specific executable steps include but are not limited to the following:

[0125] Step 3101: Perform target recognition on the first image.

[0126] In an embodiment of the present application, when executing step 3101, an existing target recognition algorithm can be called to identify the target in the acquired first image, and subsequent step 3102 or step 3103 can be executed according to the different objects recognized.

[0127] Step 3102: If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled.

[0128] In some embodiments, if an icon of the product serial number of the product to be assembled is identified from the first image, since the product serial number includes the model of the product, that is, identification information, then step 3102 is executed to scan and identify the icon of the identified product serial number to obtain the product serial number of the product to be assembled, and the model of the product to be assembled is selected from the product serial number, so that the identification information of the product to be assembled can be obtained. The above process is as follows: Figure 5 shown.

[0129] Step 3103: If the product body of the product to be assembled is identified, feature matching is performed on the main body images in the assembly data of the multiple products and the target images of the area where the product body is located in the first image, and the identification information corresponding to the main body image matching the target image is determined as the identification information of the product to be assembled.

[0130] In other embodiments, if the product body of the product to be assembled is identified from the first image, step 3103 is executed, and the first image and the body images in the assembly data of multiple products are input into the deep learning model, and the key features of the product body in each image are respectively extracted through the deep learning model, and the key features of the first image are respectively matched with the key features of each body image, and the body image matching the target image is screened out from the matching results, such as the body image with the highest matching rate, and the identification information corresponding to the screened body image is determined as the identification information of the product to be assembled.

[0131] The aforementioned deep learning model can be any image recognition algorithm, such as the image detection and recognition network of the YOLO series. However, it should be noted that this application does not limit the image recognition algorithm, and any image recognition algorithm can be used to complete the above feature extraction and comparison processing flow to screen out the subject image that matches the target image. As a specific embodiment, the aforementioned image detection and recognition network classic architecture adopts the YOLOv8 network architecture, which is the latest iteration network of the YOLO series, which can efficiently and accurately identify and locate a variety of objects in the image.

[0132] See also Figure 6 As shown in Figure 1, the network architecture of the aforementioned image detection and recognition network can be divided into three major components: the backbone network, the neck network (Neck) layer (also known as the feature enhancement layer) and the head network (Head) layer. Among them, the backbone network is responsible for extracting rich features from the original image, which may include modules such as CBS (Convolutions Bn SiL), C2f and SPPF. In particular, the introduction of the C2f module enhances the feature expression capability while maintaining the lightweight model by parallelizing more gradient flow branches; the Neck layer fuses features of different levels through a feature pyramid network (such as the Path Aggregation Network (PANet)), enhancing the model's detection capability for multi-scale targets; the Head layer directly outputs the detection results, including the location of each predicted box and the confidence of the category to which it belongs. With its deep separable convolution, full convolution module and optimized training strategies, such as improved loss functions (such as VFL and DFL), the network significantly improves the detection accuracy while maintaining high-speed reasoning.

[0133] In progress, see Figure 7 As shown, the first image is input into the aforementioned image detection and recognition network, assuming that the dimension of the first image is H×W×C (for RGB images, H and W are the height and width of the image respectively, and C is the number of channels, i.e., 3). The feature subnetwork in the image detection and recognition network can be based on the CSP (Cross Stage Partial) architecture, using components such as CSPBlock and SPPF layers. This architecture can extract multi-level features from the input image while reducing computational redundancy; then through the Neck layer, such as PANet, feature fusion and multi-scale information integration can be performed to ensure that the network has good adaptability to targets of different sizes; finally, the feature is compressed into a 1x1x256 dimension after a series of processing through operations such as global average pooling, thereby condensing the rich information of the entire image into a 1*256 feature vector, which is rich in high-level semantic information of the image for subsequent feature matching.

[0134] The key features of the product body of the product to be assembled extracted from the first image are respectively compared with the main body images (such as Figure 7 Correspondingly, the main images in the assembly data of various products also use the aforementioned image detection and recognition network to extract their respective features. In some embodiments, the aforementioned detection and recognition network can be used in advance to extract features from multiple main images and save their respective features to facilitate feature matching during implementation and improve processing speed.

[0135] After matching the features of the product to be assembled in the first image with the features of multiple subject images respectively, the image detection and recognition network outputs identification information corresponding to the subject image matching the target image, thereby obtaining the identification information of the product to be assembled.

[0136] Step 3104: Based on the identification information, obtain assembly data corresponding to the identification information from assembly data of multiple products.

[0137] In some embodiments of the present application, when executing step 3104, based on the identification information of the product to be assembled output by the deep learning model, assembly data corresponding to the identification information of the product to be assembled is screened out from the assembly data of multiple products, that is, the assembly data of the product to be assembled is obtained, wherein the assembly data includes process images and assembly demonstration data of multiple assembly processes of the product to be assembled, and the assembly demonstration data can guide the robot to perform specific assembly operations, so that the robot can imitate and complete the assembly operations for the product to be assembled based on the assembly demonstration data.

[0138] Step 320: Perform feature comparison between the first image and the process images of each assembly process in the obtained assembly data to determine the current assembly process of the product to be assembled.

[0139] In some embodiments, when executing step 320, the key features of the first image may be compared with the features of the process images of each assembly process in the obtained assembly data, and the assembly process corresponding to the process image with the highest matching rate may be determined as the current assembly process of the product to be assembled.

[0140] For example, Figure 8As shown in the figure, it is assumed that the assembly data of the washing machine includes the process images of 6 assembly processes, which are installation of the lower counterweight, installation of the upper counterweight, nail supply and fastening of the front and rear covers of the drum outer drum, installation and fastening of the rear cover, installation and fixation of the top cover, and fastening of the soap box screws. Then, by comparing the key features of the first image with the process images of the above 6 assembly processes, the current assembly process of the product to be assembled (washing machine) can be obtained. It is also assumed that the feature matching rate between the first image and the process image of the fourth assembly process is the highest, then the fourth assembly process is determined as the current assembly process of the product to be assembled (washing machine), that is, installation and fastening of the rear cover.

[0141] Step 330: Based on the current assembly process and each assembly process, determine the assembly task of the product to be assembled, where the assembly task includes at least one subtask corresponding to the assembly process to be operated.

[0142] In some embodiments, if the determined current assembly process is the last assembly process of the assembly processes in the obtained assembly data, then when executing step 330, the determined assembly task of the to-be-assembled process is the task corresponding to the last assembly process.

[0143] In other embodiments, if the current assembly process determined is not the last assembly process among the aforementioned assembly processes, when executing step 330, refer to Fig. 9 The specific examples include but are not limited to the following:

[0144] Step 3301: From each assembly process, obtain each subsequent assembly process after the current assembly process.

[0145] For example, see Figure 8 As shown, it is still assumed that the current assembly process of the determined product to be assembled (washing machine) is the fourth assembly process. In the specific implementation, when executing step 3301, based on the aforementioned six assembly processes and the current assembly process of the product to be assembled, i.e., the fourth assembly process, two subsequent assembly processes after the current assembly process can be obtained, i.e., the fifth assembly process and the sixth assembly process: top cover installation and tightening and soap box screw tightening.

[0146] Step 3302: According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, subtasks for the product to be assembled are generated in sequence.

[0147] For example, see Figure 8As shown, assuming that the current assembly process of the product to be assembled is the 4th assembly process, and the subsequent assembly processes obtained are the 5th assembly process and the 6th assembly process, then, when executing step 3302, a subtask can be generated based on the current assembly process, i.e., the 4th assembly process, and recorded as subtask 1, and based on the subsequent assembly processes obtained, i.e., the 5th assembly process and the 6th assembly process, respective subtasks can be generated, recorded as subtask 2 and subtask 3 respectively.

[0148] Step 3303: Determine each generated subtask as an assembly task for the product to be assembled.

[0149] For example, see Figure 8 As shown, assuming that the subtasks generated in sequence are subtask 1, subtask 2 and subtask 3, then when executing step 3303, subtask 1, subtask 2 and subtask 3 are determined as assembly tasks for the product to be assembled (washing machine), that is, the assembly task for the product to be assembled includes three subtasks: subtask 1, subtask 2 and subtask 3.

[0150] In this way, according to the method of the present application, the overall assembly task for the product to be assembled can be automatically split into various subtasks, and then specific assembly can be performed for each subtask, which can be compatible with and understand multiple processes of the production line, and realize the robot's autonomous understanding of generalizable operations, while ensuring the assembly quality and product reliability. It should be noted that each assembly process can also correspond to multiple subtasks, such as the aforementioned fourth assembly process, namely the installation and tightening of the rear cover, the installation of the rear cover can be used as a subtask, and the tightening can be used as another subtask. In other words, the present application can perform specific and detailed decomposition of the assembly tasks of the product to be assembled based on the actual scenario.

[0151] Step 340: Based on the assembly task and the assembly demonstration data in the obtained assembly data, control the robot's dual arms to collaboratively complete the assembly operation for the product to be assembled.

[0152] In some embodiments of the present application, when executing step 340, refer to Fig.10 As shown, the assembly operation for the product to be assembled can be completed by executing part or all of the following steps:

[0153] Step 3401: According to the process sequence of the assembly process corresponding to each subtask in the assembly task, control the robot's two arms to collaboratively complete the assembly operation for the product to be assembled, wherein the following steps are performed for each subtask in the assembly task:

[0154] Step 34011: for the current subtask in the assembly task, obtain assembly demonstration data of the assembly process corresponding to the current subtask from the obtained assembly data;

[0155] Step 34012: Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product of the current subtask, and control the right arm of the robot to perform assembly operations on the target product.

[0156] Among them, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly in the previous assembly process.

[0157] In the related art, assembly tasks include component assembly tasks and component fastening tasks, for example, Figure 8 Among the six assembly processes shown, the accessory assembly task may include the first assembly process, the second assembly process, the third assembly process, the fourth assembly process and the fifth assembly process, and the accessory fastening task may include the sixth assembly process.

[0158] In the implementation of the present application, the specific task content of the current subtask is determined based on whether the assembly demonstration data of the assembly process corresponding to the current subtask obtained from the obtained assembly data includes the accessory image.

[0159] In some embodiments, when executing step 34012, refer to Fig.11 As shown, the following steps can be performed specifically:

[0160] Step 1100: If the obtained assembly demonstration data includes an accessory image, then determine that the current subtask is an accessory assembly task.

[0161] For example, Fig.12 As shown, the acquired assembly demonstration data includes an accessory image, and the accessory image includes accessories and screws; therefore, the current subtask determined is the accessory assembly task.

[0162] Step 1110: Control the right arm of the robot to obtain a first target component that matches the component in the component image from the loading system.

[0163] In some embodiments of the present application, when executing step 1110, based on the accessories in the accessory image, a first target accessory matching the accessory in the accessory image can be detected from the feeding system through a target detection method. The area for storing accessories in the feeding system can also be divided in advance, and a type of accessories can be fixedly placed in the divided area. The information of the accessories stored on site is obtained in advance by the robot through three-dimensional terrain construction. In this way, the robot can quickly locate and obtain the accessories.

[0164] In a specific implementation, the work of obtaining accessories can be obtained by the robot's right-hand auxiliary tooling or by picking up by an end effector with a magnetic suction function. Take the first target accessory as a screw as an example. First of all, the robot's end effector is usually equipped with a magnetic suction, vacuum suction or customized screw fixture, which is customized according to the material characteristics of the screw, such as using a strong magnetic suction head for metal screws to achieve stable and lossless suction. Based on the existing intelligent feeding system to transmit screws, the screws have been quickly identified and quality screened before they are transmitted to the suction position, such as the aforementioned target detection method. The accessories in the accessories library are identified, and the screws that match the accessories in the accessories image are placed on the conveyor belt of the feeding system. This ensures that the feeding system only transmits screws that meet the specifications to the robot.

[0165] After the robot identifies the required screws from the feeding system, it can control the right arm to accurately position above the screw through the control of the high-precision servo system. The actuator of the right arm slowly descends and activates the suction mechanism at the right time to capture the screw steadily.

[0166] Step 1120: Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product, and control the right arm to assemble the first target accessory onto the target product.

[0167] In some embodiments, when executing step 1120, based on the assembly process data in the obtained assembly demonstration data, such as the assembly video, assembly image, part or all of the action sequence, the open source robot arm (A Low-cost Open-source Hardware System for Bimanual Teleoperation, ALOHA) is called to control the robot's left arm to fix the target product, and control the right arm to move according to the action sequence in the assembly process data. Among them, ALOHA is a low-cost open source two-handed remote control hardware system, and its algorithm Action Chunking with Transformers (ACT) uses a neural network model Transformers, which has imitation learning capabilities; based on 15 minutes of demonstration data, the open source robot arm can learn an action, and can directly perform end-to-end imitation learning from real demonstrations.

[0168] In some embodiments of the present application, an optimal path from the grasping point of the first target accessory to the assembly position (i.e., the installation position of the first target accessory on the target product) can be calculated based on algorithms such as A* or RRT (fast random tree search) to control the right arm to assemble the first target accessory to the target product, wherein the optimal path not only takes into account the shortest path, but also comprehensively avoids obstacles in the working environment, etc., to ensure the safety and efficiency of the robot's actions.

[0169] During the movement, the robot relies on high-precision encoders and closed-loop feedback control systems (such as PID control) to achieve millimeter-level position and posture control accuracy, ensuring stable operation even under complex working conditions. In addition, since the robot moves smoothly along the preset trajectory, it also relies on high-precision encoders and closed-loop feedback control systems during the movement process, so it can ensure that the first target accessory can be accurately assembled on the target product.

[0170] In actual scenarios, the assembled parts on the target product are as follows: Fig.12 The cover plate shown may have a certain gap or a certain hole position deviation with the main product of the target product. When the left arm of the robot is controlled to press the target product, it can ensure that there is no gap between the assembled accessories and the main product, but it cannot solve the problem of misalignment between the first hole position on the assembled accessories and the second hole position on the main product. Fig.13 As shown, when installing the circuit board, there is an error in the hole position between the circuit board and the housing. At this time, if the right arm of the robot is directly controlled to assemble accessories (screws), the expected assembly effect may not be achieved.

[0171] In some embodiments, after controlling the left arm of the robot to fix the target product, the following operations need to be performed:

[0172] Operation 1: acquiring a second image of the target product.

[0173] In order to ensure the accuracy of subsequent position adjustment, a high-resolution camera is required to shoot the target product to ensure that the image is clear and contains all necessary feature information, such as hole positions, edges, shapes, etc. In some embodiments, the captured second image can also be pre-processed, including adjusting brightness, contrast, removing noise, and possible image enhancement, etc., to improve the accuracy of subsequent image analysis.

[0174] Operation 2: identifying the hole position of the first target accessory from the second image to obtain the first position of the first target accessory.

[0175] In some embodiments of the present application, when performing operation 2, a traditional image detection algorithm, such as a YOLO algorithm, may be used to obtain a first position of the first target accessory, denoted as P actual =(x actual ,y actual ).

[0176] Operation three: determining the offset based on the hole position and the first position in the obtained assembly demonstration data.

[0177] In some embodiments, if the first image contains multiple hole positions, an image matching algorithm, such as SIFT, SURF, or ORB algorithm, may be used to obtain the above-mentioned offset based on the hole positions in the obtained assembly demonstration data and the first image.

[0178] For example, suppose the hole position in the assembly demonstration data is recorded as P ideal =(x ideal ,y ideal ), and the first position of the first target accessory is denoted as P actual =(x actual ,y actual ), then the determined offset can be represented by a vector as Δ=(Δx, Δy), where Δx=x ideal -x actual , represents the distance moved along the x-axis, Δy = y ideal -y actual , represents the distance along the y-axis. A positive value means that the left arm needs to move to the right (positive direction of the x-axis) or upward (positive direction of the y-axis); a negative value means the opposite, that is, to the left (negative direction of the x-axis) or downward (negative direction of the y-axis).

[0179] In practical applications, it may be necessary to consider factors such as the robot's working range, movement speed, acceleration limit, etc., and make necessary adjustments or constraints on the offset to ensure the feasibility of motion planning.

[0180] It should be noted that the above examples only involve movement on the x-axis and y-axis, and in actual scenarios, rotation angles may also be included.

[0181] Operation four: based on the offset, control the left arm to adjust the position of the target product, and after the adjustment is completed, control the right arm to assemble the first target accessory onto the target product.

[0182] In an embodiment of the present application, when performing operation three, based on the offset, the distance that the left arm operating accessory (such as the aforementioned cover plate, etc.) needs to move along the x-axis and y-axis, and / or the angle of rotation, etc. are calculated. Then, when performing operation four, based on the offset, the left arm is controlled to adjust the position of the target product so as to align the actual hole position with the ideal assembly position, that is, the hole position in the obtained assembly demonstration data; then, after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

[0183] In some embodiments, the assembly demonstration data also includes quality inspection data. After the adjustment is completed, an image after the adjustment can be obtained and compared with the quality inspection data, such as feature matching between the image and the image in the quality inspection data, to evaluate the adjustment effect, and when the evaluation result meets the preset conditions, such as the deviation value is less than the deviation threshold, the subsequent operation of controlling the right arm to assemble is performed.

[0184] In some embodiments of the present application, in order to efficiently and safely move to the ideal assembly position, that is, the hole position in the obtained assembly demonstration data, path planning algorithms such as Dijkstra algorithm, A* algorithm, etc. can be used for path planning, especially in complex environments with obstacles. By adopting the above-mentioned path planning algorithm, obstacles can be effectively avoided on the basis of obtaining the optimal path.

[0185] Continuing with the above example, assuming that the position adjustment can be completed by controlling the robot's left arm to move in a straight line, then the offset can be used to guide the robot's left arm to move. For example, assuming that the center point of the screw hole (the first position) actually detected is P actual =(100, 200), the hole position in the obtained assembly demonstration data is P ideal =(110, 190), then we get Δx = x ideal -x actual =110-100=10, Δy=y ideal -y actual =190-200=-10, indicating that the left arm needs to move 10 units along the positive direction of the x-axis and 10 units downward along the y-axis to complete the hole alignment operation.

[0186] Specifically, when the robot performs the current subtask according to the path planning, it involves path parsing to generate instructions, motion control trajectory tracking, environmental perception obstacle avoidance and status monitoring feedback. First, the path points output by the path planning algorithm are converted into motion instructions that can be executed by the robot, and trajectory tracking is ensured through precise control. During the movement process, the robot can use sensors to continuously monitor the surrounding environment, and can immediately evaluate and adjust the path in time when encountering obstacles. Among them, the input instructions for the robot's path parsing and instruction generation come from the results of the path planning algorithm, which is a serialized set of path points, each of which contains coordinates (x, y, z) and possible rotation angles (roll, pitch, yaw). In addition, motion parameters such as speed and acceleration limits may also be included. After receiving this data, the robot's control system converts it into an instruction format that the robot can understand and execute, such as a joint space path or a Cartesian space path. The motion instruction set inside the robot is sent to the driver for execution.

[0187] The input command of motion control trajectory tracking is the specific motion command generated by the robot control system for each joint or end effector based on the parsed path command, which may include joint angle, speed and acceleration commands. Through closed-loop control (such as PID control), the robot can monitor the deviation between its position and the target path in real time, dynamically adjust the drive signal of each joint, and ensure accurate tracking of the predetermined trajectory. The output command is the current or voltage signal sent to the motor or driver, which is used to drive the robot's joints to move along the predetermined trajectory.

[0188] Environmental perception and obstacle avoidance usually require data collected from various sensors, such as lidar, cameras, ultrasonic sensors, etc., to monitor environmental conditions and potential obstacles in real time. The robot dynamically evaluates the safety of the current path based on the aforementioned data, or based on the perception data obtained by processing the aforementioned data, and calls the obstacle avoidance algorithm to replan the path when necessary. Based on the obstacle avoidance results, it can return to the path planning module to update the path, or directly issue an emergency stop or path adjustment command to the motion control system.

[0189] Continuing with the above example, after the left arm adjusts the position of the target product, the correction of the screw hole is completed and the arm is fixed so that it no longer moves, waiting for the screw fixing operation of the right hand. Specifically, the end effector of the robot's right arm is precisely aligned with the screw hole position on the target product, and the right arm is controlled to perform the nail placement operation, that is, to place the screw into the screw hole. In this process, the posture and strength of the actuator can be fine-tuned to control the right arm to place the screw smoothly and accurately at the predetermined position. In some embodiments of the present application, since the position adjustment is performed in advance, the position accuracy is guaranteed, and further ensuring synchronization can ensure that the screw can enter smoothly without damaging the workpiece.

[0190] Then, the assembly enters the torque control stage. Specifically, the robot's assembly head usually integrates an electric or pneumatic screwdriver, which has a built-in high-precision torque sensor and intelligent control system. At the moment the screw starts to be screwed in, the preset torque and speed parameters are activated. According to the screw model, material, torque requirements, etc. in the obtained assembly demonstration data, through real-time monitoring and feedback adjustment, the right arm is controlled to fasten the first target accessory to the target product to ensure that the screw is accurately tightened to the specified torque value, effectively avoiding damage to the workpiece caused by over-tightening or safety hazards caused by over-loosening, thereby achieving high efficiency and high standards of assembly.

[0191] As a specific embodiment, for the aforementioned current assembly process: installation and fastening of the rear cover, if the rear cover is not installed, then the generated subtasks may include two specific assembly operations: installing the rear cover for the product to be assembled and fastening the rear cover with screws. The obtained assembly demonstration data includes a first accessory image and a second accessory image. The first accessory image corresponds to the first sub-target accessory, i.e., the rear cover, and the second accessory image corresponds to the second sub-target accessory, i.e., the screw.

[0192] Then, after executing step 1110, control both arms or any one of the robot's mechanical arms to obtain the first sub-target accessory, i.e., the rear cover plate, which matches the accessory in the first accessory image, from the feeding system, and place the rear cover plate on the target product based on the obtained assembly demonstration data, wherein the assembly demonstration data may be an assembly video or an assembly image, and the assembly image may be a washing machine assembled with the rear cover plate. Then, control the robot's right arm to obtain (grab) the second sub-target accessory, i.e., the screw, which matches the accessory in the second accessory image, from the feeding system, and control the robot's left arm to press the rear cover plate and the target product based on the assembly video in the obtained assembly demonstration data, and control the robot's right arm to assemble the second sub-target accessory, i.e., the screw, onto the target product to fix the rear cover plate to the target product, such as Fig.12 shown.

[0193] Among them, after controlling the robot's left arm to press the rear cover and the target product, before controlling the right arm to assemble the second sub-target accessory, i.e., the screws, onto the target product, operations one to four can be performed to adjust the position of the rear cover that needs to be adjusted. For details, see the above content and will not be repeated here.

[0194] In some other embodiments, when executing step 34012, refer to Fig.14 As shown, the following steps can also be performed:

[0195] Step 1400: if the obtained assembly demonstration data does not include an accessory image, determining that the current subtask is an accessory fastening task;

[0196] Step 1410: Acquire a third image of the target product;

[0197] Step 1420: Based on the operation object in the obtained assembly demonstration data, a second target accessory corresponding to the operation object is identified in the third image;

[0198] Step 1430: Control the left arm to fix the target product, and control the right arm to perform a tightening operation on the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data, thereby completing the current subtask.

[0199] In order to ensure the accuracy of subsequent image analysis, a high-resolution camera may be used to shoot the target product to ensure that the image is clear and contains all necessary feature information, such as edges, shapes, etc. In some embodiments, the collected third image may also be pre-processed, including adjusting brightness, contrast, removing noise, and possible image enhancement, etc., to improve the accuracy of subsequent image analysis.

[0200] For example, Fig.15As shown in , assuming that the obtained assembly demonstration data does not include the accessory image, then the current subtask determined is the accessory fastening task. Assuming that the third image obtained is as follows Fig.15 As shown, based on the operation objects in the obtained assembly demonstration data, that is, Fig.15 The screw in the left image shown is identified in the third image as the second target accessory corresponding to the operation object, and then the left arm is controlled to pinch the target product around the second target accessory, and the right arm is controlled to tighten the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data. Fig.15 Only the tightening operation for one second target component is shown, and the tightening operations for other second target components are similar.

[0201] In some embodiments, after each assembly and / or tightening of the product to be assembled, a multi-dimensional quality inspection can be performed. Specifically, the assembly points can be captured by visual inspection technology, and the captured images can be analyzed based on the quality inspection data in the assembly demonstration data obtained for the product to be assembled to verify the assembly and / or tightening of the accessories, such as whether the screws are fully embedded, whether there is any tilt or omission, and the overall flatness of the assembly surface. The torque of the fixed screws can then be re-checked to ensure that the tightening degree of the screws meets the requirements.

[0202] In some specific scenarios, sealing tests or functional tests may also be implemented to fully evaluate the assembly effect. Among them, through quality inspection, assembly points that do not meet the standards will be marked and feedback will be given, thereby triggering reassembly or manual intervention, etc., to further ensure product assembly quality.

[0203] Based on the same inventive concept, refer to Fig.16 As shown, in an embodiment of the present application, a dual-arm collaborative assembly device of a robot is provided, comprising:

[0204] An acquisition module 1610 is used to acquire a first image containing a product to be assembled in the robot operation area;

[0205] A screening module 1620, configured to obtain assembly data of the product to be assembled from assembly data of multiple products based on the first image, each assembly data being input through a human-computer interaction interface and including process images and assembly demonstration data of multiple assembly processes of the product;

[0206] A process determination module 1630 is used to perform feature comparison between the first image and the process images of each assembly process in the obtained assembly data, so as to determine the current assembly process of the product to be assembled;

[0207] A task determination module 1640, configured to determine an assembly task for the product to be assembled based on the current assembly process and the various assembly processes, wherein the assembly task includes at least one subtask corresponding to the assembly process to be operated;

[0208] The assembly module 1650 is used to control the dual arms of the robot to collaboratively complete the assembly operation for the product to be assembled based on the assembly task and the assembly demonstration data in the obtained assembly data.

[0209] In a possible implementation, each assembly data also includes a main body image and identification information of the product, and the screening module 1620 is specifically used to:

[0210] performing target recognition on the first image;

[0211] If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled;

[0212] If the product body of the product to be assembled is identified, feature matching is performed on the main body images in the assembly data of the multiple products and the target images in the first image in the area where the product body is located, and identification information corresponding to the main body image matching the target image is determined as identification information of the product to be assembled;

[0213] Based on the identification information, assembly data corresponding to the identification information is obtained from assembly data of the plurality of products.

[0214] In a possible implementation, if the current assembly process is not the last assembly process among the assembly processes, the task determination module 1640 is specifically configured to:

[0215] From the assembly processes, obtaining each subsequent assembly process after the current assembly process;

[0216] According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, sequentially generate subtasks for the product to be assembled;

[0217] The generated subtasks are determined as assembly tasks for the product to be assembled.

[0218] In a possible implementation, the assembly module 1650 is specifically used to:

[0219] According to the process sequence of the assembly process corresponding to each subtask in the assembly task, the two arms of the robot are controlled to collaborate to complete the assembly operation for the product to be assembled, wherein the following operations are performed for each subtask in the assembly task:

[0220] For a current subtask in the assembly task, obtaining assembly demonstration data of an assembly process corresponding to the current subtask from the obtained assembly data;

[0221] Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product of the current subtask, and control the right arm of the robot to perform an assembly operation on the target product;

[0222] Among them, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly in the previous assembly process.

[0223] In a possible implementation, the assembly module 1650 is further used to:

[0224] If the obtained assembly demonstration data includes an accessory image, determining that the current subtask is an accessory assembly task;

[0225] Controlling the right arm of the robot to obtain a first target component that matches the component in the component image from a loading system;

[0226] Based on the obtained assembly demonstration data, the left arm of the robot is controlled to fix the target product, and the right arm is controlled to assemble the first target accessory onto the target product.

[0227] In a possible implementation, the assembly module 1650 is further used to:

[0228] Acquire a second image of the target product;

[0229] Identify the hole position of the first target accessory from the second image to obtain a first position of the first target accessory;

[0230] Determine an offset based on the hole position in the obtained assembly demonstration data and the first position;

[0231] Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

[0232] In a possible implementation, the assembly module 1650 is further used to:

[0233] If the obtained assembly demonstration data does not include an accessory image, determining that the current subtask is an accessory fastening task;

[0234] Acquire a third image of the target product;

[0235] Based on the operation object in the obtained assembly demonstration data, identifying a second target accessory corresponding to the operation object in the third image;

[0236] The left arm is controlled to fix the target product, and the right arm is controlled to perform a tightening operation on the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data, so as to complete the current subtask.

[0237] Based on the same inventive concept, an embodiment of the present application provides a robot dual-arm collaborative assembly system, comprising:

[0238] An input host, used to display a human-computer interaction interface and collect assembly data of multiple products input by a user through the human-computer interaction interface, each assembly data including process images and assembly demonstration data of multiple assembly processes of the product;

[0239] A feeding system, used for providing the robot with accessories to be assembled;

[0240] The robot, see Fig.17 As shown, including:

[0241] Memory 171, used to store computer programs or instructions;

[0242] The processor 172 is used to execute the computer program or instructions in the memory 171 so that any one of the methods in the above embodiments is executed.

[0243] The processor 172 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 171, so that any one of the methods in the above embodiments is executed;

[0244] It should be noted that the specific connection medium between the memory 171 and the processor 172 is not limited in the embodiment of the present application. Fig.17 In the embodiment, the memory 171 and the processor 172 are connected via a bus 173, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 173 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0245] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments. Since the principle of solving the problem by the above computer-readable storage medium is similar to that of a robot's dual-arm collaborative assembly method, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0246] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0247] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0248] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0249] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0250] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A robot dual-arm collaborative assembly method, characterized in that: include: Acquire a first image containing a product to be assembled in the robot operation area; Based on the first image, obtaining assembly data of the product to be assembled from assembly data of multiple products, each assembly data is input through a human-computer interaction interface, including process images and assembly demonstration data of multiple assembly processes of the product; Performing feature comparison between the first image and the process images of each assembly process in the obtained assembly data respectively, to determine the current assembly process of the product to be assembled; Based on the current assembly process and the various assembly processes, determining an assembly task for the product to be assembled, wherein the assembly task includes at least one subtask corresponding to the assembly process to be operated; Based on the assembly task and the assembly demonstration data in the obtained assembly data, the dual arms of the robot are controlled to collaboratively complete the assembly operation on the product to be assembled.

2. The method according to claim 1, characterized in that Each assembly data also includes a main body image and identification information of a product, and then obtaining the assembly data of the product to be assembled from the assembly data of multiple products based on the first image includes: performing target recognition on the first image; If the icon of the product serial number of the product to be assembled is identified, the icon is scanned and identified to obtain the identification information of the product to be assembled; If the product body of the product to be assembled is identified, feature matching is performed on the main body images in the assembly data of the multiple products and the target images in the first image in the area where the product body is located, and identification information corresponding to the main body image matching the target image is determined as identification information of the product to be assembled; Based on the identification information, assembly data corresponding to the identification information is obtained from assembly data of the plurality of products.

3. The method according to claim 1 or 2, characterized in that If the current assembly process is not the last assembly process among the assembly processes, determining the assembly task of the product to be assembled based on the current assembly process and the assembly processes includes: From the assembly processes, obtaining each subsequent assembly process after the current assembly process; According to the process sequence of each assembly process, based on the current assembly process and each subsequent assembly process, sequentially generate subtasks for the product to be assembled; The generated subtasks are determined as assembly tasks for the product to be assembled.

4. The method according to claim 3, characterized in that The step of controlling the dual arms of the robot to collaboratively complete the assembly operation for the product to be assembled based on the assembly task and the assembly demonstration data in the obtained assembly data includes: According to the process sequence of the assembly process corresponding to each subtask in the assembly task, the two arms of the robot are controlled to collaborate to complete the assembly operation for the product to be assembled, wherein the following operations are performed for each subtask in the assembly task: For a current subtask in the assembly task, obtaining assembly demonstration data of an assembly process corresponding to the current subtask from the obtained assembly data; Based on the obtained assembly demonstration data, control the left arm of the robot to fix the target product of the current subtask, and control the right arm of the robot to perform an assembly operation on the target product; Among them, if the current subtask is the first task in the assembly task, the target product is the product to be assembled; if the current subtask is not the first task in the assembly task, the target product is the intermediate product obtained after assembly in the previous assembly process.

5. The method according to claim 4, characterized in that The method further comprises: If the obtained assembly demonstration data includes an accessory image, determining that the current subtask is an accessory assembly task; Controlling the right arm of the robot to obtain a first target component that matches the component in the component image from a loading system; Based on the obtained assembly demonstration data, the left arm of the robot is controlled to fix the target product, and the right arm is controlled to assemble the first target accessory onto the target product.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a second image of the target product; Identify the hole position of the first target accessory from the second image to obtain a first position of the first target accessory; Determine an offset based on the hole position in the obtained assembly demonstration data and the first position; Based on the offset, the left arm is controlled to adjust the position of the target product, and after the adjustment is completed, the right arm is controlled to assemble the first target accessory onto the target product.

7. The method according to claim 4, characterized in that The method further comprises: If the obtained assembly demonstration data does not include an accessory image, determining that the current subtask is an accessory fastening task; Acquire a third image of the target product; Based on the operation object in the obtained assembly demonstration data, identifying a second target accessory corresponding to the operation object in the third image; The left arm is controlled to fix the target product, and the right arm is controlled to perform a tightening operation on the second target accessory according to the torque requirement associated with the operation object in the obtained assembly demonstration data, so as to complete the current subtask.

8. A robot dual-arm collaborative assembly device, characterized in that: include: An acquisition module, used for acquiring a first image containing a product to be assembled in the robot operation area; A screening module, configured to obtain assembly data of the product to be assembled from assembly data of multiple products based on the first image, each assembly data being input through a human-computer interaction interface and including process images and assembly demonstration data of multiple assembly processes of the product; A process determination module, used for comparing the first image with the process images of each assembly process in the obtained assembly data, to determine the current assembly process of the product to be assembled; A task determination module, used to determine the assembly task of the product to be assembled based on the current assembly process and the various assembly processes, wherein the assembly task includes at least one subtask corresponding to the assembly process to be operated; The assembly module is used to control the two arms of the robot to collaboratively complete the assembly operation for the product to be assembled based on the assembly task and the assembly demonstration data in the obtained assembly data.

9. A robot dual-arm collaborative assembly system, characterized in that: include: An input host, used to display a human-computer interaction interface and collect assembly data of multiple products input by a user through the human-computer interaction interface, each assembly data including process images and assembly demonstration data of multiple assembly processes of the product; A feeding system, used for providing the robot with accessories to be assembled; The robot comprises a memory for storing computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 7.

Images