A 3D vision-guided follow-up assembly system for automobile doors

Abstract

The present application relates to a 3D vision-guided door follow-up assembly system. In this system, when the first robotic arm grabs the door to be assembled and moves it to the teaching position for secondary positioning and deviation correction, the first camera acquires the grayscale image and depth image of the door; when the second robotic arm and the line body plate chain are in the follow-up process, the second camera acquires the grayscale image and depth image of the vehicle body; the processor constructs the hinge hole pose matrix based on the grayscale image and depth image of the vehicle body and thereby determines the real-time installation pose of the hinge hole; constructs the thread hole pose matrix based on the grayscale image and depth image of the door, and determines the real-time installation pose of the door based on the thread hole pose matrix and the hinge hole pose matrix; the first robotic arm aligns the door to be assembled with the vehicle body to be assembled based on the real-time installation pose of the door; the second robotic arm tightens the bolts based on the real-time installation pose of the hinge hole, so that the door to be assembled is installed on the vehicle body to be assembled. Using this method can improve the assembly efficiency between the door and the vehicle body.

Description

Technical Field

[0001] This application relates to the field of intelligent technologies for automobile body production and manufacturing, and particularly to a 3D vision-guided follow-up assembly system for automobile doors. Background Art

[0002] With the rapid development of artificial intelligence, robots are applied in different fields to perform different tasks. Especially in modern automobile assembly, during the assembly process between the door and the body, assembly robots are widely used. Currently, the main process for door assembly in vehicle manufacturers is to use mechanical tools to assist workers in aligning and tightening bolts. That is, even with the assistance of automated equipment, manual operation is still required. And manual door assembly requires multiple workers to form a group, and the assembly process takes a long time, resulting in low efficiency. Therefore, how to improve the efficiency of door assembly is an urgent problem to be solved. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a 3D vision-guided follow-up assembly system for automobile doors that can improve the assembly efficiency between the door and the body.

[0004] In a first aspect, this application provides a 3D vision-guided follow-up assembly system for automobile doors, characterized in that the follow-up assembly system for automobile doors at least includes a linear plate chain, a follow-up mechanism, a first camera fixed to the ground, a first robotic arm, a second robotic arm, a second camera fixed to the end of the second robotic arm, and a processor;

[0005] The first camera is used to obtain a door grayscale image and a door depth image of the door to be assembled when the first robotic arm grabs the door to be assembled to the teaching position for secondary positioning and deviation correction; both the door grayscale image and the door depth image include the threaded holes on the door to be assembled;

[0006] The second camera is used to obtain a body grayscale image and a body depth image of the body to be assembled when the second robotic arm is in a follow-up process with the linear plate chain; both the body grayscale image and the body depth image include the hinge holes on the body to be assembled;

[0007] The processor is used to construct a hinge hole pose matrix that matches the hinge holes based on the body grayscale image and the body depth image, and determine the real-time installation pose of the hinge holes in the real-time assembly situation based on the hinge hole pose matrix; construct a threaded hole pose matrix that matches the threaded holes based on the door grayscale image and the door depth image, and determine the real-time installation pose of the door in the real-time assembly situation based on the threaded hole pose matrix and the hinge hole pose matrix;

[0008] The first robotic arm is used to align the door to be assembled with the body to be assembled based on the real-time installation pose of the door;

[0009] The second robotic arm is used to tighten the bolts based on the real-time installation pose of the hinge holes, so as to install the door to be assembled onto the vehicle body to be assembled.

[0010] The line body plate chain is used to carry the vehicle body to be assembled forward;

[0011] The follow-up mechanism is used to keep the first robotic arm, the second robotic arm and the line body plate chain moving synchronously.

[0012] In one embodiment, the second camera is specifically used to move to a specified position after the second robotic arm adsorbs the bolt, and when the second robotic arm and the line body plate chain are in the follow-up process, obtain the body grayscale image and the body depth image of the vehicle body to be assembled;

[0013] The first robotic arm is specifically used to move the door to be assembled from a preset transition position to an assembly position, and adjust the door to be assembled through the real-time installation pose of the door, so that the threaded hole is aligned with the hinge hole;

[0014] The second robotic arm is specifically used to sleeve the adsorbed bolt into the hinge hole based on the real-time installation pose of the hinge hole; after the threaded hole is aligned with the hinge hole, tighten the bolt to install the door to be assembled onto the vehicle body to be assembled.

[0015] In one embodiment, the processor is specifically used to: determine the fusion pose of the hinge hole template in the teaching assembly case and the real-time fusion pose of the hinge hole in the real-time assembly case; determine the real-time installation pose of the hinge hole in the real-time assembly case through the fusion pose of the hinge hole template, the real-time fusion pose of the hinge hole, and the installation pose of the hinge hole template in the teaching assembly case.

[0016] In one embodiment, the processor is specifically used to: determine the fusion pose of the threaded hole template in the teaching assembly case and the real-time fusion pose of the threaded hole in the real-time assembly case; determine the real-time installation pose of the door in the real-time assembly case based on the fusion pose of the hinge hole template and the real-time fusion pose of the hinge hole, the fusion pose of the threaded hole template and the real-time fusion pose of the threaded hole, and the installation pose of the door template in the teaching assembly case.

[0017] In one embodiment, the threaded hole on the door to be assembled and the hinge hole on the vehicle body to be assembled have an assembly corresponding relationship;

[0018] A processor, specifically configured to: determine a set of thread hole pose matrices and a set of body hinge hole pose matrices based on the assembly correspondence between each thread hole and each body hinge hole, where the number of thread hole pose matrices in the set of thread hole pose matrices is the same as the number of body hinge hole pose matrices in the set of body hinge hole pose matrices, and there is an assembly correspondence between the thread hole that matches the thread hole pose matrix and the body hinge hole that matches the body hinge hole pose matrix; perform pose fusion on the thread hole pose matrices in the set of thread hole pose matrices to obtain a real-time fused pose of the thread holes; perform pose fusion on the body hinge hole pose matrices in the set of body hinge hole pose matrices to obtain a real-time fused pose of the body hinge holes.

[0019] In one embodiment, the processor is specifically configured to: perform ellipse recognition on the door grayscale image to obtain a plurality of thread hole features; perform point cloud restoration for each thread hole feature based on the door depth map to obtain the target thread hole neighborhood point cloud corresponding to each thread hole feature; determine the corresponding thread hole plane point cloud and the thread hole inner contour point cloud based on each target thread hole neighborhood point cloud; determine the corresponding thread hole plane normal based on the thread hole plane point cloud, and determine the corresponding thread hole center based on the thread hole inner contour point cloud.

[0020] In one embodiment, the processor is specifically configured to perform grayscale neighborhood extraction for each thread hole feature to obtain the region of interest of the thread hole corresponding to each thread hole feature; perform point cloud data restoration on each region of interest of the thread hole based on the door depth map to obtain the first thread hole plane point cloud corresponding to each region of interest of the thread hole; remove the invalid values in each first thread hole plane point cloud to obtain the second thread hole plane point cloud; perform statistical filtering on each second thread hole plane point cloud to remove the outliers in the second thread hole plane point cloud to obtain the third thread hole plane point cloud; calculate the flatness of each third thread hole plane point cloud, and determine the third thread hole plane point cloud whose flatness meets the flatness threshold as the target thread hole neighborhood point cloud.

[0021] In one embodiment, the processor is specifically configured to: project each target thread hole neighborhood point cloud onto the thread hole parameter plane to obtain the target thread hole plane projection point cloud corresponding to each target thread hole neighborhood point cloud; extract the corresponding thread hole edge point cloud from each target thread hole plane projection point cloud based on the angle distribution, where the thread hole edge point cloud includes the thread hole inner edge point cloud and the thread hole outer edge point cloud; extract the thread hole inner edge point cloud from each thread hole edge point cloud, and perform circle fitting based on each thread hole inner edge point cloud to obtain the thread hole center corresponding to each thread hole.

[0022] In one embodiment, the processor is specifically configured to: perform clustering processing on each third threaded hole plane point cloud by using a region growing clustering method, and determine, from the third threaded hole plane point cloud, a threaded hole plane point cloud cluster including the largest number of point clouds; and use the random sample consensus algorithm to perform plane fitting on the threaded hole plane point cloud cluster to obtain a threaded hole parameter plane.

[0023] In one embodiment, the processor is specifically configured to: construct a threaded hole pose matrix for each threaded hole based on the normal direction of each threaded hole plane and the center of the threaded hole; wherein, the threaded hole pose matrix matches the threaded holes in the door to be assembled.

[0024] In the above 3D vision-guided follow-up assembly system for automotive doors, the body to be assembled is carried forward by the line body plate chain, and the synchronous movement between the first robotic arm, the second robotic arm and the line body plate chain is ensured by the follow-up mechanism. Therefore, when the first robotic arm and the second robotic arm are in the follow-up process, a hinge hole pose matrix is constructed through the acquired body grayscale image and body depth image, and the real-time installation pose of the hinge hole in the real-time assembly situation is determined based on the hinge hole pose matrix. Based on this, when the first robotic arm grabs the door to be assembled to the teaching position for secondary positioning and deviation correction, a threaded hole pose matrix is constructed through the acquired door grayscale image and door depth image. Thus, through the threaded hole pose matrix and the hinge hole pose matrix, the real-time installation pose of the door in the real-time assembly situation can be determined, that is, there is no need for manual participation in the assembly process of the door and the body. Only image acquisition is required at the required positions and during the follow-up process to complete the determination of the pose matrix. Thereby, the first robotic arm moves the door to be assembled from the preset transition position to the assembly position through the real-time installation pose of the door, and the second robotic arm tightens the bolts based on the real-time installation pose of the hinge hole, so that the door to be assembled is installed on the body to be assembled. That is, by automatically calibrating and determining the poses of the first robotic arm and the second robotic arm, the door to be assembled is installed on the body to be assembled, and since the first robotic arm and the second robotic arm are in follow-up, that is, the automatic assembly process does not need to stop the production line. On the basis of avoiding manual participation in the assembly, the assembly efficiency between the door and the body can be improved. Description of the Drawings

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0026] Figure 1 It is a schematic diagram of the assembly of the door and the body in one embodiment;

[0027] Figure 2 Schematic diagram of the assembly process of a vehicle door and a vehicle body in an embodiment;

[0028] Figure 3 Schematic diagram of the process of assembling a vehicle door and a vehicle body in a 3D vision-guided vehicle door follow-up assembly system in an embodiment;

[0029] Figure 4 Schematic diagram of the process of assembling a vehicle door and a vehicle body in a 3D vision-guided vehicle door follow-up assembly system in another embodiment;

[0030] Figure 5 Schematic diagram of the process of determining the pose matrix of a threaded hole in an embodiment;

[0031] Figure 6 Schematic diagram of the overall process of ellipse recognition, circle fitting, and pose calculation in an embodiment;

[0032] Figure 7 Schematic diagram of the timing process of the visual assembly step and the mechanical assembly step in a system in an embodiment;

[0033] Figure 8 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0034] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] With the rapid development of artificial intelligence, robots are applied in different fields to perform different tasks. Especially in modern vehicle assembly, during the process of assembling a vehicle door and a vehicle body, assembly robots are widely used. Currently, vehicle door assembly in vehicle manufacturers mainly relies on mechanical tools to assist manual alignment, tightening bolts, etc. That is, even with the assistance of automated equipment, manual actual operation is still required. And manual assembly of vehicle doors requires multiple people to form a group, and the assembly process takes a long time, resulting in low efficiency. Therefore, how to improve the efficiency of vehicle door assembly is an urgent problem to be solved.

[0036] Based on this, the present application provides a 3D vision-guided automotive door follow-up assembly system for improving the assembly efficiency between the door and the vehicle body. The 3D vision-guided automotive door follow-up assembly system provided by the embodiments of the present application at least includes a line body plate chain, a follow-up mechanism, a first camera fixed to the ground, a first robotic arm, a second robotic arm, a second camera fixed to the end of the second robotic arm, and a processor. It can be understood that since the present application needs to perform follow-up processing on the first robotic arm and the second robotic arm, the vehicle body to be assembled is carried forward by the line body plate chain. To ensure the follow-up between the first robotic arm and the second robotic arm, the follow-up mechanism is used to ensure that the first robotic arm, the second robotic arm, and the line body plate chain move synchronously.

[0037] Among them, the first camera fixed to the ground is a three-dimensional (3D) camera, and the number of the first cameras fixed to the ground is at least two. The aforementioned first camera is used to capture data, and the first camera can capture and collect 2D grayscale images or 3D depth images. The first camera and the first robotic arm form an eye-to-hand system, that is, the first camera is used to collect the data of the door thread holes on the door to be assembled after the first robotic arm grabs the door to be assembled. Similarly, the second camera fixed to the end of the second robotic arm is also a 3D camera, and the number of the second cameras is at least two. The second camera and the second robotic arm form an eye-on-hand system, that is, the second camera is used to collect the data of the body hinge holes on the vehicle body to be assembled.

[0038] Secondly, the first robotic arm is specifically a loading robotic arm. At this time, a door clamping mechanism is installed at the end of the first robotic arm. That is, the first robotic arm grabs the door to be assembled through the door clamping mechanism. Then, the door clamping mechanism clamps the door to be assembled and presents the door thread holes on the door to be assembled to the first camera for image capture and collection in a suitable pose. That is, at this time, the first camera can thus collect grayscale images and depth images of the door thread holes on the assembled door. In addition, the first robotic arm can also move the door to be assembled from the preset transition position to the assembly position according to the real-time installation pose of the door under the real-time assembly condition calculated by the control method of the processor introduced in the embodiments of the present application, and adjust the door to be assembled through the real-time installation pose of the door to align the thread holes with the hinge holes.

[0039] Similarly, the second robotic arm is specifically a tightening robotic arm. At this time, a hinge gun mechanism is installed at the upper end of the second robotic arm. That is, the second robotic arm tightens the adsorbed bolt through the hinge gun mechanism. That is, the hinge gun mechanism adsorbs the bolt and, based on the real-time installation pose of the hinge hole in the real-time assembly situation calculated by the control method of the processor introduced in the embodiments of the present application, moves the adsorbed bolt to the position of the hinge gun sleeve hole, so that the bolt enters the body hinge hole, thereby installing the door to be assembled on the body to be assembled. Therefore, the processor is used to execute the control method shown in the embodiments of the present application to solve the real-time installation pose of the hinge hole and the real-time installation pose of the door.

[0040] And the line body plate chain is used to slide on the follow-up mechanism. That is, the body to be assembled can be placed on the line body plate chain to ensure that when the body to be assembled runs on the follow-up mechanism, the running speed of the line body plate chain is controlled to ensure that the body to be assembled advances to the assembly position at a constant speed. That is, the line body plate chain is used to carry the body to be assembled forward.

[0041] The follow-up mechanism is used to ensure that the first robotic arm and the second robotic arm can follow at the same speed as the line body plate chain during the assembly process. That is, it is ensured that the first robotic arm, the second robotic arm, and the body to be assembled are all in a relatively static state, so that all assembly tasks can be completed during the follow-up without stopping the production line from advancing. Then both the first robotic arm and the second robotic arm can slide on the follow-up mechanism to ensure that the follow-up mechanism cooperates to complete the door assembly. Therefore, the follow-up mechanism is used to keep the first robotic arm, the second robotic arm, and the line body plate chain moving in synchronization.

[0042] To facilitate understanding of the process of assembling the door and the body in the embodiments of the present application, as Figure 1 shown in the assembly schematic diagram of the door and the body, the first robotic arm grabs the door 1 through the door clamping mechanism. There are multiple threaded holes 2 on the door 1. In the embodiments of the present application, the number of threaded holes 2 is 4. And there are body hinges 3 on the body, and there are multiple hinge holes 4 on the hinges 3. In the embodiments of the present application, the number of hinge holes 4 is 4. That is, the number of threaded holes 2 is the same as the number of hinge holes 4, and each threaded hole 2 has a matching hinge hole 4. For example, the threaded holes 2 specifically include threaded hole 21, threaded hole 22, threaded hole 23, and threaded hole 24, and the hinge holes 4 specifically include hinge hole 41, hinge hole 42, hinge hole 43, and hinge hole 44. At this time, threaded hole 21 matches hinge hole 41, threaded hole 22 matches hinge hole 42, threaded hole 23 matches hinge hole 43, and threaded hole 24 matches hinge hole 44.

[0043] Therefore, the second robotic arm needs to adsorb the bolt 5, pass the adsorbed bolt 5 through the hinge hole 4, and then screw it into the threaded hole 2 that matches the hinge hole 4. As can be seen from the foregoing example, the second robotic arm needs to adsorb the bolt 5, pass the adsorbed bolt 5 through the hinge hole 41, and then screw the hinge hole 41 into the threaded hole 21 that matches it. Similarly, after passing the adsorbed bolt 5 through the hinge hole 42, screw the hinge hole 42 into the threaded hole 22 that matches it. After passing the adsorbed bolt 5 through the hinge hole 43, screw the hinge hole 43 into the threaded hole 23 that matches it. And after passing the adsorbed bolt 5 through the hinge hole 44, screw the hinge hole 44 into the threaded hole 24 that matches it. Therefore, through the foregoing operations, the planes where the hinge 3 and the threaded holes 2 on the car door 1 are located can be made to fit, completing the follow-up assembly of the car door and the vehicle body.

[0044] For ease of understanding, the assembly process of the car door and the vehicle body in the 3D vision-guided follow-up assembly system of the car door is as Figure 2 shown in the schematic diagram of the assembly process of the car door and the vehicle body

[0045] Step 201: Grasp the door to be assembled. That is, after the door to be assembled is in place, first collect the threaded holes on the door to be assembled through the first camera fixed on the ground to obtain the threaded hole image. Then guide the first robotic arm to grasp the door to be assembled.

[0046] Step 202: Perform secondary alignment correction on the door to be assembled. The first robotic arm grasps the door to be assembled to the teaching position for secondary positioning and alignment correction. At this time, obtain the door grayscale image and the door depth image of the door to be assembled through the first camera. At this time, both the door grayscale image and the door depth image include the threaded holes on the door to be assembled. At this time, perform positioning and alignment correction based on the door grayscale image and the teaching position of the door depth image. After completing the secondary positioning and alignment correction at the teaching position, the first robotic arm grasps the door to be assembled and moves it to the preset transition position. The method of performing secondary positioning and alignment correction at the teaching position is the step of constructing a threaded hole pose matrix that matches the threaded holes based on the door grayscale image and the door depth image by the processor in the embodiments of the present application. Subsequent embodiments will introduce this in detail.

[0047] Step 203: Follow-up of the body to be assembled. The second robotic arm adsorbs the bolt through the hinge gun mechanism and moves to the designated position, and then the hinge gun mechanism connects the base slide rail of the first robotic arm, the base slide rail of the second robotic arm, and the slide rail of the line body chain. At this time, the line body chain carries the body to be assembled forward, and the line body chain, the body to be assembled, the first robotic arm, and the second robotic arm are in a relatively static state, that is, at this time, the second robotic arm is in the follow-up process with the line body chain. Specifically, a follow-up mechanism is used to ensure that the first robotic arm, the second robotic arm, and the line body chain move synchronously.

[0048] Step 204: Hinge positioning of the body to be assembled. When the second robotic arm and the in-line pallet chain are in the follow-up process as introduced in Step 203, the body gray-scale image and the body depth image of the body to be assembled are obtained through the second camera. At this time, the step of constructing a hinge hole pose matrix matching the hinge holes based on the body gray-scale image and the body depth image by the processor in the embodiment of the present application is executed. At this time, the real-time installation pose of the hinge holes in the real-time assembly situation is determined based on the hinge hole pose matrix, so as to complete the body positioning and guide the second robotic arm to perform body positioning based on the real-time installation pose of the hinge holes.

[0049] Step 205: Centering and tightening of the door to be assembled. Since the real-time installation pose of the hinge holes and the thread hole pose matrix can be determined through the foregoing steps, the processor can calculate the real-time installation pose of the door in the real-time assembly situation based on the thread hole pose matrix and the hinge hole pose matrix, so as to guide the first robotic arm to move the door to be assembled from the preset transition position to the assembly position through the real-time installation pose of the door, so that the thread holes on the door to be assembled are centered with the hinge holes on the body to be assembled. Based on this, the second robotic arm screws the adsorbed bolts into the body hinge holes through the hinge gun mechanism to tighten the bolts, so as to install the door to be assembled on the body to be assembled.

[0050] The foregoing process will be introduced in detail below. In an exemplary embodiment, as Figure 3 shown, a 3D vision-guided automotive door follow-up assembly system is provided. The 3D vision-guided automotive door follow-up assembly system at least includes an in-line pallet chain, a follow-up mechanism, a first camera fixed to the ground, a first robotic arm, a second robotic arm, a second camera fixed to the end of the second robotic arm, and a processor. Among them:

[0051] Step 302: When the first camera grabs the door to be assembled to the teaching position for secondary positioning and deviation correction, the door gray-scale image and the door depth image of the door to be assembled are obtained; both the door gray-scale image and the door depth image include the thread holes on the door to be assembled.

[0052] Among them, since there are multiple thread holes on the door to be assembled, at least one thread hole on the door to be assembled is included in the door gray-scale image and the door depth image. As can be seen from the assembly schematic diagram of the door and the body shown above, there are multiple thread holes 2 on the door 1. In the embodiment of the present application, the number of the thread holes 2 is 4, and what is included in the door gray-scale image and the door depth image is the thread holes 2 existing on the door 1. Secondly, the first camera is a 3D camera, and the number of the first cameras is at least two. Therefore, the first camera can capture and collect 2D gray-scale images, and can also capture and collect 3D depth images, and the first camera and the first robotic arm form an eye-in-hand system. Figure 1

[0053] ​Specifically, after the door to be assembled arrives at the work position, the door clamping mechanism installed at the end of the first robotic arm is guided to grasp the door to be assembled, and the first robotic arm grasps the door to be assembled to the teaching position. Therefore, when the first robotic arm grasps the door to be assembled to the teaching position for secondary positioning and deviation correction, the first camera captures the plane of the threaded holes on the door to be assembled to obtain a grayscale image and a depth image of the door including at least one threaded hole.

[0054] Step 304: When the second robotic arm and the line body plate chain are in a follow-up process, the second camera obtains a grayscale image and a depth image of the vehicle body to be assembled; both the grayscale image and the depth image of the vehicle body include hinge holes on the vehicle body to be assembled.

[0055] Among them, the line body plate chain is used to carry the vehicle body to be assembled forward; the follow-up mechanism is used to keep the first robotic arm, the second robotic arm and the line body plate chain moving synchronously. Secondly, since there are multiple hinge holes on the door to be assembled, at least one hinge hole on the door to be assembled is included in the grayscale image and the depth image of the vehicle body. The aforementioned threaded hole is specifically a vehicle body hinge hole, as described above. Figure 1 As can be seen from the assembly schematic diagram of the door and the vehicle body shown above, there is a vehicle body hinge 3 on the vehicle body, and there are multiple hinge holes 4 on the hinge 3. In the embodiment of the present application, the multiple hinge holes 4 existing on the vehicle body hinge 3 are collected in the grayscale image and the depth image of the vehicle body. Secondly, the second camera is also a 3D camera, and the number of the second cameras is at least two. The second cameras and the second robotic arm form an eye-in-hand system.

[0056] Specifically, as introduced above, the vehicle body to be assembled is placed on the line body plate chain, and the line body plate chain is used to carry the vehicle body to be assembled forward to ensure that when the vehicle body to be assembled runs on the follow-up mechanism, the running speed of the line body plate chain is controlled to ensure that the vehicle body to be assembled advances to the assembly position at a constant speed. Therefore, specifically, the follow-up mechanism is used to ensure that the first robotic arm, the second robotic arm and the line body plate chain move synchronously, and can follow the line body plate chain at the same speed during the assembly process, that is, to ensure that the first robotic arm, the second robotic arm and the vehicle body to be assembled are all in a relatively static state. Based on this, when the second robotic arm and the line body plate chain are in a follow-up process, that is, when the vehicle body to be assembled runs on the follow-up mechanism during the assembly process, at this time, the follow-up mechanism is used to ensure that the first robotic arm, the second robotic arm and the line body plate chain move synchronously, then the first robotic arm, the second robotic arm and the vehicle body to be assembled are all in a relatively static state. At this time, the second camera is used to collect the data of the vehicle body hinge holes on the vehicle body to be assembled, that is, the second camera captures the vehicle body hinge holes on the vehicle body to be assembled to obtain a grayscale image and a depth image of the vehicle body including at least one hinge hole.

[0057] Step 306: The processor constructs a hinge hole pose matrix that matches the hinge holes based on the body grayscale image and the body depth image, and determines the real-time installation pose of the hinge holes in the real-time assembly scenario based on the hinge hole pose matrix; constructs a threaded hole pose matrix that matches the threaded holes based on the door grayscale image and the door depth image, and determines the real-time installation pose of the door in the real-time assembly scenario based on the threaded hole pose matrix and the hinge hole pose matrix.

[0058] Among them, the hinge hole pose matrix does not have the property of orthogonality, that is, there is no fixed direction vector in the hinge hole pose matrix. Therefore, the obtained hinge hole pose matrix is a simple description only used to represent the pose of the hinge holes. Therefore, precisely because the hinge hole pose matrix matched by the hinge holes does not have the property of orthogonality, that is, the obtained real-time installation pose of the hinge holes is obtained based on the fused pose of the hinge hole templates, and the fused pose of the hinge hole templates is obtained by performing pose fusion on the hinge hole pose matrices matched by at least two hinge holes. Similarly, the threaded hole pose matrix does not have the property of orthogonality, that is, there is no fixed direction vector in the threaded hole pose matrix. Therefore, the obtained threaded hole pose matrix is a simple description only used to represent the pose of the threaded holes. Therefore, precisely because the threaded hole pose matrix matched by the threaded holes does not have the property of orthogonality, that is, the obtained real-time fused pose of the threaded holes is obtained by performing pose fusion on the threaded hole pose matrices matched by at least two threaded holes.

[0059] Specifically, after the second camera captures the body hinge holes of the body to be assembled to obtain the body grayscale image and the body depth image including at least one hinge hole, the second camera will transmit the obtained body grayscale image and body depth image to the processor. At this time, the processor can construct a hinge hole pose matrix that matches the hinge holes based on the body grayscale image and the body depth image, that is, the processor will perform ellipse recognition on the body grayscale image to obtain the hinge hole features corresponding to the hinge holes included in each body grayscale image, and then perform point cloud restoration on the hinge hole features corresponding to each hinge hole based on the body depth image, so as to obtain the hinge hole point clouds corresponding to each hinge hole feature respectively, and then construct the hinge hole pose matrix corresponding to each hinge hole based on the hinge hole point clouds. Since the obtained hinge hole pose matrix does not have the property of orthogonality, it is necessary to perform pose fusion on the hinge hole pose matrices matched by the hinge holes of the body hinges belonging to the same body to be assembled to obtain the real-time fused pose of the hinge holes in the real-time assembly scenario, and then determine the real-time installation pose of the hinge holes in the real-time assembly scenario through the obtained real-time fused pose of the hinge holes.

[0060] Similarly, after the first camera captures images of the door thread holes of the door to be assembled to obtain a door grayscale image and a door depth image including at least one thread hole, the first camera will transmit the obtained door grayscale image and door depth image to the processor. At this time, the processor can construct a thread hole pose matrix that matches the thread hole based on the door grayscale image and the door depth image. That is, the first camera will perform ellipse recognition on the door grayscale image to obtain the thread hole features corresponding to the thread holes included in each door grayscale image, and then perform point cloud restoration on the thread hole features corresponding to each thread hole based on the door depth image, so as to obtain the thread hole plane point clouds corresponding to each thread hole feature respectively. Furthermore, a thread hole pose matrix corresponding to each thread hole is constructed based on the thread hole plane point clouds. Since the obtained thread hole pose matrix does not have the property of orthogonality, it is necessary to perform pose fusion based on the thread hole pose matrices that match the thread holes belonging to the same door to be assembled to obtain the real-time fusion pose of the thread holes in the real-time assembly situation.

[0061] Considering that in practical applications, for the body incoming materials and door incoming materials of the same batch, the positions where the body incoming materials and door incoming materials are placed on the production line will not be exactly the same. That is, when there is a deviation in the production line placement positions of the body incoming materials and door incoming materials, if the pose matrix, fusion pose, and final assembly pose are calculated every time, it will increase the duration of pose calculation and assembly. Therefore, visual template making can be carried out, that is, the actual assembly between the body and the door is not completed, and an assembly process identical to the actual situation is simulated. At this time, the image data collected by the first camera and the second camera are obtained, and the teaching poses of the first robotic arm and the second robotic arm at the teaching positions and during the follow-up process and other positions are determined. Then, the teaching poses corresponding to the first robotic arm and the second robotic arm are used as the template poses in the teaching assembly situation. The aforementioned template poses may include, but are not limited to, the hinge hole template fusion pose, the thread hole template fusion pose, the hinge hole template installation pose, and the door template installation pose. Therefore, in practical applications, by combining the image data collected in the visual template making and the template poses in the teaching assembly situation, calculations are performed on the actually collected images and the determined poses to accurately obtain the real-time installation pose of the door and the real-time installation pose of the hinge hole in the real-time assembly situation and improve the efficiency of determining the installation pose. The following will introduce in detail how to determine the real-time installation pose of the hinge hole in the real-time assembly situation and how to determine the real-time installation pose of the door in the real-time assembly situation:

[0062] First, introduce the method for determining the real-time installation pose of the hinge hole in the case of real-time assembly: In a specific embodiment, the processor determines the real-time installation pose of the hinge hole in the case of real-time assembly based on the hinge hole pose matrix, including: the processor determines the fused pose of the hinge hole template in the case of teaching assembly and the real-time fused pose of the hinge hole in the case of real-time assembly; the real-time installation pose of the hinge hole in the case of real-time assembly is determined by the fused pose of the hinge hole template, the real-time fused pose of the hinge hole, and the installation pose of the hinge hole template in the case of teaching assembly.

[0063] Among them, the real-time fused pose of the hinge hole is determined based on the hinge hole pose matrix, and the fused pose of the hinge hole template is the installation pose of the hinge hole obtained by running the same assembly process as the actual situation in the template wheel. That is, the precise installation pose is taught by manually operating the second robotic arm at the assembly position, and then the second camera captures and collects the hinge holes of the vehicle body during the manual teaching operation to obtain the template vehicle body grayscale image and the template vehicle body depth image, and based on the template vehicle body grayscale image and the template vehicle body depth image, the fused pose of the hinge hole template in the case of teaching assembly is determined. At this time, manual hole matching is performed for the door bolt hole and the vehicle body hinge hole, and the installation pose of the second robotic arm in the case of teaching assembly when recording the installation pose taught manually is recorded. The installation pose of the second robotic arm in the case of teaching assembly is the installation pose of the hinge hole template in the case of teaching assembly. Through the above method, the fused pose of the hinge hole template in the case of teaching assembly and the installation pose of the hinge hole template in the case of teaching assembly can be obtained. At this time, the fused pose of the hinge hole template in the case of teaching assembly and the installation pose of the hinge hole template in the case of teaching assembly are bound and stored in terms of data.

[0064] Specifically, in the actual assembly case, the processor first determines the fused pose of the hinge hole template in the case of teaching assembly, that is, the processor can extract the fused pose of the hinge hole template in the case of teaching assembly from the data stored by running the assembly process in the template wheel. Therefore, after the processor determines the hinge hole pose matrix matched by each hinge hole, pose fusion is performed based on the thread hole pose matrix matched by the thread holes belonging to the same door to be assembled to obtain the real-time fused pose of the hinge hole on the door to be assembled. Finally, the real-time installation pose of the hinge hole in the case of real-time assembly is determined by the fused pose of the hinge hole template, the real-time fused pose of the hinge hole, and the installation pose of the hinge hole template in the case of teaching assembly.

[0065] Next, introduce the method for determining the real-time installation pose of the hinge hole in the case of real-time assembly by the fused pose of the hinge hole template, the real-time fused pose of the hinge hole, and the installation pose of the hinge hole template in the case of teaching assembly. That is, the determination of the real-time installation pose of the hinge hole in the case of real-time assembly is introduced based on formula (1):

[0066] ; (1)

[0067] Wherein, represents the real-time installation pose of the hinge hole in the case of real-time assembly; represents the template installation pose of the hinge hole in the case of teaching assembly; represents the fused pose of the hinge hole template in the case of teaching assembly; represents the real-time fused pose of the hinge hole obtained in the case of real-time assembly.

[0068] Therefore, considering the fused pose of the hinge hole template, the real-time installation pose of the hinge hole in the case of real-time assembly can be accurately obtained, thereby improving the efficiency and reliability of determining the real-time installation pose of the hinge hole, and thus improving the reliability of the follow-up assembly of the door and the vehicle body.

[0069] The following introduces the method for determining the real-time installation pose of the door in the case of real-time assembly: In a specific embodiment, the processor determines the real-time installation pose of the door in the case of real-time assembly based on the screw hole pose matrix and the hinge hole pose matrix, including: the processor determines the fused pose of the screw hole template in the case of teaching assembly and the real-time fused pose of the screw hole in the case of real-time assembly; based on the fused pose of the hinge hole template and the real-time fused pose of the hinge hole, the fused pose of the screw hole template and the real-time fused pose of the screw hole, and the template installation pose of the door in the case of teaching assembly, the real-time installation pose of the door in the case of real-time assembly is determined.

[0070] Among them, the real-time fusion pose of the hinge hole in the case of real-time assembly is determined based on the hinge hole pose matrix, while the fusion pose of the hinge hole template is the installation pose of the hinge hole obtained by running the same assembly process as the actual situation in the template wheel, that is, the fusion pose of the hinge hole template in the case of teaching assembly and the installation pose of the hinge hole template in the case of teaching assembly can be obtained through the methods described above. At this time, the fusion pose of the hinge hole template in the case of teaching assembly and the installation pose of the hinge hole template in the case of teaching assembly are stored by data binding. Similarly, the real-time fusion pose of the threaded hole in the case of real-time assembly is determined based on the threaded hole pose matrix, while the fusion pose of the threaded hole template is the fusion pose of the threaded hole obtained by running the same assembly process as the actual situation in the template wheel, that is, the first robotic arm is manually operated to teach the precise grasping pose of the car door at the teaching position, and then the first camera takes pictures and collects the threaded holes of the car door at the teaching position during the manual teaching operation to obtain the template car door grayscale image and the template car door depth image, and the fusion pose of the threaded hole template is determined based on the template car door grayscale image and the template car door depth image. At this time, the bolt hole of the car door and the threaded hole of the body are manually sleeved, and the installation pose of the first robotic arm in the case of teaching assembly is recorded when recording the installation pose of the manual teaching. The installation pose of the first robotic arm in the case of teaching assembly is the installation pose of the car door template in the case of teaching assembly. Therefore, the fusion pose of the threaded hole template in the case of teaching assembly and the installation pose of the car door template in the case of teaching assembly are stored by data binding.

[0071] Specifically, in the actual assembly case, the processor determines the fusion pose of the hinge hole template and the installation pose of the hinge hole template in the case of teaching assembly, and determines the fusion pose of the threaded hole template and the installation pose of the car door template in the case of teaching assembly in a similar manner to the foregoing embodiments, that is, the processor can extract the fusion pose of the hinge hole template, the installation pose of the hinge hole template, the fusion pose of the threaded hole template, and the installation pose of the car door template in the case of teaching assembly from the data stored by running the assembly process in the template wheel. Therefore, after the processor determines the threaded hole pose matrix matched by each threaded hole, pose fusion is performed based on the threaded hole pose matrices matched by the threaded holes belonging to the same car door to be assembled to obtain the real-time fusion pose of the threaded holes on the car door to be assembled. And the fusion pose of the hinge hole template is determined through the methods described above, and then based on the fusion pose of the hinge hole template and the real-time fusion pose of the hinge hole, the fusion pose of the threaded hole template and the real-time fusion pose of the threaded hole, and the installation pose of the car door template in the case of teaching assembly, the real-time installation pose of the car door in the case of real-time assembly is determined.

[0072] The method for determining the real-time installation pose of the car door in the real-time assembly situation based on the pose fusion of the hinge hole template and the real-time fusion pose of the hinge hole, the pose fusion of the threaded hole template and the real-time fusion pose of the threaded hole, and the installation pose of the car door template in the teaching assembly situation will be introduced below based on formula (2):

[0073] ; (2)

[0074] Among them, represents the real-time installation pose of the car door in the real-time assembly situation; represents the installation pose of the car door template in the teaching assembly situation; represents the pose fusion of the hinge hole template in the teaching assembly situation; represents the real-time fusion pose of the hinge hole obtained in the real-time assembly situation; represents the pose fusion of the threaded hole template in the teaching assembly situation; represents the real-time fusion pose of the threaded hole obtained in the real-time assembly situation.

[0075] And it can be understood that the necessary condition for formula (2) to hold is that the base coordinate systems of the first robotic arm and the second robotic arm are completely coincident. High-precision devices such as laser trackers can be used for the calibration of the robotic arm coordinate system. The calibration method of the robotic arm coordinate system will not be introduced here.

[0076] Therefore, considering the pose fusion of the hinge hole template, the teaching pose of the car door installation template, and the pose fusion of the threaded hole template, the real-time installation pose of the car door in the real-time assembly situation can be accurately obtained, thereby improving the efficiency and reliability of determining the real-time installation pose of the car door, and further improving the reliability of the follow-up assembly of the car door and the vehicle body.

[0077] As can be seen from the foregoing introduction, neither the hinge hole pose matrix nor the threaded hole pose matrix has the property of orthogonality. Therefore, multiple pose matrices are required for pose fusion to obtain the required fusion pose. The method for obtaining the fusion pose by pose fusion will be introduced below: In a specific embodiment, the threaded holes on the car door to be assembled have an assembly correspondence with the hinge holes on the vehicle body to be assembled. Similar to the foregoing introduction, such as Figure 1The shown assembly schematic diagram of the car door and the vehicle body, where the threaded hole 21 matches the hinge hole 41, the threaded hole 22 matches the hinge hole 42, the threaded hole 23 matches the hinge hole 43, and the threaded hole 24 matches the hinge hole 44. That is, there is an assembly correspondence between the threaded hole 21 on the door to be assembled and the hinge hole 41 on the vehicle body to be assembled, between the threaded hole 22 on the door to be assembled and the hinge hole 42 on the vehicle body to be assembled, between the threaded hole 23 on the door to be assembled and the hinge hole 43 on the vehicle body to be assembled, and between the threaded hole 24 on the door to be assembled and the hinge hole 44 on the vehicle body to be assembled.

[0078] Based on this, the processor also determines a threaded hole pose matrix group and a hinge hole pose matrix group based on the assembly correspondence between each threaded hole and each vehicle body hinge hole. Among them, the number of threaded hole pose matrices in the threaded hole pose matrix group is the same as the number of hinge hole pose matrices in the hinge hole pose matrix group, and there is an assembly correspondence between the threaded hole that the threaded hole pose matrix matches and the vehicle body hinge hole that the hinge hole pose matrix matches; perform pose fusion on the threaded hole pose matrices in the threaded hole pose matrix group to obtain the real-time fused pose of the threaded holes; perform pose fusion on the hinge hole pose matrices in the hinge hole pose matrix group to obtain the real-time fused pose of the hinge holes.

[0079] Among them, the fused pose is used to represent the current pose, that is, the real-time fused pose of the threaded holes is used to represent the current pose of the threaded holes on the car door in the case of real-time assembly, and the real-time fused pose of the hinge holes is used to represent the current pose of the threaded holes on the vehicle body hinge in the case of real-time assembly. Secondly, the number of threaded hole pose matrices in the threaded hole pose matrix group is the same as the number of hinge hole pose matrices in the hinge hole pose matrix group, and the threaded holes that the threaded hole pose matrices match. For example, if the number of threaded hole pose matrices in the threaded hole pose matrix group is 2, then the number of hinge hole pose matrices in the hinge hole pose matrix group is also 2. And there is an assembly correspondence between the vehicle body hinge holes that match the hinge hole pose matrices. For example, the threaded hole pose matrix group includes: the threaded hole pose matrix A1 of the threaded hole 21, and the threaded hole pose matrix A2 of the threaded hole 22. Then the hinge hole pose matrix can include: the hinge hole pose matrix B1 of the hinge hole 41, and the hinge hole pose matrix B2 of the hinge hole 42.

[0080] Specifically, considering the robustness and applicability in the actual scenario, in this embodiment, after performing pose fusion on each hinge hole pose matrix and each threaded hole pose matrix respectively, the obtained fused poses are used to represent the threaded holes and hinge holes corresponding to the multiple recognized round holes. Therefore, considering that there is an assembly correspondence between the threaded holes on the door to be assembled and the hinge holes on the vehicle body to be assembled, that is, it is necessary to determine the threaded hole pose matrix group and the hinge hole pose matrix group in consideration of the aforementioned assembly correspondence, that is, to ensure that the number of matrices in the threaded hole pose matrix group and the hinge hole pose matrix group is the same, and the hinge holes and threaded holes corresponding to the matrices are matched. For example, if the threaded hole pose matrix A1 of the threaded hole 21, the threaded hole pose matrix A2 of the threaded hole 22, and the threaded hole pose matrix A3 of the threaded hole 23 are determined. And the hinge hole pose matrix B1 of the hinge hole 41, the hinge hole pose matrix B2 of the hinge hole 42, and the hinge hole pose matrix B4 of the hinge hole 44 are determined. Through the assembly correspondence of the aforementioned example, that is, there is an assembly correspondence between the threaded hole 21 and the hinge hole 41, there is an assembly correspondence between the threaded hole 22 and the hinge hole 42, and there is no assembly correspondence between the threaded hole 23 and the hinge hole 44. Therefore, the constructed threaded hole pose matrix group includes the threaded hole pose matrix A1 of the threaded hole 21 and the threaded hole pose matrix A2 of the threaded hole 22, and the constructed hinge hole pose matrix includes the hinge hole pose matrix B1 of the hinge hole 41 and the hinge hole pose matrix B2 of the hinge hole 42.

[0081] Further, for the threaded hole pose matrix group, pose fusion is performed on the threaded hole pose matrices in the threaded hole pose matrix group, that is, matrix average calculation is performed on the threaded hole pose matrices belonging to the same threaded hole pose matrix group, and the obtained matrix average calculation result is determined as the real-time fused pose of the threaded holes in the real-time assembly situation. Similarly, for the hinge hole pose matrix group, pose fusion is performed on the hinge hole pose matrices in the hinge hole pose matrix group, that is, matrix average calculation is performed on the hinge hole pose matrices belonging to the same hinge hole pose matrix group, and the obtained matrix average calculation result is determined as the real-time fused pose of the hinge holes in the real-time assembly situation.

[0082] Step 308, the first robotic arm aligns the door to be assembled with the vehicle body to be assembled based on the real-time installation pose of the door.

[0083] Specifically, after the processor determines the real-time installation pose of the door in the real-time assembly situation, the first robotic arm is controlled by the real-time installation pose of the door to align the door to be assembled with the vehicle body to be assembled, that is, to ensure that the threaded holes on the door to be assembled are aligned with the hinge holes on the body hinge of the vehicle body to be assembled.

[0084] Step 310, the second robotic arm tightens the bolts based on the real-time installation pose of the hinge holes, so as to install the door to be assembled onto the vehicle body to be assembled.

[0085] Specifically, after the processor determines the real-time installation pose of the hinge holes in the case of real-time assembly, at this time, the vehicle body is positioned based on the real-time installation pose of the hinge holes to determine the alignment of the door to be assembled and the vehicle body to be assembled, that is, to ensure again that the threaded holes on the door to be assembled are aligned with the hinge holes on the body hinge of the vehicle body to be assembled. At this time, the bolts are tightened, so that the bolts can assemble the aligned hinge holes and threaded holes, thereby installing the door to be assembled onto the vehicle body to be assembled.

[0086] In the above 3D vision-guided automotive door follow-up assembly system, when the first robotic arm and the second robotic arm are in the follow-up process, a hinge hole pose matrix is constructed through the acquired vehicle body grayscale image and vehicle body depth image, and the real-time installation pose of the hinge holes in the case of real-time assembly is determined based on the hinge hole pose matrix. Based on this, when the first robotic arm grabs the door to be assembled to the teaching position, a threaded hole pose matrix is constructed through the acquired door grayscale image and door depth image. Thus, through the threaded hole pose matrix and the hinge hole pose matrix, the real-time installation pose of the door in the case of real-time assembly can be determined, that is, there is no need for manual participation in the assembly process of the door and the vehicle body. Only image acquisition is required at the required positions and during the follow-up process to complete the determination of the pose matrix, so that the first robotic arm moves the door to be assembled from the preset transition position to the assembly position through the real-time installation pose of the door, and the second robotic arm tightens the bolts based on the real-time installation pose of the hinge holes, so as to install the door to be assembled onto the vehicle body to be assembled. That is, through the automatic calibration and determination of the poses of the first robotic arm and the second robotic arm, the door to be assembled is installed onto the vehicle body to be assembled. And because the first robotic arm and the second robotic arm are in follow-up, that is, the automatic assembly process does not need to stop the production line. On the basis of avoiding manual participation in the assembly, the assembly efficiency between the door and the vehicle body can be improved.

[0087] Since bolt adsorption and bolt tightening are required in practical applications, the following details the assembly method between the door and the vehicle body in practical applications: In an exemplary embodiment, step 304 includes step 402, step 308 includes step 404, and step 310 includes step 406, as Figure 4 shown, where:

[0088] Step 402, after the second camera adsorbs the bolt, the second robotic arm moves to the designated position, and when the second robotic arm is in the follow-up process with the line body plate chain, the vehicle body grayscale image and vehicle body depth image of the vehicle body to be assembled are acquired.

[0089] Specifically, place the body to be assembled on the line body pallet chain to ensure that when the body to be assembled runs on the follow-up mechanism, the body to be assembled is carried forward by the line body pallet chain, and control the running speed of the line body pallet chain to ensure that the body to be assembled advances to the assembly position at a constant speed. At this time, it is necessary to first control the hinge gun mechanism installed at the end of the second robotic arm to adsorb the bolt and then move to the specified position. Then, through the follow-up mechanism, ensure that the first robotic arm, the second robotic arm, and the line body pallet chain move synchronously to ensure that the first robotic arm, the second robotic arm, and the body to be assembled on the line body pallet chain are all in a relatively static state. At this time, collect the data of the body hinge holes on the body to be assembled through the second camera, and at this time, the hinge gun mechanism installed at the end of the second robotic arm adsorbs the bolt. Thus, take pictures and capture the body hinge holes of the body to be assembled through the second camera to obtain a body grayscale image and a body depth image including at least one hinge hole.

[0090] Step 404: The first robotic arm moves the door to be assembled from the preset transition position to the assembly position, and adjusts the door to be assembled through the real-time installation pose of the door to align the threaded holes with the hinge holes.

[0091] Specifically, after the processor determines the real-time installation pose of the door in the real-time assembly situation, first control the first robotic arm to move the door to be assembled from the preset transition position to the assembly position through the door clamping mechanism installed at the end, and adjust the door to be assembled through the real-time installation pose of the door calculated above to align the threaded holes with the hinge holes. For example, based on the foregoing example, at this time, it is necessary to align the threaded hole 21 with the hinge hole 41, the threaded hole 22 with the hinge hole 42, the threaded hole 23 with the hinge hole 43, and the threaded hole 24 with the hinge hole 44.

[0092] Step 406: The second robotic arm sleeves the adsorbed bolt into the hinge hole based on the real-time installation pose of the hinge hole; after the threaded hole is aligned with the hinge hole, tighten the bolt to install the door to be assembled on the body to be assembled.

[0093] Specifically, after the processor determines the real-time installation pose of the hinge holes in the real-time assembly situation, at this time, the body is positioned based on the real-time installation pose of the hinge holes to determine the alignment of the door to be assembled and the body to be assembled. That is, it is necessary to align the threaded holes and the hinge holes based on the real-time installation pose of the hinge holes. That is, the pose of the body to be assembled is controlled by the second robotic arm, and the pose of the door to be assembled is controlled by the first robotic arm. Ensure that after the door to be assembled reaches the assembly position, the threaded holes on the door to be assembled are aligned with the hinge holes on the body hinge of the body to be assembled, that is, the alignment result shown in step 404 is obtained. At this time, control the hinge gun mechanism installed at the end of the second robotic arm to sleeve the adsorbed bolt into the aligned hinge hole, and after the door to be assembled reaches the assembly position and it is determined that the threaded hole and the hinge hole are aligned, control the hinge gun mechanism installed at the end of the second robotic arm to tighten the bolt, so as to install the door to be assembled onto the body to be assembled.

[0094] For ease of understanding, as can be seen from the foregoing example, the second robotic arm needs to adsorb the bolt 5, pass the adsorbed bolt 5 through the hinge hole 41, and then screw the hinge hole 41 into the matching threaded hole 21. Similarly, after passing the adsorbed bolt 5 through the hinge hole 42, screw the hinge hole 42 into the matching threaded hole 22. After passing the adsorbed bolt 5 through the hinge hole 43, screw the hinge hole 43 into the matching threaded hole 23. And after passing the adsorbed bolt 5 through the hinge hole 44, screw the hinge hole 44 into the matching threaded hole 24. Therefore, through the foregoing operations, the planes where the threaded holes 2 on the hinge 3 and the door 1 are located can be made to fit, so as to install the door to be assembled onto the body to be assembled, thereby completing the follow-up assembly of the door and the body.

[0095] In this embodiment, there is no need for manual participation in the assembly process of the door and the body. Through automatic bolt adsorption, by automatically calibrating and determining the poses of the first robotic arm and the second robotic arm, the first robotic arm and the second robotic arm can ensure that the threaded holes on the door to be assembled are aligned with the hinge holes on the body hinge of the body to be assembled during the follow-up process, and then install the door to be assembled onto the body to be assembled by tightening the bolts, thereby completing the follow-up assembly of the door and the body. Since the corresponding automatic assembly can be completed during the follow-up process, there is no need to stop the production line and avoid manual participation, thereby improving the assembly efficiency and reliability between the door and the body.

[0096] Next, the method for determining the pose matrix of the threaded hole will be introduced. It can be understood that in practical applications, the method for determining the pose matrix of the hinge hole is similar, which will not be elaborated here. In an exemplary embodiment, as Figure 5 shown, where:

[0097] Step 502: The processor performs ellipse recognition on the door gray-scale image to obtain multiple threaded hole features; performs point cloud restoration for each threaded hole feature based on the door depth map to obtain the target threaded hole neighborhood point cloud corresponding to each threaded hole feature; determines the corresponding threaded hole plane point cloud and the threaded hole inner contour point cloud based on each target threaded hole neighborhood point cloud; determines the corresponding threaded hole plane normal based on the threaded hole plane point cloud, and determines the corresponding threaded hole center based on the threaded hole inner contour point cloud; constructs the threaded hole pose matrix of each threaded hole inner contour point cloud based on the threaded hole center of each threaded hole plane point cloud, the threaded hole plane normal where each threaded hole inner contour point cloud is located, and the preset coordinate value.

[0098] Among them, the target threaded hole neighborhood point cloud is a point cloud without invalid values and without outliers. Secondly, the threaded hole inner contour point cloud is a point cloud circle obtained by performing circle fitting on each threaded hole inner edge point cloud. And the threaded hole pose matrix matches the threaded holes in the door to be assembled, that is, the threaded hole pose matrix A1 corresponding to the threaded hole 21, the threaded hole pose matrix A2 corresponding to the threaded hole 22, and the threaded hole pose matrix A3 matching the threaded hole 23 introduced in the foregoing example.

[0099] Specifically, after the processor obtains the door gray-scale image and the door depth map collected by the first camera, it first performs ellipse recognition on the door gray-scale image to obtain multiple threaded hole features. The following introduces the method for performing 2D ellipse recognition on the door gray-scale image to obtain multiple threaded hole features:

[0100] Since the door gray-scale image is specifically a 2D gray-scale image collected for the door to be assembled, and the 2D gray-scale image includes gradient information, so ellipse features are recognized for the gradient information included in the door gray-scale image. Since a circle is a degenerate form of an ellipse, performing ellipse recognition on threaded holes has better robustness. Performing ellipse recognition on the door gray-scale image to obtain multiple threaded hole features specifically includes the following three steps: edge detection and smooth segment extraction, generating candidate ellipses with valid coding, and result de-duplication and verification. The foregoing steps are introduced separately below:

[0101] 1. Edge detection and smooth segment extraction. For the gradient information included in the door gray-scale image, approximate the edge through multiple series of line segments, and use fast vector calculation to extract smooth arcs, that is, at this time, approximate edge line segments can be obtained based on the gradient information included in the door gray-scale image.

[0102] 2. Generating candidate ellipses with valid coding. Construct a directed graph for the foregoing approximate edge line segments. If two nodes in the directed graph satisfy constraint conditions such as adaptive shape, tangent similarity, and distribution compensation, connect the two nodes that satisfy the foregoing constraint conditions with a directed edge, which also means that the two nodes that satisfy the foregoing constraint conditions may come from the same ellipse. Candidate ellipses can be generated through the foregoing method.

[0103] 3. Result deduplication and verification. The candidate ellipses generated by the foregoing method are verified and deduplicated through gradient verification and clustering methods. That is, by sampling the points considered to be on the ellipse and substituting them into the ellipse equation, and comparing with the gradient values obtained by the Sobel operator, the matching scores of the candidate ellipses are obtained. If the matching score of a candidate ellipse is higher than the set value, the candidate ellipse is considered valid. Secondly, the Euclidean distance is calculated for the valid candidate ellipses, and the similar valid candidate ellipses are clustered based on the Euclidean distance to suppress duplicate ellipses.

[0104] Therefore, through the method introduced above, we can perform ellipse recognition on the door gray-scale image to obtain multiple thread hole features, and the thread hole features include the centers of the thread hole features and the pixel point coordinates that make up the thread hole features. It can be understood that the method of performing ellipse recognition on the door gray-scale image in this embodiment exemplifies the use of a 2D ellipse recognition method, and corresponding results can also be obtained by using other ellipse recognition methods in practical applications. For example, Hough circle recognition, etc. Therefore, the 2D ellipse recognition should not be understood as a specific limitation on the ellipse recognition of the door gray-scale image.

[0105] Furthermore, multiple thread hole features are obtained by performing ellipse recognition on the door gray-scale image through the foregoing method, and the centers of the thread hole features and the pixel point coordinates that make up the thread hole features are obtained. Next, the point cloud restoration of the thread hole features will be combined with the door depth map. That is, the door depth map is obtained by the first camera by setting to collect depth information, and the 3D data can be represented by depth maps, point clouds, meshes, and volume meshes. The conversion of the depth map to the point cloud is actually a coordinate system transformation, from the image coordinate system in the depth map to the world coordinate system. The constraint condition for the transformation of the image coordinate system to the world coordinate system is the camera internal parameter. As introduced above, the acquisition of the door gray-scale image and the door depth map is in the same coordinate system. By performing ellipse recognition on the door gray-scale image, multiple thread hole features can be obtained, and the thread hole features include the centers of the thread hole features and the pixel point coordinates that make up the thread hole features. That is, at this time, based on the door gray-scale image, the coordinate system transformation of the centers of the thread hole features and the pixel point coordinates that make up the thread hole features is performed based on the camera internal parameter of the first camera, so as to obtain the target thread hole neighborhood point clouds corresponding to each thread hole feature.

[0106] Based on this, the processor further determines the corresponding threaded hole plane point cloud and the inner contour point cloud of the threaded hole for each target threaded hole neighborhood point cloud, determines the corresponding threaded hole plane normal based on the threaded hole plane point cloud, and determines the corresponding threaded hole center based on the inner contour point cloud of the threaded hole. That is, the processor performs plane projection based on each target threaded hole neighborhood point cloud, and performs circle fitting on the edge point cloud obtained from the plane projection, so as to obtain the inner contour point cloud of the threaded hole for each target threaded hole neighborhood point cloud respectively, then determines the corresponding threaded hole plane normal based on the threaded hole plane point cloud, and determines the corresponding threaded hole center based on the inner contour point cloud of the threaded hole. Thus, based on the threaded hole center of each threaded hole plane point cloud, the threaded hole plane normal where each inner contour point cloud of the threaded hole is located, and the preset coordinate value, the threaded hole pose matrix of each inner contour point cloud of the threaded hole is constructed.

[0107] Considering that the threaded hole features obtained by ellipse recognition may have actual errors in the data included in the threaded hole features due to image acquisition quality and actual environment problems, that is, it is necessary to perform corresponding data processing on the point cloud corresponding to the threaded hole features. The following is an introduction to this: In a specific embodiment, point cloud restoration is performed for each threaded hole feature based on the door depth map, and the target threaded hole neighborhood point cloud corresponding to each threaded hole feature is obtained, including: performing a plane screenshot for each threaded hole feature to obtain the threaded hole feature plan view corresponding to each threaded hole feature; performing gray-scale neighborhood extraction for each threaded hole feature to obtain the region of interest of the threaded hole corresponding to each threaded hole feature; performing point cloud data restoration on each region of interest of the threaded hole based on the door depth map to obtain the first threaded hole plane point cloud corresponding to each region of interest of the threaded hole; removing the invalid values in each first threaded hole plane point cloud to obtain the second threaded hole plane point cloud; performing statistical filtering on each second threaded hole plane point cloud to remove the outliers in the second threaded hole plane point cloud to obtain the third threaded hole plane point cloud; calculating the flatness of each third threaded hole plane point cloud, and determining the third threaded hole plane point cloud whose flatness meets the flatness threshold as the target threaded hole neighborhood point cloud.

[0108] Specifically, after the processor performs ellipse recognition on the door gray-scale map to obtain multiple threaded hole features, for each threaded hole feature, gray-scale neighborhood extraction is performed to obtain the region of interest of the threaded hole corresponding to each threaded hole feature, and then point cloud data restoration is performed on each region of interest of the threaded hole based on the door depth map to obtain the first threaded hole plane point cloud corresponding to each region of interest of the threaded hole. Then, the invalid values in each first threaded hole plane point cloud are removed to obtain the second threaded hole plane point cloud. The aforementioned invalid value is a non-numeric value (NotaNumber, NaN value), and the non-numeric value represents an undefined or non-representable value, so as to ensure that the second threaded hole plane point cloud does not include invalid values.

[0109] Further, statistical filtering is performed on the point clouds of each second threaded hole plane. Outliers are determined based on the statistical filtering results of each threaded hole plane point cloud, and the outliers in the second threaded hole plane point cloud are removed to obtain the third threaded hole plane point cloud, so as to ensure that the third threaded hole plane point cloud does not include outliers. Finally, the flatness of each third threaded hole plane point cloud is calculated, that is, a plane fitting is performed on the third threaded hole plane point cloud to obtain a threaded hole parameter plane, and then the flatness of the third threaded hole plane point cloud on the threaded hole parameter plane is calculated. Based on the flatness threshold, the flatness of each third threaded hole plane point cloud is screened, and the third threaded hole plane point cloud with a flatness meeting the flatness threshold is determined as the target threaded hole neighborhood point cloud, thereby forming an accurate target threaded hole neighborhood point cloud containing 3D circular hole features and the nearby plane.

[0110] The following introduces the method for obtaining the inner contour point cloud of the threaded hole and the center of the threaded hole of the threaded hole plane point cloud of the threaded hole through circle fitting: In a specific embodiment, each target threaded hole neighborhood point cloud is projected onto the threaded hole parameter plane to obtain the target threaded hole plane projection point cloud corresponding to each target threaded hole neighborhood point cloud; based on the angular distribution, the corresponding threaded hole edge point cloud is extracted from each target threaded hole plane projection point cloud. The threaded hole edge point cloud includes the inner threaded hole edge point cloud and the outer threaded hole edge point cloud; the inner threaded hole edge point cloud is extracted from each threaded hole edge point cloud, and circle fitting is performed based on each inner threaded hole edge point cloud to obtain the threaded hole center corresponding to each threaded hole.

[0111] Specifically, the processor projects each target threaded hole neighborhood point cloud onto the threaded hole parameter plane to obtain the target threaded hole plane projection point cloud corresponding to each target threaded hole neighborhood point cloud, and then extracts the corresponding threaded hole edge point cloud from each target threaded hole plane projection point cloud based on the angular distribution. The threaded hole edge point cloud includes the inner threaded hole edge point cloud and the outer threaded hole edge point cloud. At this time, an inner and outer edge judgment is performed on the threaded hole edge point cloud, that is, it is judged whether each edge point belongs to the inner edge of the threaded hole or the outer edge of the threaded hole, and the edge points belonging to the outer edge of the threaded hole are extracted, thereby forming the inner threaded hole edge point cloud. Finally, the inner threaded hole edge point cloud is extracted from each threaded hole edge point cloud, and circle fitting is performed based on each inner threaded hole edge point cloud to obtain the threaded hole center corresponding to each threaded hole.

[0112] The following introduces the method for obtaining the parameter plane: In a specific embodiment, the processor also uses the region growing clustering method to perform clustering processing on each third threaded hole plane point cloud, and determines the threaded hole plane point cloud cluster with the largest number of included point clouds from the third threaded hole plane point cloud; the random sample consensus algorithm is used to perform plane fitting on the threaded hole plane point cloud cluster to obtain the threaded hole parameter plane.

[0113] Specifically, the processor uses the region growing clustering method to cluster each third threaded hole plane point cloud, and determines the cluster of threaded hole plane point clouds with the largest number of included point clouds from the third threaded hole plane point cloud. At this time, the cluster of threaded hole plane point clouds with the largest number contains the data of the 3D round hole and its nearby planes. Finally, the random sample consensus (RANSAC) algorithm is used to perform plane fitting on the cluster of threaded hole plane point clouds to obtain multiple plane equation coefficients, that is, the first plane equation coefficient can be obtained. , the second plane equation coefficient , the third plane equation coefficient , and the fourth plane equation coefficient . The threaded hole parameter plane can be constructed through the foregoing multiple plane equation coefficients.

[0114] In a specific embodiment, the processor specifically constructs a threaded hole pose matrix for each threaded hole based on the normal direction of each threaded hole plane and the center of the threaded hole; wherein, the threaded hole pose matrix matches the threaded hole in the door to be assembled.

[0115] Specifically, the processor generates a 4×4 matrix T for each inner contour point cloud of the threaded hole to describe the threaded hole pose matrix of each inner contour point cloud of the threaded hole. At this time, first determine the center of the threaded hole of the threaded hole plane point cloud, and denote the center of the threaded hole of the threaded hole plane point cloud as , and use the normal of the threaded hole plane of the plane where the inner contour point cloud of the threaded hole is projected as the direction vector of z (obtained after normalization). At this time, ensure that the direction vector of z points to the optical center, that is, the direction of the normal of the threaded hole plane where each inner contour point cloud of the threaded hole is located , and then determine the preset coordinate value according to the scene requirements in actual application . Therefore, the threaded hole pose matrix of the inner contour point cloud of the threaded hole is as shown in formula (3):

[0116] (3)

[0117] Wherein, represents the threaded hole pose matrix of the inner contour point cloud of the threaded hole, represents the center of the threaded hole of the threaded hole plane point cloud, represents the direction of the normal of the threaded hole plane where the inner contour point cloud of the threaded hole is located, and represent the preset coordinate values, and .

[0118] It can be understood that in this embodiment, the 3D circle fitting algorithm can also obtain the center coordinates and the fitting results of the round point cloud using other fitting methods. Therefore, the 3D circle fitting algorithm should not be understood as a specific limitation of this application.

[0119] Similarly, the foregoing similar steps can also be performed for the hinge holes, that is, the processor specifically performs ellipse recognition on the body grayscale image to obtain multiple hinge hole ellipses; based on the body depth image, point cloud restoration is performed for each hinge hole ellipse to obtain the target hinge hole point clouds corresponding to the respective hinge hole ellipses; based on the respective target hinge hole point clouds, the hinge hole point cloud circles corresponding to each of them and the hinge hole point cloud centroids of the hinge hole point cloud circles are determined; based on the hinge hole point cloud centroids, the normal vectors of the hinge hole planes where the hinge hole point cloud circles are located, and the preset coordinate values, the hinge hole pose matrix of each hinge hole point cloud circle is constructed; wherein, the hinge hole pose matrix matches the hinge holes in the body to be assembled. The specific implementation manner is similar to the foregoing embodiment and will not be elaborated here.

[0120] In this embodiment, ellipse recognition is performed on the door grayscale image to preliminarily locate the threaded holes, and then point cloud restoration is combined with the door depth image to ensure that the obtained point cloud can more accurately obtain the circular hole information of the threaded holes. Therefore, subsequent pose calculation and pose fusion can ensure the reliability of the determined pose, thereby ensuring the reliability of the positioning and assembly of the door and the body, and further improving the assembly accuracy between the door and the body.

[0121] As can be seen from the foregoing introduction, there are ellipse recognition, circle fitting, and pose calculation. Next, the overall process of ellipse recognition, circle fitting, and pose calculation will be introduced, as Figure 6 shown:

[0122] Step 601, data input. That is, the first camera transmits to the processor the door grayscale image and the door depth image of the door to be assembled collected when the first robotic arm grabs the door to be assembled to the teaching position.

[0123] Step 602, data extraction and processing. That is, first perform ellipse recognition on the door grayscale image to obtain multiple threaded hole features, then perform grayscale image neighborhood extraction for each threaded hole feature to obtain the region of interest of the threaded hole corresponding to each threaded hole feature, and based on the door depth image, perform point cloud data restoration for each region of interest of the threaded hole to obtain the first threaded hole plane point cloud corresponding to each region of interest of the threaded hole. Thus, the invalid values in each first threaded hole plane point cloud are removed to obtain the second threaded hole plane point cloud; perform statistical filtering on each second threaded hole plane point cloud to remove the outliers in the second threaded hole plane point cloud to obtain the third threaded hole plane point cloud. At this time, use the region growing clustering method to perform clustering processing on each third threaded hole plane point cloud, determine the threaded hole plane point cloud cluster with the largest number of included point clouds from the third threaded hole plane point cloud, and use the random sample consensus algorithm to perform plane fitting on the threaded hole plane point cloud cluster to obtain the threaded hole parameter plane. Therefore, the flatness of each third threaded hole plane point cloud can be calculated, and the third threaded hole plane point cloud whose flatness meets the flatness threshold is determined as the target threaded hole neighborhood point cloud.

[0124] Step 603: Circle fitting. Specifically, project the point clouds in the neighborhoods of the target threaded holes onto the parameter planes of the threaded holes to obtain the projected point clouds of the target threaded hole planes corresponding to the point clouds in the neighborhoods of the respective target threaded holes. Then, extract the corresponding threaded hole edge point clouds from the projected point clouds of the target threaded hole planes based on the angular distribution. The threaded hole edge point clouds include the inner edge point clouds and the outer edge point clouds of the threaded holes. Finally, extract the inner edge point clouds of the threaded holes from the threaded hole edge point clouds and perform circle fitting based on the inner edge point clouds of the threaded holes to obtain the centers of the threaded holes corresponding to the respective threaded holes.

[0125] Step 604: Pose matrix generation. Based on the normal vectors of the planes of the threaded holes and the centers of the threaded holes, construct the pose matrices of the threaded holes for each threaded hole.

[0126] The following is a common introduction to the timing flowchart of the follow-up assembly of the car door and the vehicle body through the visual assembly steps and the mechanical assembly steps in the system, as Figure 7 shown in the timing flowchart of the visual assembly steps and the mechanical assembly steps in the system:

[0127] Step 701: Car door positioning. Corresponding to the first robotic arm in the mechanical assembly step grasping the car door to be assembled, the second robotic arm adsorbing the bolt, and the second robotic arm preparing for follow-up. That is, the first robotic arm grasps the car door to be assembled to the taught position. At this time, the first camera acquires the grayscale image and the depth image of the car door to be assembled. And the second robotic arm moves to the specified position after adsorbing the bolt and prepares for follow-up.

[0128] Step 702: Car door deviation correction. For the grayscale image and the depth image of the car door to be assembled acquired by the first camera, perform deviation correction processing on the first robotic arm and determine that the first robotic arm is ready for follow-up.

[0129] Step 703: Start follow-up. After the second robotic arm moves to the specified position after adsorbing the bolt and it is determined that the robotic arm is ready for follow-up. And after performing deviation correction processing on the first robotic arm based on the grayscale image and the depth image of the car door to be assembled acquired by the first camera and it is determined that the first robotic arm is ready for follow-up. That is, after it is determined that both the first robotic arm and the second robotic arm are ready for follow-up, start the follow-up for the first robotic arm and the second robotic arm.

[0130] Step 704, hinge hole positioning. That is, when the second robotic arm and the line body plate chain are in a follow-up process, the second camera acquires the body grayscale image and the body depth image of the vehicle body to be assembled, and the processor constructs a hinge hole pose matrix matching the hinge hole based on the body grayscale image and the body depth image, and determines the real-time installation pose of the hinge hole in the real-time assembly situation based on the hinge hole pose matrix, thus completing the hinge hole positioning. At this time, the second robotic arm specifically positions the hinge hole based on the real-time installation pose of the hinge hole, and slews the adsorbed bolt into the hinge hole.

[0131] Step 705, combined calculation. Since when the first robotic arm grabs the vehicle door to be assembled to the teaching position, the first camera acquires the door grayscale image and the door depth image of the vehicle door to be assembled, at this time the processor constructs a threaded hole pose matrix matching the threaded hole based on the door grayscale image and the door depth image, and determines the real-time installation pose of the vehicle door in the real-time assembly situation based on the hinge hole pose matrix and the threaded hole pose matrix obtained from the foregoing calculations, thereby moving the vehicle door to be assembled from the preset transition position to the assembly position, and adjusting the vehicle door to be assembled through the real-time installation pose of the vehicle door to align the threaded hole with the hinge hole. Then, after the threaded hole and the hinge hole are aligned, the second robotic arm tightens the bolt so as to install the vehicle door to be assembled on the vehicle body to be assembled, thus completing the follow-up assembly of the vehicle door and the vehicle body.

[0132] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it realizes the control method executed by the processor in the foregoing embodiments. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0134] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0135] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it realizes the control steps executed by the processor in the foregoing embodiments.

[0136] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it realizes the control steps executed by the processor in the foregoing embodiments.

[0137] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it realizes the control steps executed by the processor in the foregoing embodiments.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0139] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0140] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0141] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A 3D vision-guided automobile door follow-up assembly system, characterized in that: The automobile door follow-up assembly system at least includes a line plate chain, a follow-up mechanism, a first camera fixed to the ground, a first mechanical arm, a second mechanical arm, a second camera fixed to the end of the second mechanical arm, and a processor; The first camera is used to obtain a door grayscale image and a door depth image of the door to be assembled when the first robot arm grabs the door to be assembled to the teaching position for secondary positioning and deviation correction; the door grayscale image and the door depth image both include threaded holes on the door to be assembled; The second camera is used to obtain a body grayscale image and a body depth image of the body to be assembled when the second robot arm moves to a specified position after sucking the bolts and the second robot arm and the line plate chain are in a follow-up process; the body grayscale image and the body depth image both include hinge holes on the body to be assembled; The processor is used to construct a hinge hole pose matrix matching the hinge hole based on the vehicle body grayscale image and the vehicle body depth image, and determine the real-time installation pose of the hinge hole in a real-time assembly situation based on the hinge hole pose matrix; construct a threaded hole pose matrix matching the threaded hole based on the vehicle door grayscale image and the vehicle door depth image, and determine the real-time installation pose of the door in a real-time assembly situation based on the threaded hole pose matrix and the hinge hole pose matrix; The first mechanical arm is used to move the vehicle door to be assembled from a preset transition position to an assembly position, and adjust the vehicle door to be assembled through the real-time installation posture of the vehicle door so that the threaded hole is aligned with the hinge hole; The second mechanical arm is used to insert the adsorbed bolt into the hinge hole based on the real-time installation posture of the hinge hole; and tighten the bolt after the threaded hole is aligned with the hinge hole, so as to install the vehicle door to be assembled on the vehicle body to be assembled; The line plate chain is used to carry the vehicle body to be assembled forward; The follower mechanism is used to keep the first mechanical arm, the second mechanical arm and the line plate chain in synchronous motion.

2. The system according to claim 1, characterized in that The processor is specifically used for: Determine the template fusion pose of the hinge hole in the teaching assembly case and the real-time fusion pose of the hinge hole in the real-time assembly case; The real-time installation posture of the hinge hole in the real-time assembly situation is determined through the hinge hole template fusion posture, the hinge hole real-time fusion posture, and the hinge hole template installation posture in the teaching assembly situation.

3. The system according to claim 1, characterized in that The processor is specifically used for: Determine the threaded hole template fusion pose in the teaching assembly case and the threaded hole real-time fusion pose in the real-time assembly case; Based on the hinge hole template fusion posture and the hinge hole real-time fusion posture, the threaded hole template fusion posture and the threaded hole real-time fusion posture, and the door template installation posture in the teaching assembly situation, the real-time installation posture of the door in the real-time assembly situation is determined.

4. The system according to claim 3, characterized in that The threaded hole on the vehicle door to be assembled and the hinge hole on the vehicle body to be assembled have an assembly corresponding relationship; The processor is specifically used for: Based on the assembly correspondence between each of the threaded holes and each of the vehicle body hinge holes, a threaded hole pose matrix group and a hinge hole pose matrix group are determined, wherein the number of threaded hole pose matrices in the threaded hole pose matrix group is consistent with the number of hinge hole pose matrices in the hinge hole pose matrix group, and there is an assembly correspondence between the threaded holes matched by the threaded hole pose matrices and the vehicle body hinge holes matched by the hinge hole pose matrices; Performing posture fusion on the threaded hole posture matrices in the threaded hole posture matrix group to obtain the real-time fused posture of the threaded hole; The hinge hole pose matrices in the hinge hole pose matrix group are subjected to pose fusion to obtain the real-time fused pose of the hinge hole.

5. The system according to any one of claims 1 to 4, characterized in that: The processor is specifically used for: Performing ellipse recognition on the door grayscale image to obtain a plurality of threaded hole features; Performing point cloud recovery for each of the threaded hole features based on the door depth map to obtain a target threaded hole neighborhood point cloud corresponding to each of the threaded hole features; Determine the corresponding threaded hole plane point cloud and threaded hole inner contour point cloud based on the neighborhood point cloud of each target threaded hole; The corresponding threaded hole plane normal is determined based on the threaded hole plane point cloud, and the corresponding threaded hole center is determined based on the threaded hole inner contour point cloud.

6. The system according to claim 5, characterized in that The processor is specifically used for: Performing grayscale image neighborhood extraction on each of the threaded hole features to obtain a threaded hole region of interest corresponding to each threaded hole feature; performing point cloud data recovery on each of the threaded hole regions of interest based on the door depth map to obtain a first threaded hole plane point cloud corresponding to each of the threaded hole regions of interest; Removing invalid values ​​in each of the first threaded hole plane point clouds to obtain a second threaded hole plane point cloud; performing statistical filtering on each of the second threaded hole plane point clouds to remove outliers in the second threaded hole plane point cloud to obtain a third threaded hole plane point cloud; The flatness of each of the third threaded hole plane point clouds is calculated, and the third threaded hole plane point cloud whose flatness meets the flatness threshold is determined as the target threaded hole neighborhood point cloud.

7. The system according to claim 6, characterized in that The processor is specifically used for: Projecting each of the target threaded hole neighborhood point clouds onto the threaded hole parameter plane to obtain the target threaded hole plane projection point clouds corresponding to each of the target threaded hole neighborhood point clouds; Extracting corresponding threaded hole edge point clouds from each target threaded hole plane projection point cloud based on the angle distribution, wherein the threaded hole edge point clouds include threaded hole inner edge point clouds and threaded hole outer edge point clouds; The inner edge point cloud of the threaded hole is extracted from the edge point cloud of each threaded hole, and a circle fitting is performed based on the inner edge point cloud of each threaded hole to obtain the threaded hole center corresponding to each threaded hole.

8. The system according to claim 7, characterized in that The processor is specifically used for: Performing clustering processing on each of the third threaded hole plane point clouds using a region growing clustering method, and determining a threaded hole plane point cloud cluster including a maximum number of point clouds from the third threaded hole plane point clouds; A random sampling consensus algorithm is used to perform plane fitting on the threaded hole plane point cloud cluster to obtain the threaded hole parameter plane.

9. The system according to claim 5, characterized in that The processor is specifically used for: Based on the plane normal of each threaded hole and the center of the threaded hole, a threaded hole pose matrix of each threaded hole is constructed; wherein the threaded hole pose matrix matches the threaded hole in the vehicle door to be assembled.

10. The system according to claim 1, characterized in that The follower mechanism is used to ensure that the first mechanical arm, the second mechanical arm and the vehicle body to be assembled are in a relatively static state.

Images