Visual guidance method and system based on multi-camera combined positioning and computer equipment

Abstract

The invention relates to the technical field of machine vision guidance, and discloses a vision guidance method and system based on multi-camera joint positioning and computer equipment. In the off-line template stage, local point cloud data of the large-size part is collected; and in the online guiding stage, part data collected by the same camera is registered with the offline local point cloud, and a rotation translation matrix from the offline template point cloud to the online guiding point cloud is calculated, so that the local deflection of the large-size part under the camera coordinate system is determined. After the local area deflection data of the large-size part under each camera is determined, fusion calculation can be carried out on multiple pieces of local deflection information, the overall positioning pose of the part is obtained, and therefore the mechanical arm is guided to complete grabbing operation. The problem that a single camera cannot meet high-precision visual guidance of large-size parts can be solved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision guidance technology, and in particular to a vision guidance method, system and computer equipment based on multi-camera joint positioning. Background Art

[0002] Existing visual positioning solutions often utilize a single 2D / 3D camera to photograph parts and collect grayscale or depth map data. The captured data is then used to identify and locate features, such as circular holes, square holes, and corner points within a part. The principle of camera imaging indicates that camera accuracy depends on both its resolution and the size of its field of view. For a given field of view, higher resolution translates to a smaller size represented by each pixel in the image, resulting in higher camera accuracy. Conversely, lower resolution translates to lower accuracy. Therefore, when locating larger parts, a high-precision camera struggles to capture complete part feature information, while a camera with a wider field of view struggles to achieve high accuracy. Using a high-precision camera can only capture features from a small, localized area of ​​the part. Using localized features for positioning can easily lead to significant deviations in the overall positioning of larger parts due to feature positioning errors, making it impossible to accurately grasp the workpiece.

[0003] Furthermore, most current feature recognition and positioning algorithms struggle to maintain stable recognition performance in complex lighting conditions and working environments. To achieve stable recognition of component features, manual selection of features, such as circular holes, that are easier for vision algorithms to identify is often required. However, in some scenarios, components lack distinct features like circular holes or screw holes, and using corner points or other features can lead to unstable recognition algorithms due to lighting shadows or other scene factors.

[0004] It can be seen that the existing technology has the problem that a single camera cannot meet the high-precision visual guidance requirements for large-sized parts; and it is difficult to stably perform visual positioning guidance for parts that do not have obvious features that facilitate visual recognition. Summary of the Invention

[0005] The present invention provides a visual guidance method, system and computer equipment based on multi-camera joint positioning to solve the problems in the prior art that a single camera cannot meet the high-precision visual guidance requirements for large-sized parts; and it is difficult to stably perform visual positioning guidance for parts without obvious features that facilitate visual recognition.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, the present invention provides a visual guidance method based on multi-camera joint positioning, comprising:

[0008] S1: Obtain the local area of ​​a large-size component captured by N cameras and calculate the component pose information of the local area captured by each camera;

[0009] S2: Calculate the pose information of the entire large-scale component online based on the pose information of the component in the local area under each camera;

[0010] S3: determining a transformation relationship between the large-scale component pose information calculated offline and the large-scale component pose information calculated online;

[0011] S4: Calculate the required offset information of the end gripper according to the transformation relationship, and guide the robotic arm to complete the corresponding offset according to the end gripper offset information to achieve grasping.

[0012] In the second aspect, the present application provides a visual guidance system based on multi-camera joint positioning, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the above-mentioned first aspect method are implemented.

[0013] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0014] Beneficial effects:

[0015] The visual guidance method based on multi-camera joint positioning provided by the present invention collects local point cloud data of large-scale parts in the offline template stage; in the online guidance stage, the part data collected by the same camera is used to align with the offline local point cloud, and the rotation and translation matrix from the offline template point cloud to the online guidance point cloud is calculated to determine the local deflection of the large-scale part in the camera coordinate system. After determining the local area deflection data of the large-scale part under each camera, the multiple local deflection information can be fused and calculated to obtain the overall positioning posture of the part, which is used to guide the robotic arm to complete the grasping operation. This application can solve the problem that a single camera cannot meet the high-precision visual guidance requirements for large-scale parts.

[0016] A further approach proposes using localized point cloud data of parts acquired from 3D cameras to match it with point cloud data collected during the offline configuration phase. By fusing visual positioning information from different features across multiple cameras, this approach enables precise positioning guidance for grasping or placing large parts. This approach can also provide stable visual positioning guidance for parts without obvious features that facilitate visual recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1This is a flow chart of a visual guidance method based on multi-camera joint positioning according to a preferred embodiment of the present invention;

[0018] Figure 2 This is a flow chart of a method for visually identifying and locating a surface of a local area of ​​a component without obvious features according to a preferred embodiment of the present invention;

[0019] Figure 3 Flowchart of the visual guidance method for feature fusion according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0021] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0022] See Figure 1 , this application provides a visual guidance method based on multi-camera joint positioning, including:

[0023] S1: Obtain the local area of ​​a large-size component captured by N cameras and calculate the component pose information of the local area under each camera.

[0024] In this step, N is a positive integer. In one example, N is 3, and in another example, N is 4. This is only an example and not a limitation.

[0025] Among them, obtaining the local area of ​​the large-size component photographed by N cameras means using N cameras to photograph the large-size component. During the shooting, each camera photographs a partial area of ​​the large-size component, which constitutes the images photographed by N cameras, and the complete image of the large-size component can be obtained.

[0026] S2: The pose information of the entire large-scale component is calculated online based on the pose information of the component in the local area under each camera.

[0027] S3: Determine the transformation relationship between the large-size component pose information calculated offline and the large-size component pose information calculated online.

[0028] In this step, the relative relationship between the large-scale component pose information calculated offline and the large-scale component pose information calculated online is T(R, t).

[0029] S4: Calculate the required offset information of the end gripper according to the transformation relationship, and guide the robotic arm to complete the corresponding offset according to the end gripper offset information to achieve grasping.

[0030] The above-mentioned visual guidance method based on multi-camera joint positioning collects local point cloud data of large-scale parts in the offline template stage; in the online guidance stage, the part data collected by the same camera is used to align with the offline local point cloud, and the rotation and translation matrix from the offline template point cloud to the online guidance point cloud is calculated to determine the local deflection of the large-scale part in the camera coordinate system. After determining the local area deflection data of the large-scale part under each camera, the multiple local deflection information can be fused and calculated to obtain the overall positioning posture of the part, and thus guide the robotic arm to complete the grasping operation. This application can solve the problem that a single camera cannot meet the high-precision visual guidance requirements for large-scale parts.

[0031] The following is a complete example describing the steps of the high-precision vision guidance method based on multi-camera joint positioning:

[0032] In this example, first for each camera, such as Figure 2 As shown in FIG, a method for visual recognition and positioning of the surface of a local area of ​​a component without obvious features is performed. The main steps are as follows:

[0033] During the offline phase:

[0034] The robot arm with the gripper is brought close to the component, so that each local position of the component enters the field of view depth of each camera, and the camera collects local data;

[0035] The depth map data collected by the camera is converted into point cloud data through the camera internal parameters, and the noise in the point cloud is removed by the denoising algorithm, and the point cloud data of non-part parts is manually removed;

[0036] Calculate the center of mass of the point cloud data in the camera coordinate system, and use the principal component analysis algorithm to calculate the three principal axis vectors of the point cloud. Establish a coordinate system with the center of mass as the origin and the three principal axis vectors as the coordinate axes. After converting it to the robot base coordinate system through the camera external parameters, save the data.

[0037] During the online phase:

[0038] The robotic arm brings the gripper and camera close to the component, and brings each local area of ​​the component into the depth of field of the camera on the gripper, and the camera collects data;

[0039] The depth map of the online stage is converted into online point cloud data P through the camera internal parameter calculation;

[0040] Load the offline point cloud data Q and calculate the point in the online point cloud data P that is closest to each point in the offline point cloud data Q, that is, Where R is the rotation matrix of the point cloud, t is the translation vector, q is the point in the offline point cloud data, and the initial step is a 4x4 identity matrix;

[0041] According to the above steps, the relative relationship between the corresponding points of the offline point cloud and the online point cloud is calculated, and the SVD decomposition algorithm is used to solve Rt, that is, The online point cloud data P is transformed using the calculated Rt. Specifically, the transformation can be performed by using a point cloud registration algorithm such as ICP to calculate the pose relationship (rotation matrix R, translation vector t) between the point clouds Q and P. The point cloud P is a three-dimensional matrix composed of a stack of XYZ elements, and the transformation is based on R×P+t.

[0042] Since the above steps of solving Rt may cause the matching between point clouds to fall into the local optimal problem, that is, the matching effect is poor, the present invention introduces the Welsch function to add a weight to the distance between each point, that is, The weights of point pairs that are farther away are smaller, so they are not easily disturbed by point cloud noise, causing the above steps to fall into a local optimal solution;

[0043] Calculate the distance between each point in the transformed point cloud data P and the nearest point in the offline point cloud data Q to determine whether the offline point cloud and the online point cloud are accurately matched. If not, recalculate the relative relationship between the corresponding points of the offline point cloud and the online point cloud based on the above steps and repeat the above matching steps.

[0044] In this step, the accuracy of the match between the offline point cloud and the online point cloud can be determined by: if the point cloud distance value of each point is less than a threshold, it is considered converged. If the proportion of converged point clouds to the total number of point cloud data P is greater than a threshold (for example, 90%), the match is considered correct. This is only an example and is not limited to this.

[0045] According to the calculated relative relationship T(R, t) transformation matrix, hereinafter referred to as Rt rotation transformation matrix, the offline stored coordinate system is transformed to the newly captured point cloud, and this coordinate system is used as the centroid and axis of the online captured data;

[0046] Completed the local visual recognition and positioning algorithm for a single high-precision small-field-of-view camera;

[0047] By collecting local point cloud data of large-scale parts offline, and using the part data collected by the same camera during the identification and guidance phase to align it with the aforementioned offline local point cloud, and calculating the rotation and translation data from the offline point cloud to the point cloud collected during the guidance phase, the local deflection of the large-scale part in the camera coordinate system is determined. After determining the local area deflection data of a large-scale part captured by a high-precision, small-field-of-view camera, the local deflection information of the same part collected by multiple cameras in other locations can be fused and calculated to obtain the precise position of the entire part, which is used to guide the gripper to complete the guided grasping operation.

[0048] Through the offline and online steps described above, local features of large parts captured online can be precisely located within the coordinate system of a single camera. However, using positioning data from a single camera can easily lead to significant errors in the overall guidance of large parts. This is because even slight local angular deviations can cause significant deviations in parts of the large part that are farther away from this local area. Therefore, it is necessary to use multiple high-precision cameras to capture and locate large parts, combining this overall positioning information to achieve high-precision guidance of the entire large part.

[0049] Furthermore, a high-precision guidance method for joint positioning of data from multiple cameras is as follows:

[0050] As mentioned above, for high-precision grasping of large-sized parts, this application will use multiple high-precision 3D cameras with small fields of view to locate the features of the parts in the local area of ​​each camera, calculate the positioning pose information of the local area of ​​the parts, and convert this data into the robot base coordinate system based on the calibrated external parameter data. In order to achieve high-precision guided grasping, such as Figure 3 As shown in Figure 2, the high-precision guidance method for this feature fusion is also divided into two stages:

[0051] Offline stage:

[0052] Through the above steps, the positioning data of the local area photographed by each camera of the large-size component is calculated and obtained;

[0053] After storing the above data, the robot's flange end is manually taught so that the designed complex gripper can meet the set requirements when grasping large-sized parts. The set requirements are to be able to grasp large-sized parts stably and accurately, and the flange end position data at this time is recorded;

[0054] Based on the hand-eye calibration results of N cameras, the pose information of the local area of ​​the component captured by the N cameras is converted to the flange coordinate system of the end of the robot arm, and fused based on the least squares method to obtain the overall pose information of the component and save it;

[0055] Online stage:

[0056] Use each camera to capture a local area of ​​a large-scale component, and use the above-mentioned featureless component surface visual recognition and positioning algorithm to calculate the component pose information of the local area under each camera;

[0057] The same least squares method used in the offline phase is used to calculate the pose information of the entire large-scale component.

[0058] Calculate the transformation relationship Rt between the offline large-scale component pose information and the online large-scale component pose information, and use this Rt data to calculate the data information that the gripper needs to offset;

[0059] By completing the corresponding offset, the robot's end-guide gripper can accurately guide and position large-sized parts and complete the grasping action.

[0060] In summary, to address the need for high-precision vision-guided positioning of large-sized parts, this paper proposes a method for performing positioning feature recognition and multi-camera joint positioning using a high-precision camera with a small field of view in complex lighting scenarios. Using this method, we have successfully completed high-precision positioning and guided grasping of large parts such as front and rear doors, rear covers, front covers, and chassis in multiple automobile OEMs, with positioning guidance accuracy better than ±0.5mm.

[0061] The present application also provides a high-precision visual guidance system based on multi-camera joint positioning, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the above-mentioned high-precision visual guidance method based on multi-camera joint positioning are implemented.

[0062] This application also provides a computer device comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the high-precision visual guidance method based on multi-camera joint positioning. This computer device can implement each embodiment of the high-precision visual guidance method based on multi-camera joint positioning described above, and can achieve the same beneficial effects. A detailed description thereof is omitted here.

[0063] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A visual guidance method based on multi-camera joint positioning, characterized in that: include: S1: Obtain the local area of ​​the large-size component photographed by N cameras, and calculate the component pose information of the local area under each camera; S2: Calculate the pose information of the entire large-size component online based on the pose information of the component in the local area under each camera; S3: determining the transformation relationship between the large-size component pose information calculated offline and the large-size component pose information calculated online; S4: Calculate the required offset information of the end gripper according to the transformation relationship, and guide the robot arm to complete the corresponding offset according to the end gripper offset information to achieve grasping.

2. The visual guidance method based on multi-camera joint positioning according to claim 1 is characterized in that: The calculating of the component pose information of the local area under each camera includes: S11: For each camera, store its coordinate system in the offline stage, and convert the depth map data collected in the offline stage into offline point cloud data Q, and convert the depth map data collected in the online stage into online point cloud data P; S12: determining a relative relationship T(R, t) between corresponding points of the offline point cloud data Q and the online point cloud data P, and transforming the online point cloud data P based on the relative relationship T(R, t); S13: Introduce the Welsch function to add weights to the distance between each point; S14: Calculate the distance between each point in the transformed point cloud data P and the nearest point in the offline point cloud data Q, and determine whether the offline point cloud and the online point cloud match accurately. If not, repeat the above steps S12 to S14. S15: transforming the offline stored coordinate system into the newly captured point cloud according to the relative relationship T(R, t) transformation matrix, and using the coordinate system as the centroid and axis of the online captured data; S16: Calculate component pose information of the local area under the camera based on the centroid and axis of the online captured data.

3. The visual guidance method based on multi-camera joint positioning according to claim 2 is characterized in that: For each camera, the coordinate system of the camera in the offline stage is stored, and the depth map data collected in the offline stage is converted into offline point cloud data Q, including: In the offline stage, the robot arm with the gripper is brought close to the component, so that each local position of the component enters the field of view depth of each camera, and the camera collects local data; The depth map data collected by the camera is converted into point cloud data through the camera internal parameters, and the noise in the point cloud is removed by using the denoising algorithm, and the point cloud data of the non-part part is removed to obtain the offline point cloud data Q; Calculate the center of mass of the offline point cloud data Q in the camera coordinate system, and use the principal component analysis algorithm to calculate the three principal axis vectors of the point cloud. Establish a coordinate system with the center of mass as the origin and the three principal axis vectors as the coordinate axes, and then convert it to the robot base coordinate system through the camera external parameters and save the data.

4. The visual guidance method based on multi-camera joint positioning according to claim 2 is characterized in that: The determining of the relative relationship T(R, t) between the corresponding points of the offline point cloud data Q and the online point cloud data P comprises: Calculate the point in the online point cloud data P that is closest to each point in the offline point cloud data Q; The relative relationship T(R, t) between the corresponding points of the offline point cloud data Q and the online point cloud data P is determined according to the point with the shortest distance.

5. The visual guidance method based on multi-camera joint positioning according to claim 2 is characterized in that: Before S3, the method further includes: The pose information of large-scale components calculated in the offline stage.

6. The visual guidance method based on multi-camera joint positioning according to claim 2 is characterized in that: The large-size component pose information calculated in the offline stage includes: In the offline stage, the component pose information of the local area of ​​the large-size component captured by N cameras is obtained; The component pose information of the local area is stored, and the flange end of the robot is manually taught so that the end gripper meets the set requirements when grasping the large-sized component, and the flange end pose data at this time is recorded; Based on the hand-eye calibration results of N cameras, the pose information of the components in the local area captured by the N cameras is converted to the flange coordinate system at the end of the robot arm, and fused based on the least squares method to obtain the overall pose information of the components and save it.

7. A visual guidance system based on multi-camera joint positioning, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in claims 1 to 6 when executing the computer program.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images