Depth image splicing method based on five-axis platform position information

Through the depth image stitching method based on the position information of the five-axis platform, combined with 2D and 3D cameras, the 3D scanning trajectory is edited using the five-axis algorithm, which solves the problem of high contour occlusion and imaging quality dependence in the depth image stitching of mobile phone mid-frame products. Achieve efficient and fast splicing effect.

CN119067845APending Publication Date: 2024-12-03CHANGZHOU MINGSEAL ROBOT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155099.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art has problems of shadowing caused by high contour occlusion and high dependence on imaging quality and many preliminary preparations in the depth image stitching of mobile phone midframe products, resulting in low splicing efficiency and low accuracy.

Method used

Using the depth image stitching method based on the position information of the five-axis platform, the 3D scan trajectory is edited by the combination of 2D cameras and 3D cameras, and the 3D scan trajectory is set up along the edge of the product to eliminate the shadows caused by height occlusion, and achieve fast tracking. Edit and splice.

Benefits of technology

It achieves rapid and efficient shadow removal, reduces dependence on imaging quality, simplifies preliminary preparation, is suitable for products of different sizes and types, and improves splicing efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067845A_ABST
    Figure CN119067845A_ABST
Patent Text Reader

Abstract

The invention discloses a depth image splicing method based on five-axis platform position information. The depth image splicing method comprises the following steps: step 1, installing a 3D camera and a 2D camera; 2, correcting the 3D camera and the BC operation platform; step 3, calibrating the 2D camera; 4, calibrating the position relation between the 3D camera and the 2D camera; 5, setting a 3D camera scanning track through the 2D camera, namely editing the 3D scanning track through the 2D camera in combination with a five-axis algorithm; 6, scanning the depth image of each edge, and recording scanning starting and ending positions and a rotation angle; and 7, according to the position information, splicing each edge depth image to a corresponding position. According to the method, scanning track editing is rapidly realized, and shadow generated by height shielding is eliminated through end-to-end splicing; jigsaw track editing and algorithm adjustment can be quickly realized by replacing products with different sizes or types; the splicing algorithm has low requirements on imaging quality, and does not need excessive early-stage deployment and preparation work.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of depth image stitching methods, and in particular to a depth image stitching method based on position information of a five-axis platform. Background Art

[0002] When dispensing and inspecting the middle frame or screen of a mobile phone, it is often necessary to guide according to the change in the height of the product's dispensing benchmark, and at the same time, it is necessary to detect the width and height of the glue after dispensing. At present, the industry uses 3D cameras to collect product depth image information to achieve the above functions. Since conventional high-precision 3D cameras cannot achieve one-time panoramic imaging of middle frame or screen products, it is necessary to stitch the segmented collected depth images. The traditional depth image stitching processing methods mainly include the following two methods: ① The depth image stitching method based on network model training. The network model construction is complex and requires a lot of computing resources and time. In addition, a certain amount of training sets and test sets cannot be provided in an industrial environment, which leads to limited model generalization capabilities. ② The depth image stitching method based on overlapping area point cloud registration has certain requirements for the initial position of the registration. If the initial alignment deviation is large, it is easy to fall into the local optimum. Secondly, this method has certain requirements for the data imaging quality. The overlapping area point clouds obtained at different positions will be different, which will affect the registration effect and directly lead to poor stitching effect. The technical problems that need to be solved at this stage are:

[0003] 1. The shadow problem caused by the height occlusion of the contour when the line laser scans the depth image of the mobile phone mid-frame product line: The line laser based on the triangulation principle can quickly and accurately realize the product depth image acquisition. However, when scanning and imaging the mid-frame product, due to the height occlusion of the edge contour, there will be a measurement blind spot at the R corner position, resulting in the loss of glue dispensing groove information, and unable to realize 3D guided gluing or detection. Traditional segmented scanning is inefficient and cannot accurately reflect the location information of defects or abnormalities in the overall product.

[0004] 2. The traditional deep image stitching has high dependence on image quality, great product relevance, and a lot of preliminary preparation work: The traditional deep image stitching method based on feature information relies on the surface contour features of the product. When the features are not obvious or the image quality cannot be guaranteed, the stitching accuracy will be directly affected. In addition, the model trained based on the product or the affine relationship obtained by registration requires certain preparation work. When the workpiece is replaced, the machine adjustment efficiency is low and the switch cannot be completed quickly. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0006] To this end, the present invention proposes a depth image stitching method based on the position information of a five-axis platform, which stitches images along a specific scanning trajectory, including the following steps:

[0007] Step 1: Install the 3D camera and 2D camera: Install the 3D camera and 2D camera at the end of the Z2 axis, adjust the installation height of the 3D camera to the working range, make the 3D camera laser line perpendicular to the X2 axis, and calibrate the 3D camera laser line to be parallel to the BC work platform on the Y1 and Y2 axes;

[0008] The Z2 axis is the up and down sliding axis of the visual system;

[0009] The X2 axis is the left and right sliding axis of the visual system, or it is called the 3D camera scanning axis;

[0010] The Y1 axis is the front and rear sliding axis of the left workstation work platform;

[0011] The Y2 axis is the front and rear sliding axis of the right workstation work platform;

[0012] The BC work platform is a work platform that flips along the B axis and / or rotates along the C axis;

[0013] Step 2: Calibration of 3D camera and BC work platform: Scan the BC work platform with a 3D camera, calculate the parallelism between the B axis and the X2 axis, and compensate by retracting the B axis so that the flip surface of the B axis is parallel to the X2 axis;

[0014] The B axis is the flip axis around the Y axis;

[0015] The C axis is the rotation axis rotating around the Z axis;

[0016] Step 3: Calibrate the 2D camera: Calibrate the conversion relationship between the 2D camera pixel coordinates and the axis coordinate position

[0017] Step 4: Calibrate the position relationship between the 3D camera and the 2D camera: Based on the center feature of the tool setting instrument, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is calculated based on the 2D camera photo recognition and the 3D line scan depth image extraction;

[0018] Step 5: Set the 3D camera scanning trajectory through the 2D camera, that is, edit the 3D scanning trajectory through the 2D camera combined with the five-axis algorithm: edit the 3D scanning trajectory through the 2D camera image center, and convert it into the actual 3D scanning trajectory through the five-axis algorithm relationship;

[0019] Step 6: Scan the depth image of each edge, record the start and end positions of the scan, and the rotation angle: Scan the depth image of each edge of the product in turn, record the start and end positions of the scan and the C-axis rotation angle;

[0020] Step 7: Based on the position information, stitch each edge depth image to the corresponding position: Based on the above position information and pixel equivalent information, calculate the position of each edge depth image in the stitched image and call the stitching algorithm to stitch the end to end.

[0021] The beneficial effect of the present invention is that, based on the position deviation relationship between the 2D and 3D camera calibration, a rectangular trajectory is taught by a 2D camera, and combined with the five-axis platform algorithm, the 3D scanning trajectory is switched to the 3D camera to realize 3D scanning trajectory editing, that is, a 3D camera rectangular scanning trajectory is set along the edge of the product by a 2D camera, and four straight edges are scanned in sequence in a clockwise or counterclockwise direction, and the depth image of each edge is converted and spliced ​​to the corresponding position according to the start and end positions and rotation angle information of each edge scan, so as to quickly realize scanning trajectory editing, and at the same time calculate the splicing position based on the scanning trajectory position information, and eliminate the shadow caused by high occlusion by splicing the ends clockwise or counterclockwise; it has no product dependence: the replacement of products of different sizes or types can quickly realize puzzle trajectory editing and algorithm adjustment; the splicing algorithm has low requirements on imaging quality, and does not require too much preliminary deployment and preparation work.

[0022] According to one embodiment of the present invention, in step 1, the calibrated 3D camera laser line is parallel to the BC work platform on the Y1 and Y2 axes. The specific calibration method is: move the BC work platform into the working range of the 3D camera, calculate the height difference Δh between two points on the contour line through the 2.5D contour line measurement function, and adjust the installation angle by adding or removing shims at the upper and lower bolt holes of the 3D camera so that the height difference Δh between the two points on the contour line is close to zero.

[0023] According to one embodiment of the present invention, the B-axis is an axis that can be flipped around the Y-axis in a range of [-90°, 90°].

[0024] According to one embodiment of the present invention, the C-axis is an axis that can rotate around the Z-axis in a range of [-360°, 360°].

[0025] According to one embodiment of the present invention, in step 2, the specific steps of parallelism backoff compensation are:

[0026] Step 21: Scan the depth image of the BC work platform and select two points p1 (r1, c1) and p2 (r2, c2) in the same row direction on the image;

[0027] Step 22, obtain the height value h1 at p1, and obtain the height value h2 at p2;

[0028] Step 23: Given that the 3D depth image pixel equivalent is E_3d, the calculation formula for the fallback compensation angle a is:

[0029] a=arctan((h2-h1) / (c2-c1)*E_3d)

[0030] Step 24, repeat steps 21 to 23, take multiple groups of point pairs for calculation, calculate the retraction compensation angle set A = {a1, a2, a3...an}, where n is a positive integer greater than or equal to 1, and output the mean value of the retraction compensation angle set A = {a1, a2, a3...an} as the B-axis retraction compensation value and send it to the job control module of the software system.

[0031] According to one embodiment of the present invention, in step 3, the conversion relationship Describes the image coordinate system O i -X i Y i Pixel shifting distance to mechanical coordinates OX a Y a The relationship matrix between the axis movement spacing, the transformation relationship is rotation transformation first and then scaling transformation.

[0032] According to one embodiment of the present invention, in step 4, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is implemented by:

[0033] Step 41: The 2D camera captures the center of the tool setting instrument, extracts the pixel coordinates M (R_2d, C_2d) of the center point, and records the axis coordinates P_2d (X_2d, Y_2d, Z_2d) during the shooting. The Z_2d value is replaced by the Z1 axis coordinate when the point laser reads zero on the tool setting instrument. The Z1 axis is the up and down sliding axis of the dispensing device.

[0034] Step 42: The 3D camera scans the center of the tool setting instrument, and extracts the pixel coordinates N (R_3d, C_3d) of the center of the tool setting instrument in the depth map, extracts the height value G_3d according to the position of point N, and records the axis coordinates P_3d (X_3d, Y_3d, Z_3d) of the scanning start position;

[0035] Step 43: Taking the upper left corner of the 3D depth image as the origin, the calculation formula of the axis coordinate P0_3d (X0_3d, Y0_3d, Z0_3d) when the center of the tool setting instrument is aligned to the origin under the 3D depth image is:

[0036]

[0037] Where E_3d is the 3D depth image pixel equivalent;

[0038] Step 44: Taking the center of the 2D image as the origin, the calculation formula of the axis coordinate P0_2d (X0_2d, Y0_2d, Z0_2d) when the center of the tool setting instrument is aligned to the origin under the 2D image is:

[0039]

[0040] Among them, the meanings of each symbol in the formula are:

[0041] E_2d is the 2D image pixel equivalent;

[0042] Width is the 2D image width pixel value;

[0043] Height is the high pixel value of the 2D image;

[0044] Step 45: The position deviation relationship between 3D and 2D in the axis coordinate system is described as: P_offset(X_offset, Y_offset, Z_offset), and the calculation formula is:

[0045]

[0046] According to one embodiment of the present invention, in step 5, the steps of editing the 3D scanning trajectory by combining a 2D camera with a five-axis algorithm are as follows:

[0047] Step 51: edit a rectangular track in a clockwise direction along the outer edge of the rectangular product through the center of the 2D camera image, where T, U, V, and W are the coordinates of the four vertices of the rectangular track under the 2D camera. Since the center of the 2D image is aligned with the upper left position of the 3D depth image, in order to ensure that the 3D scan can scan the edge of the rectangular product, the 2D camera image determines the outer expansion distance d when determining the scanning track, and the range of d is 0<d<L, where L is the length of the laser contour line when scanning;

[0048] Step 52: Determine the 3D scanning trajectory based on the 2D camera, set the coordinates of point T to (Xt, Yt, Zt, Bt, Ct), switch the 3D camera to point T according to the relationship between the 3D and 2D position deviations, set the 3D camera position at this time to the first edge scanning starting point Ts (Xts, Yts, Zts, Uts, Vts), and translate along the positive direction of the X2 axis by a distance L1 to the first edge scanning ending point Te (Xte, Yte, Zte, Ute, Vte), where L1 is the length of the rectangular trajectory;

[0049] Step 53: Assume that the coordinates of point U are (Xu, Yu, Zu, Bu, Cu), and the coordinates of the rotation center under the 2D camera are R_2D (Xc, Yc). Then the coordinates of point U after rotating 90 degrees counterclockwise around the rotation center R_2D are U' (Xu', Yu', Zu', Bu', Cu'), and the calculation formula is:

[0050]

[0051] Wherein, Phi is the arc value of 180 degrees;

[0052] Switch the 3D camera to the U' position and record the position of the 3D camera at this time as the second edge scan starting point Us (Xus, Yus, Zus, Uus, Vus), and translate along the positive direction of the X2 axis by a distance of L2 to the second edge scan end point Ue (Xue, Yue, Zue, Uue, Vue), where L2 is the width of the rectangular track;

[0053] Step 54: Let the coordinates of point V be (Xv, Yv, Zv, Bv, Cv), and calculate the coordinates of point V after rotating 180 degrees counterclockwise around the rotation center R_2D to be V'(Xv', Yv', Zv', Bv', Cv') according to the formula. Switch the 3D camera to the V' position and record the position of the 3D camera at this time as the starting point Vs (Xvs, Yvs, Zvs, Uvs, Vvs) of the third edge scan, and translate along the positive direction of the X2 axis by a distance of L1 to the end point Ve (Xve, Yve, Zve, Uve, Vve) of the third edge scan, where L1 is the length of the rectangular trajectory, and the calculation formula is:

[0054]

[0055] Step 56: Assume the coordinates of point W to be (Xw, Yw, Zw, Bw, Cw), and calculate the coordinates of point W after rotating 270 degrees counterclockwise around the rotation center R_2D to be W' (Xw', Yw', Zw', Bw', Cw') according to the formula. Switch the 3D camera to the W' position and record the position of the 3D camera at this time as the starting point of the fourth edge scan Ws (Xws, Yws, Zws, Uws, Vws), and translate along the positive direction of the X2 axis by a distance of L2 to the end point of the fourth edge scan We (Xwe, Ywe, Zwe, Uwe, Vwe), where L2 is the width of the rectangular track, and the calculation formula is:

[0056]

[0057] According to one embodiment of the present invention, in step 6, the depth image set of each edge is defined as M = {m0, m1, m2, m3}, the scanning start position set is defined as S = {s0, s1, s2, s3}, the end position set is E = {e0, e1, e2, e3}, the C-axis rotation angle set is J = {j0, j1, j2, j3}, the width of the spliced ​​depth image is W_tiled = ceil((e0-s0) / E_3d), and the height of the spliced ​​depth image is H_tiled = ceil((e1-s1) / E_3d), wherein the ceil() function represents rounding up; E_3d is the 3D depth image pixel equivalent; the position of each edge depth image in the spliced ​​image is calculated based on the position and pixel equivalent information, which is defined as the pixel coordinates of the upper left corner of each edge depth image in the spliced ​​image after rotation and translation transformation.

[0058] According to an embodiment of the present invention, the position of each edge depth image in the spliced ​​image is calculated based on the position and pixel equivalent information, and the specific calculation steps are as follows:

[0059] Step 61, calling the rotate_image(-90*i,"constant") operator to rotate the depth image set, the depth image set after rotation is Mr = {m0_r, m1_r, m2_r, m3_r}, where i is the subscript number of the depth image before the rotation transformation;

[0060] Step 62, by default, the upper left corner of the first edge depth image is aligned with the upper left corner of the stitched image, and the coordinates of m0_r in the stitched image are p0_r=(0,0);

[0061] Step 63, the image coordinates of the second edge after stitching p1_r=(0, (e0-s0) / E_3d-Width_m1_r), where Width_m1_r is the pixel value of the width of the m1_r image;

[0062] Step 64: The image coordinates of the third edge after splicing are p2_r=((e1-s1) / E_3d-Height_m2_r, 0), where Height_m2_r is the height pixel value of the m2_r image;

[0063] Step 65, the image coordinates of the fourth edge after splicing p3_r=(Height_m0_r, 0), where Height_m0_r is the height pixel value of the m0_r image;

[0064] Step 66: After the fourth edge rotation operation, the depth image m3_r is cropped along the row direction of the pixel coordinates. The cropping range in the row direction is [0, Height_m0_r], where Height_m0_r is the height pixel value of the m0_r image.

[0065] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 is a flow chart of the present invention;

[0069] Figure 2 It is the position relationship diagram between the axes of the five-axis platform;

[0070] Figure 3 This is a schematic diagram of a 2D camera editing rectangular scanning trajectory;

[0071] Figure 4 It is a schematic diagram of the position relationship of the upper left corner of the transformed depth image in the spliced ​​image;

[0072] Figure 5 It is the real object depth map of the shadow-occluded area before stitching;

[0073] Figure 6 This is the effect diagram of eliminating the shadow occlusion area after splicing.

[0074] The numbers in the figure are:

[0075] X1, left and right sliding axis of dispensing device;

[0076] X2, left and right sliding axis of the visual system;

[0077] Y1, front and rear sliding axis of the left work platform;

[0078] Y2, front and rear sliding axis of the right workstation operating platform;

[0079] Z1, up and down sliding axis of dispensing device;

[0080] Z2, up and down sliding axis of the visual system;

[0081] B1, an axis that can be flipped 90 degrees left and right around the Y1 axis;

[0082] B2, an axis that can be flipped 90 degrees left and right around the Y2 axis;

[0083] C1, an axis that can rotate 360 ​​degrees around the Z1 axis;

[0084] C2, an axis that can rotate 360 ​​degrees around the Z2 axis;

[0085] p0_r, the coordinates of the upper left corner of the first edge in the stitched image;

[0086] p1_r, the coordinates of the upper left corner of the second edge in the stitched image;

[0087] p2_r, the coordinates of the upper left corner of the third edge in the stitched image;

[0088] p3_r, the coordinates of the upper left corner of the fourth edge in the stitched image. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] In the description of the present invention, it is necessary to understand that the orientations or positional relationships indicated by the terms “one side”, “the other side”, “both sides”, “between”, “middle”, “upper end”, “lower end”, etc. are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0091] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection, and can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0092] The depth image stitching method based on the position information of the five-axis platform according to an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0093] See Figure 1 The depth image stitching method based on the position information of the five-axis platform of the present invention is to stitch along a specific scanning trajectory, where the specific scanning trajectory refers to a rectangular trajectory taught clockwise or counterclockwise along the outer edge of the product by a 2D camera, and a 3D camera scanning trajectory converted based on a five-axis algorithm and a position deviation relationship between a 2D camera and a 3D camera. The depth image stitching method based on the position information of the five-axis platform includes the following steps:

[0094] Step 1, install the 3D camera and 2D camera: install the 3D camera and 2D camera at the end of the Z2 axis, adjust the installation height of the 3D camera to the working range, make the 3D camera laser line perpendicular to the X2 axis, and calibrate the 3D camera laser line to be parallel to the BC work platform on the Y1 and Y2 axes. Among them, the Z2 axis is the up and down sliding axis of the visual system; the X2 axis is the left and right sliding axis of the visual system, or the 3D camera scanning axis; the Y1 axis is the front and back sliding axis of the left station work platform; the Y2 axis is the front and back sliding axis of the right station work platform; the BC work platform is a work platform that flips along the B axis and / or rotates along the C axis.

[0095] As attached Figure 2 As shown, in the first step, the 3D camera laser line is calibrated to be parallel to the BC work platform on the Y1 and Y2 axes. The specific calibration method is: move the BC work platform into the working range of the 3D camera, calculate the height difference Δh between two points on the contour line through the 2.5D contour line measurement function, and adjust the installation angle by adding or removing shims at the upper and lower bolt holes of the 3D camera so that the height difference Δh between the two points on the contour line is close to zero.

[0096] Step 2: Calibration of 3D camera and BC work platform: Scan the BC work platform with a 3D camera, calculate the parallelism between the B axis and the X2 axis, and compensate by retracting the B axis so that the flip surface of the B axis is parallel to the X2 axis; the B axis is the flip axis flipped around the Y axis; the C axis is the rotation axis rotated around the Z axis; specifically, the B axis is an axis that can flip around the Y axis in the range of [-90°, 90°]. The C axis is an axis that can rotate around the Z axis in the range of [-360°, 360°].

[0097] In step 2, the specific steps of parallelism backoff compensation are:

[0098] Step 21: Scan the depth image of the BC work platform and select two points p1 (r1, c1) and p2 (r2, c2) in the same row direction on the image;

[0099] Step 22, obtain the height value h1 at p1, and obtain the height value h2 at p2;

[0100] Step 23: Given that the 3D depth image pixel equivalent is E_3d, the calculation formula for the fallback compensation angle a is:

[0101] a=arctan((h2-h1) / (c2-c1)* E_3d) (1)

[0102] Step 24, repeat steps 21 to 23, take multiple groups of point pairs for calculation, calculate the retraction compensation angle set A = {a1, a2, a3...an}, where n is a positive integer greater than or equal to 1, and output the mean value of the retraction compensation angle set A = {a1, a2, a3...an} as the B-axis retraction compensation value and send it to the job control module of the software system.

[0103] Step 3: Calibrate the 2D camera: Calibrate the conversion relationship between the 2D camera pixel coordinates and the axis coordinate position

[0104] See Figure 2 , X1 represents the left and right sliding axis of the dispensing device; X1 can also be defined as the front slide rail of the horizontal beam; X2 represents the left and right sliding axis of the visual system; X2 can also be defined as the rear slide rail of the horizontal beam; Y1 represents the front and rear sliding axis of the left workstation work platform; Y1 can also be defined as the left slide rail horizontally perpendicular to the X1 axis; Y2 represents the front and rear sliding axis of the right workstation work platform; Y2 can also be defined as the right slide rail horizontally perpendicular to the X1 axis; Z1 represents the up and down sliding axis of the dispensing device; Z1 can also be defined as the front slide rail vertically perpendicular to the X1 axis; Z2 represents the up and down sliding axis of the visual system; Z2 can also be defined as a rear slide rail that is perpendicular to the X2 axis; B1 represents the axis that can be flipped 90 degrees left and right around the Y1 axis; B1 can also be defined as the left workstation flip axis; B2 represents the axis that can be flipped 90 degrees left and right around the Y2 axis; B2 can also be defined as the right workstation flip axis; C1 represents the axis that can rotate 360 ​​degrees around the Z1 axis; C1 can also be defined as the left workstation rotation axis; C2 represents the axis that can rotate 360 ​​degrees around the Z2 axis; C2 can also be defined as the right workstation rotation axis.

[0105] In step 3, the transformation relationship Describes the image coordinate system O i -X i Y i Pixel shifting distance to mechanical coordinates OX a Y a The relationship matrix between the axis movement spacing, the transformation relationship is rotation transformation first and then scaling transformation:

[0106]

[0107] The meanings of the various symbols in formula (2) are:

[0108] x a Indicates the moving axis spacing of the machine in the X direction;

[0109] y a Indicates the moving axis spacing of the machine in the Y direction;

[0110] sx Represents X i Axis in X a scale factor on the axis;

[0111] s y Represents Y i Axis in Y a scale factor on the axis;

[0112] x i Indicates the moving pixel spacing of the image in the horizontal direction;

[0113] y i Indicates the moving pixel spacing in the vertical direction of the image;

[0114] S represents the scaling matrix

[0115] R represents the rotation matrix θ is the rotation angle.

[0116] Step 4: Calibrate the position relationship between the 3D camera and the 2D camera: Based on the center feature of the tool setting instrument, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is calculated based on the 2D camera photo recognition and the 3D line scan depth image extraction;

[0117] In step 4, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is implemented as follows:

[0118] Step 41: The 2D camera captures the center of the tool setting instrument, extracts the pixel coordinates M (R_2d, C_2d) of the center point, and records the axis coordinates P_2d (X_2d, Y_2d, Z_2d) during the shooting. The Z_2d value is replaced by the Z1 axis coordinate when the point laser reads zero on the tool setting instrument. The Z1 axis is the up and down sliding axis of the dispensing device.

[0119] Step 42: The 3D camera scans the center of the tool setting instrument, and extracts the pixel coordinates N (R_3d, C_3d) of the center of the tool setting instrument in the depth map, extracts the height value G_3d according to the position of point N, and records the axis coordinates P_3d (X_3d, Y_3d, Z_3d) of the scanning start position;

[0120] Step 43: Taking the upper left corner of the 3D depth image as the origin, the calculation formula of the axis coordinate P0_3d (X0_3d, Y0_3d, Z0_3d) when the center of the tool setting instrument is aligned to the origin under the 3D depth image is:

[0121]

[0122] Where E_3d is the 3D depth image pixel equivalent;

[0123] Step 44: Taking the center of the 2D image as the origin, the calculation formula of the axis coordinate P0_2d (X0_2d, Y0_2d, Z0_2d) when the center of the tool setting instrument is aligned to the origin under the 2D image is:

[0124]

[0125] Among them, the meanings of each symbol in formula (4) are:

[0126] E_2d is the 2D image pixel equivalent;

[0127] Width is the 2D image width pixel value;

[0128] Height is the high pixel value of the 2D image;

[0129] Step 45: The position deviation relationship between 3D and 2D in the axis coordinate system is described as: P_offset(X_offset, Y_offset, Z_offset), and the calculation formula is:

[0130]

[0131] Step 5: Set the 3D camera scanning trajectory through the 2D camera, that is, edit the 3D scanning trajectory through the 2D camera combined with the five-axis algorithm: edit the 3D scanning trajectory through the 2D camera image center, and convert it into the actual 3D scanning trajectory through the five-axis algorithm relationship;

[0132] In step 5, the 3D scanning trajectory is edited by combining the 2D camera with the five-axis algorithm. The steps are as follows:

[0133] Step 51, as attached Figure 3 As shown, a rectangular track is edited clockwise along the outer edge of a rectangular product through the center of the 2D camera image, where T, U, V, and W are the coordinates of the four vertices of the rectangular track under the 2D camera, and the track editing order is shown by the arrow. Since the center of the 2D image is aligned with the upper left position of the 3D depth image, in order to ensure that the 3D scan can scan the edge of the rectangular product, the 2D camera image should be appropriately expanded by a certain distance d when determining the scanning track, and the range of d is 0<d<L, where L is the length of the laser contour line when scanning;

[0134] Step 52, as attached Figure 3As shown, the 3D scanning trajectory is determined based on the 2D camera, and the coordinates of point T are set to (Xt, Yt, Zt, Bt, Ct). According to the relationship between the 3D and 2D position deviations, the 3D camera is switched to point T and the position of the 3D camera at this time is set to the first edge scanning starting point Ts (Xts, Yts, Zts, Uts, Vts), and the 3D camera is translated along the positive direction of the X2 axis by a distance L1 to the first edge scanning end point Te (Xte, Yte, Zte, Ute, Vte), where L1 is the length of the rectangular trajectory;

[0135] Step 53: Assume that the coordinates of point U are (Xu, Yu, Zu, Bu, Cu), and the coordinates of the rotation center under the 2D camera are R_2D (Xc, Yc). Then the coordinates of point U after rotating 90 degrees counterclockwise around the rotation center R_2D are U' (Xu', Yu', Zu', Bu', Cu'), and the calculation formula is:

[0136]

[0137] Wherein, Phi is the arc value of 180 degrees;

[0138] Switch the 3D camera to the U' position and record the position of the 3D camera at this time as the second edge scan starting point Us (Xus, Yus, Zus, Uus, Vus), and translate along the positive direction of the X2 axis by a distance of L2 to the second edge scan end point Ue (Xue, Yue, Zue, Uue, Vue), where L2 is the width of the rectangular track;

[0139] Step 54: Let the coordinates of point V be (Xv, Yv, Zv, Bv, Cv), calculate the coordinates of point V after rotating 180 degrees counterclockwise around the rotation center R_2D according to formula (7) to be V'(Xv', Yv', Zv', Bv', Cv'), switch the 3D camera to the V' position and record the position of the 3D camera at this time as the starting point Vs (Xvs, Yvs, Zvs, Uvs, Vvs) of the third edge scan, and translate along the positive direction of the X2 axis by a distance of L1 to the end point Ve (Xve, Yve, Zve, Uve, Vve) of the third edge scan, where L1 is the length of the rectangular trajectory;

[0140]

[0141] Step 56: Assume the coordinates of point W to be (Xw, Yw, Zw, Bw, Cw), calculate the coordinates of point W after rotating 270 degrees counterclockwise around the rotation center R_2D to be W'(Xw', Yw', Zw', Bw', Cw') according to formula (8), switch the 3D camera to the W' position and record the position of the 3D camera at this time as the starting point of the fourth edge scan Ws (Xws, Yws, Zws, Uws, Vws), translate along the positive direction of the X2 axis by a distance of L2 to the end point of the fourth edge scan We (Xwe, Ywe, Zwe, Uwe, Vwe), where L2 is the width of the rectangular track;

[0142]

[0143] Step 6: Scan the depth image of each edge, record the start and end positions of the scan, and the rotation angle: Scan the depth image of each edge of the product in turn, record the start and end positions of the scan and the C-axis rotation angle;

[0144] In step 6, the depth image set of each edge is defined as M = {m0, m1, m2, m3}, the scanning start position set is defined as S = {s0, s1, s2, s3}, the end position set is E = {e0, e1, e2, e3}, the C-axis rotation angle set is J = {j0, j1, j2, j3}, the width of the spliced ​​depth image is W_tiled = ceil((e0-s0) / E_3d), and the height of the spliced ​​depth image is H_tiled = ceil((e1-s1) / E_3d), where the ceil() function represents rounding up; E_3d is the 3D depth image pixel equivalent; the position of each edge depth image in the spliced ​​image is calculated based on the position and pixel equivalent information, which is defined as the pixel coordinates of the upper left corner of the depth image of each edge after rotation and translation transformation in the spliced ​​image. The position of each edge depth image in the stitched image is calculated based on the position and pixel equivalent information, as shown in the attached figure. Figure 4 As shown, the specific calculation steps are as follows:

[0145] Step 61, call the rotate_image(-90*i,"constant") operator to rotate the depth image set. The depth image set after rotation is Mr = {m0_r, m1_r, m2_r, m3_r}, where i is the subscript number of the depth image before the rotation transformation. Specifically, m0_r represents the depth image after the first edge rotation operation; m1_r represents the depth image after the second edge rotation operation; m2_r represents the depth image after the third edge rotation operation; m3_r represents the depth image after the fourth edge rotation operation.

[0146] The concatenation operator is as follows:

[0147] tile_images_offset(images,tiledImage,[0,0,(e1-s1) / E_3d-Height_m2_r,Height_m0_r],\

[0148] [0,(e0-s0) / E_3d-Width_m1_r,0,0],\

[0149] [0,0,0,Height_m0_r],\

[0150] [0,0,0,0],\

[0151] [Height_m0_r,Height_m1_r,Height_m2_r,Height_m3_r],\

[0152] [Width_m0_r,Width_m1_r,Width_m2_r,Width_m3_r],\

[0153] W_tiled,H_tiled)

[0154] in,

[0155] images is the rotated depth image set Mr;

[0156] tiledImage is the output of the stitched depth image;

[0157] (e1-s1) / E_3d-Height_m2_r is the column coordinate of p1_r;

[0158] (e1-s1) / E_3d-Height_m2_r is the row coordinate of p2_r.

[0159] Step 62, by default, the upper left corner of the first edge depth image is aligned with the upper left corner of the stitched image, and the coordinates of m0_r in the stitched image are p0_r=(0,0);

[0160] Step 63, the image coordinates of the second edge after stitching p1_r=(0, (e0-s0) / E_3d-Width_m1_r), where Width_m1_r is the pixel value of the width of the m1_r image;

[0161] Step 64: The image coordinates of the third edge after splicing are p2_r=((e1-s1) / E_3d-Height_m2_r, 0), where Height_m2_r is the height pixel value of the m2_r image;

[0162] Step 65, the image coordinates of the fourth edge after splicing p3_r=(Height_m0_r, 0), where Height_m0_r is the height pixel value of the m0_r image;

[0163] Step 66: To prevent the shadow of the fourth edge from covering the normal outline of the first edge, the depth image m3_r is cropped along the row direction of the pixel coordinates after the rotation operation of the fourth edge. The cropping range in the row direction is [0, Height_m0_r], where Height_m0_r is the height pixel value of the m0_r image.

[0164] Step 7: Based on the position information, stitch each edge depth image to the corresponding position: Based on the above position information and pixel equivalent information, calculate the position of each edge depth image in the stitched image and call the stitching algorithm to stitch the end to end.

[0165] As attached Figure 5 As shown, the area in the wide rectangular frame is the normal collection area of ​​the contour feature, and the area in the narrow rectangular frame is the shadow area caused by the occlusion of the contour feature. The arrow indicates the shadow area of ​​the contour information occlusion, and the tail of the arrow indicates the normal area of ​​the contour information. The tail of the arrow points to the arrow, indicating that the normal area of ​​the contour information covers the shadow area of ​​the contour information occlusion after splicing, thereby achieving the purpose of eliminating the shadow caused by occlusion. According to the above-obtained image coordinates of each edge after splicing, the head and tail are spliced. After splicing, the area in the wide rectangular frame can cover the area in the narrow rectangular frame, thereby eliminating the occlusion shadow problem caused by the laser triangulation imaging principle. The effect after splicing is shown in the attached figure. Figure 6 shown.

[0166] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A depth image stitching method based on five-axis platform position information, characterized in that: Jigsaw puzzles along a specific scanning trajectory, including the following steps: Step 1: Install the 3D camera and 2D camera: Install the 3D camera and 2D camera at the end of the Z2 axis, adjust the installation height of the 3D camera to the working range, make the 3D camera laser line perpendicular to the X2 axis, and calibrate the 3D camera laser line to be parallel to the BC work platform on the Y1 and Y2 axes; The Z2 axis is the up and down sliding axis of the visual system; The X2 axis is the left and right sliding axis of the visual system, or it is called the 3D camera scanning axis; The Y1 axis is the front and rear sliding axis of the left workstation work platform; The Y2 axis is the front and rear sliding axis of the right workstation work platform; The BC work platform is a work platform that flips along the B axis and / or rotates along the C axis; Step 2: Calibration of 3D camera and BC work platform: Scan the BC work platform with a 3D camera, calculate the parallelism between the B axis and the X2 axis, and compensate by retracting the B axis so that the flip surface of the B axis is parallel to the X2 axis; The B axis is the flip axis around the Y axis; The C axis is the rotation axis rotating around the Z axis; Step 3: Calibrate the 2D camera: Calibrate the conversion relationship between the 2D camera pixel coordinates and the axis coordinate position Step 4: Calibrate the position relationship between the 3D camera and the 2D camera: Based on the center feature of the tool setting instrument, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is calculated based on the 2D camera photo recognition and the 3D line scan depth image extraction; Step 5: Set the 3D camera scanning trajectory through the 2D camera, that is, edit the 3D scanning trajectory through the 2D camera combined with the five-axis algorithm: edit the 3D scanning trajectory through the 2D camera image center, and convert it into the actual 3D scanning trajectory through the five-axis algorithm relationship; Step 6: Scan the depth image of each edge, record the start and end positions of the scan, and the rotation angle: Scan the depth image of each edge of the product in turn, record the start and end positions of the scan and the C-axis rotation angle; Step 7: Based on the position information, stitch each edge depth image to the corresponding position: Based on the above position information and pixel equivalent information, calculate the position of each edge depth image in the stitched image and call the stitching algorithm to stitch the end to end.

2. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: In the first step, the calibrated 3D camera laser line is parallel to the BC work platform on the Y1 and Y2 axes. The specific calibration method is: move the BC work platform into the working range of the 3D camera, calculate the height difference Δh between two points on the contour line through the 2.5D contour line measurement function, and adjust the installation angle by adding or removing shims at the upper and lower bolt holes of the 3D camera so that the height difference Δh between the two points on the contour line is close to zero.

3. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: The B-axis is an axis that can be flipped around the Y-axis in the range of [-90°, 90°].

4. The depth image stitching method based on the five-axis platform position information according to claim 1, characterized in that: The C-axis is an axis that can rotate around the Z-axis in the range of [-360°, 360°].

5. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: In step 2, the specific steps of parallelism backoff compensation are: Step 21: Scan the depth image of the BC work platform and select two points p1 (r1, c1) and p2 (r2, c2) in the same row direction on the image; Step 22, obtain the height value h1 at p1, and obtain the height value h2 at p2; Step 23: Given that the 3D depth image pixel equivalent is E_3d, the calculation formula for the fallback compensation angle a is: a=arctan((h2-h1) / (c2-c1)*E_3d) Step 24: Repeat steps 21 to 23, take multiple sets of point pairs for calculation, and calculate the set of backoff compensation angles A = {a1, a2, a3...an}, where: n is a positive integer greater than or equal to 1, and the average value of the output retraction compensation angle set A = {a1, a2, a3...an} is sent to the job control module of the software system as the B-axis retraction compensation value.

6. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: In step 3, the conversion relationship Describes the image coordinate system O i -X i Y i Pixel shifting distance to mechanical coordinates OX a Y a The relationship matrix between the axis movement spacing, the transformation relationship is rotation transformation first and then scaling transformation.

7. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: In step 4, the position deviation relationship between the 3D camera and the 2D camera in the axis coordinates is implemented by: Step 41: The 2D camera captures the center of the tool setting instrument, extracts the pixel coordinates M (R_2d, C_2d) of the center point, and records the axis coordinates P_2d (X_2d, Y_2d, Z_2d) during the shooting. The Z_2d value is replaced by the Z1 axis coordinate when the point laser reads zero on the tool setting instrument. The Z1 axis is the up and down sliding axis of the dispensing device. Step 42: The 3D camera scans the center of the tool setting instrument, and extracts the pixel coordinates N (R_3d, C_3d) of the center of the tool setting instrument in the depth map, extracts the height value G_3d according to the position of point N, and records the axis coordinates P_3d (X_3d, Y_3d, Z_3d) of the scanning start position; Step 43: Taking the upper left corner of the 3D depth image as the origin, the calculation formula of the axis coordinate P0_3d (X0_3d, Y0_3d, Z0_3d) when the center of the tool setting instrument is aligned to the origin under the 3D depth image is: Where E_3d is the 3D depth image pixel equivalent; Step 44: Taking the center of the 2D image as the origin, the calculation formula of the axis coordinate P0_2d (X0_2d, Y0_2d, Z0_2d) when the center of the tool setting instrument is aligned to the origin under the 2D image is: Among them, the meanings of each symbol in the formula are: E_2d is the 2D image pixel equivalent; Width is the 2D image width pixel value; Height is the high pixel value of the 2D image; Step 45: The position deviation relationship between 3D and 2D in the axis coordinate system is described as: P_offset(X_offset, Y_offset, Z_offset), and the calculation formula is:

8. The depth image stitching method based on the five-axis platform position information according to claim 1 is characterized in that: In step 5, the 3D scanning trajectory is edited by combining a 2D camera with a five-axis algorithm, and the steps are as follows: Step 51: edit a rectangular track in a clockwise direction along the outer edge of the rectangular product through the center of the 2D camera image, where T, U, V, and W are the coordinates of the four vertices of the rectangular track under the 2D camera. Since the center of the 2D image is aligned with the upper left position of the 3D depth image, in order to ensure that the 3D scan can scan the edge of the rectangular product, the 2D camera image determines the outer expansion distance d when determining the scanning track, and the range of d is 0<d<L, where L is the length of the laser contour line when scanning; Step 52: Determine the 3D scanning trajectory based on the 2D camera, set the coordinates of point T to (Xt, Yt, Zt, Bt, Ct), switch the 3D camera to point T according to the relationship between the 3D and 2D position deviations, set the 3D camera position at this time to the first edge scanning starting point Ts (Xts, Yts, Zts, Uts, Vts), and translate along the positive direction of the X2 axis by a distance L1 to the first edge scanning ending point Te (Xte, Yte, Zte, Ute, Vte), where L1 is the length of the rectangular trajectory; Step 53: Assume that the coordinates of point U are (Xu, Yu, Zu, Bu, Cu), and the coordinates of the rotation center under the 2D camera are R_2D (Xc, Yc). Then the coordinates of point U after rotating 90 degrees counterclockwise around the rotation center R_2D are U' (Xu', Yu', Zu', Bu', Cu'), and the calculation formula is: Wherein, Phi is the arc value of 180 degrees; Switch the 3D camera to the U' position and record the position of the 3D camera at this time as the second edge scan starting point Us (Xus, Yus, Zus, Uus, Vus), and translate along the positive direction of the X2 axis by a distance of L2 to the second edge scan end point Ue (Xue, Yue, Zue, Uue, Vue), where L2 is the width of the rectangular track; Step 54: Let the coordinates of point V be (Xv, Yv, Zv, Bv, Cv), and calculate the coordinates of point V after rotating 180 degrees counterclockwise around the rotation center R_2D to be V'(Xv', Yv', Zv', Bv', Cv') according to the formula. Switch the 3D camera to the V' position and record the position of the 3D camera at this time as the starting point Vs (Xvs, Yvs, Zvs, Uvs, Vvs) of the third edge scan, and translate along the positive direction of the X2 axis by a distance of L1 to the end point Ve (Xve, Yve, Zve, Uve, Vve) of the third edge scan, where L1 is the length of the rectangular trajectory, and the calculation formula is: Step 56: Assume the coordinates of point W to be (Xw, Yw, Zw, Bw, Cw), and calculate the coordinates of point W after rotating 270 degrees counterclockwise around the rotation center R_2D to be W' (Xw', Yw', Zw', Bw', Cw') according to the formula. Switch the 3D camera to the W' position and record the position of the 3D camera at this time as the starting point of the fourth edge scan Ws (Xws, Yws, Zws, Uws, Vws), and translate along the positive direction of the X2 axis by a distance of L2 to the end point of the fourth edge scan We (Xwe, Ywe, Zwe, Uwe, Vwe), where L2 is the width of the rectangular track, and the calculation formula is:

9. The depth image stitching method based on the five-axis platform position information according to claim 1, characterized in that: In step 6, the depth image set of each edge is defined as M = {m0, m1, m2, m3}, the scanning start position set is defined as S = {s0, s1, s2, s3}, the end position set is E = {e0, e1, e2, e3}, the C-axis rotation angle set is J = {j0, j1, j2, j3}, the width of the spliced ​​depth image is W_tiled = ceil((e0-s0) / E_3d), and the height of the spliced ​​depth image is H_tiled = ceil((e1-s1) / E_3d), where the ceil() function represents rounding up; E_3d is the 3D depth image pixel equivalent; the position of each edge depth image in the spliced ​​image is calculated based on the position and pixel equivalent information, which is defined as the pixel coordinates of the upper left corner of the depth image of each edge after rotation and translation transformation in the spliced ​​image.

10. The depth image stitching method based on five-axis platform position information according to claim 7, characterized in that: The position of each edge depth image in the stitched image is calculated based on the position and pixel equivalent information. The specific calculation steps are as follows: Step 61: Call the rotate_image(-90*i,"constant") operator to rotate the depth image set. After rotation, the depth image set is Mr = {m0_r, m1_r, m2_r, m3_r}, where: i is the subscript number of the depth image before the rotation transformation; Step 62, by default, the upper left corner of the first edge depth image is aligned with the upper left corner of the stitched image, and the coordinates of m0_r in the stitched image are p0_r=(0,0); Step 63, the image coordinates of the second edge after stitching p1_r=(0, (e0-s0) / E_3d-Width_m1_r), where Width_m1_r is the pixel value of the width of the m1_r image; Step 64: The image coordinates of the third edge after splicing are p2_r=((e1-s1) / E_3d-Height_m2_r, 0), where Height_m2_r is the height pixel value of the m2_r image; Step 65, the image coordinates of the fourth edge after splicing p3_r=(Height_m0_r, 0), where Height_m0_r is the height pixel value of the m0_r image; Step 66: After the fourth edge rotation operation, the depth image m3_r is cropped along the row direction of the pixel coordinates. The cropping range in the row direction is [0, Height_m0_r], where Height_m0_r is the height pixel value of the m0_r image.