Augmented reality projection method based on annular coding mark points

Abstract

The invention provides an augmented reality projection method based on annular coding mark points, and the method comprises the steps: uniformly arranging the annular coding mark points on the surface of a to-be-assembled workpiece through a projector, and recording the three-dimensional coordinates of each mark point; capturing a to-be-assembled workpiece image containing each mark point by using a camera to obtain a two-dimensional pixel coordinate of each mark point in a camera coordinate system; sorting the two-dimensional pixel coordinates, and obtaining a corresponding relation between the two-dimensional pixel coordinates and the three-dimensional coordinates based on a recording sequence of the three-dimensional coordinates; projecting a preset Gray code and a phase shift code by using a projector, and capturing an image by using a camera so as to convert the two-dimensional pixel coordinate into a two-dimensional coordinate under a projector coordinate system; and calculating a relative relation matrix of the two-dimensional coordinates and the three-dimensional coordinates to obtain two-dimensional projection coordinates, and performing projection on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates. According to the method, the efficiency of augmented reality projection can be improved by improving the automatic identification precision of the mark points.

Description

Technical Field

[0001] The present application belongs to the field of three-dimensional optical projection and assembly engineering technology, and in particular, relates to an augmented reality projection method based on ring-coded markers. Background Art

[0002] In 3D optical projection measurement, the identification and positioning of markers is one of the key links. Common marker types include circular markers, square markers, etc., but these markers are prone to misidentification or inaccurate identification in complex scenes. In order to improve the recognition accuracy and reliability of markers, ring-coded markers came into being.

[0003] In the existing field of optical projection and assembly engineering technology, how to accurately and efficiently identify the markers on the measured object has always been a hot topic and difficulty in research. Traditional markers require too much manual participation, and the position of the markers in the camera's field of view needs to be manually selected. Although it can improve the recognition accuracy to a certain extent, in complex environments or when the object's posture changes greatly, more markers need to be manually selected, and its reliability is often greatly affected. Summary of the invention

[0004] In view of this, the purpose of this application is to provide an augmented reality projection method based on ring-coded markers, through the design of ring-coded markers, and the automated recognition process, that is, automatically identifying ring-coded markers and obtaining the two-dimensional pixel coordinates of all markers, and calculating the transformation matrix with the three-dimensional coordinates, there is no manual selection of markers, which greatly reduces the degree of manual participation and improves the recognition accuracy and stability of markers. Even in complex environments and when the posture of objects changes significantly, it can still maintain excellent performance, effectively improving the efficiency and reliability of augmented reality projection.

[0005] The present application provides an augmented reality projection method based on annular coded markers, the method comprising:

[0006] Use a projector to evenly arrange circular coded marking points on the surface of the workpiece to be assembled, and record the three-dimensional coordinates of each marking point;

[0007] The camera is used to capture the image of the workpiece to be assembled containing each mark point, so as to obtain the two-dimensional pixel coordinates of each mark point in the camera coordinate system;

[0008] Sorting the two-dimensional pixel coordinates, and obtaining a corresponding relationship between the two-dimensional pixel coordinates and the three-dimensional coordinates based on the recording order of the three-dimensional coordinates;

[0009] Using a projector to project the preset Gray code and phase shift code, and using a camera to capture an image, so as to convert the two-dimensional pixel coordinates into two-dimensional coordinates in the projector coordinate system;

[0010] A relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates is calculated to obtain two-dimensional projection coordinates, and projection is performed on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates.

[0011] Furthermore, the annular code is encoded by a 12-bit binary number, and the marking point is composed of three concentric rings, and the concentric rings are divided into a white area and a black area; wherein the white area is represented as 1 and the black area is represented as 0.

[0012] Furthermore, the method of capturing the image of the workpiece to be assembled containing each mark point by using a camera to obtain the two-dimensional pixel coordinates of each mark point in the camera coordinate system includes:

[0013] Use a camera to capture an image of the workpiece to be assembled containing each mark point;

[0014] After preprocessing the image of the workpiece to be assembled, using the Canny edge detection method, detecting the area with concentric ring characteristics in the preprocessed image of the workpiece to be assembled, so as to identify each initial ROI;

[0015] Performing perspective correction on the image of the workpiece to be assembled containing the initial ROI to map the ellipse into a standard circle to identify each clear ROI;

[0016] The OTSU algorithm is used to perform binarization processing on the image of the workpiece to be assembled containing a clear ROI, and the clear ROI is divided into a plurality of regions to be determined, so as to identify each landmark point;

[0017] The two-dimensional pixel coordinates of each marker point in the camera coordinate system are extracted; wherein the two-dimensional pixel coordinates are used to characterize the encoding order of each marker point.

[0018] Furthermore, each landmark point is identified by:

[0019] For each area to be determined, four pre-set concentric circles are used to cover the area to be determined; wherein the radii of the concentric circles are 0.5r, 1.5r, 2.5r, and 3.5r respectively, and r is the minimum radius of the concentric rings of the marking points;

[0020] For each concentric circle, the pixel values ​​of the area to be determined are uniformly sampled on the circumference of the concentric circle, and the pixel values ​​of each sampling point are averaged to determine the pixel average value of the concentric circle;

[0021] Determining whether the area to be determined satisfies the annular coding according to the average values ​​of the pixels of the four concentric circles;

[0022] The to-be-determined area satisfying the annular coding is determined as the landmark point.

[0023] Further, whether the area to be determined satisfies the circular coding is determined in the following manner:

[0024] When the pixel average value of the 0.5r concentric circles is displayed as white, the pixel average value of the 1.5r concentric circles is displayed as black, the pixel average value of the 2.5r concentric circles is not displayed as black or white, and the pixel average value of the 3.5r concentric circles is displayed as black, it is determined that the area to be determined satisfies the said circular coding; wherein, when the pixel value is 255, it is displayed as white, and when the pixel value is 0, it is displayed as black.

[0025] Furthermore, extracting the two-dimensional pixel coordinates of each marker point in the camera coordinate system includes:

[0026] For each marker point, sample the 12-bit binary number of the marker point from any position on the circumference of the 2.5r concentric circle;

[0027] Convert each group of 12 binary numbers obtained into decimal numbers;

[0028] A group of 12-bit binary numbers corresponding to the minimum decimal number is determined as the binary code value of the mark point, and the minimum decimal number is determined as the two-dimensional pixel coordinates of the mark point.

[0029] Furthermore, the projecting of the preset Gray code and phase shift code by a projector and capturing an image by a camera to convert the two-dimensional pixel coordinates into two-dimensional coordinates in a projector coordinate system includes:

[0030] Using a projector to project the preset Gray code and phase-shift code, and using a camera to capture the projection images of the Gray code and phase-shift code;

[0031] Decoding the projected image to obtain a relative position relationship between the camera and the projector;

[0032] Based on the two-dimensional pixel coordinates and according to the relative position relationship between the camera and the projector, the two-dimensional coordinates of each marker point in the projector coordinate system are obtained.

[0033] Furthermore, the calculation of the relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates to obtain the two-dimensional projection coordinates, and projection on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates, includes:

[0034] Using the PNP algorithm, the relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates is calculated to obtain the relative position relationship between the workpiece to be assembled and the projector;

[0035] Determine the points of interest of the workpiece to be assembled and the corresponding three-dimensional coordinates, and obtain the two-dimensional projection coordinates of the workpiece to be assembled based on the three-dimensional coordinates corresponding to the points of interest and the relative position relationship between the workpiece to be assembled and the projector;

[0036] Projection is performed on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates to obtain an augmented reality projection result of the workpiece to be assembled.

[0037] Furthermore, the lenses of the projector and the camera are both facing the workpiece to be assembled, and the field of view of the projector can cover the field of view of the camera.

[0038] The augmented reality projection method based on annular coded markers provided by the present application, through the design of annular coded markers and an automated recognition process, automatically identifies annular coded markers and obtains the two-dimensional pixel coordinates of all markers, and simultaneously calculates the transformation matrix with the three-dimensional coordinates. Manual selection of markers is not involved, which greatly reduces the degree of manual participation and improves the recognition accuracy and stability of markers. Even in complex environments and when the posture of objects changes significantly, it can still maintain excellent performance, effectively improving the efficiency and reliability of augmented reality projection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of an augmented reality projection method based on annular coded marker points provided in an embodiment of the present application is shown;

[0040] Figure 2 A schematic diagram of landmark points obtained based on circular coding provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the technical solution more clear, the technical solution is further described in detail below in conjunction with specific implementation methods. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the technical solution.

[0042] Since the augmented reality projection method based on circular coded marking points of the present application is implemented by the cooperation of a camera and a projector, an augmented reality projection system is built using a camera and a projector so that the lenses of the projector and the camera are both facing the workpiece to be assembled. After the camera and the projector are focused on the workpiece to be assembled, it is necessary to ensure that the field of view of the projector covers the field of view of the camera. During the implementation of the augmented reality projection method, the positions of the camera and the projector remain unchanged.

[0043] See also Figure 1 , Figure 1 The flowchart of the augmented reality projection method based on the ring-coded marker points provided by the embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0044] S101. Use a projector to evenly arrange circular coded marking points on the surface of the workpiece to be assembled, and record the three-dimensional coordinates of each marking point.

[0045] The annular code is encoded by a 12-bit binary number, the marking point is composed of three concentric rings, and the concentric rings are divided into a white area and a black area; the white area is represented by 1 and the black area is represented by 0.

[0046] In this step, three concentric rings of annular coding are formed by four concentric circles, wherein the ratio of the diameters of the four concentric circles from the inner to the outer is 2:3:5:6. Here, the ratio can be adjusted according to actual needs. The three concentric rings are respectively divided into white area and black area.

[0047] Specifically, a 12-bit binary number consisting of 0 and 1 is obtained through 12-bit binary encoding. According to the principle that white area represents 1 and black area represents 0, the middle layer of concentric rings is divided into white and black alternating areas; in addition, the innermost and outermost concentric rings are all divided into black areas. Figure 2 , Figure 2 Shown is a schematic diagram of landmark points obtained based on circular coding provided in an embodiment of the present application.

[0048] Furthermore, when arranging the marking points on the surface of the workpiece to be assembled, the three-dimensional coordinates of each marking point are recorded simultaneously. Here, a measuring tool (such as a laser tracker) can be used to measure the three-dimensional coordinates of each marking point.

[0049] S102 , using a camera to capture an image of the workpiece to be assembled containing each mark point, so as to obtain a two-dimensional pixel coordinate of each mark point in a camera coordinate system.

[0050] In the specific implementation, the two-dimensional pixel coordinates of each marker point in the camera coordinate system are obtained by the following method:

[0051] Step 1021: Use a camera to capture an image of the workpiece to be assembled that includes each mark point.

[0052] In this step, the exposure parameters of the camera are set so that each marking point can be clearly seen in the captured image of the workpiece to be assembled.

[0053] Step 1022: After preprocessing the image of the workpiece to be assembled, use the Canny edge detection method to detect the area with concentric ring characteristics in the preprocessed image of the workpiece to be assembled, so as to identify each initial ROI.

[0054] In this step, after the image of the workpiece to be assembled is preprocessed by grayscale and denoising, the Canny edge detection method is used to extract the contour. By filtering out contours with shorter contours and smaller roundness, the area with concentric ring characteristics in the preprocessed image of the workpiece to be assembled is detected to identify each initial ROI.

[0055] Step 1023 : Perform perspective correction on the image of the workpiece to be assembled containing the initial ROI, so as to map the ellipse into a standard circle, so as to identify each clear ROI.

[0056] In this step, since any ellipse has a minimum circumscribed rectangle, the four vertices of the circumscribed rectangle of the larger ellipse in the concentric circles are used as transformation points, and the corresponding four target points are used to calculate a transformation matrix. Then, bilinear interpolation is performed on the local area to perform perspective correction on the image of the workpiece to be assembled and identify each clear ROI.

[0057] Step 1024: Use the OTSU algorithm to perform binarization processing on the image of the workpiece to be assembled containing the clear ROI, and divide the clear ROI into a plurality of regions to be determined, so as to identify each marker point.

[0058] Specifically, each landmark is identified by:

[0059] Step 201: For each area to be determined, four pre-set concentric circles are used to cover the area to be determined; wherein the radii of the concentric circles are 0.5r, 1.5r, 2.5r, and 3.5r, respectively, and r is the minimum radius of the concentric rings of the marker points.

[0060] Step 202: for each concentric circle, uniformly sample the pixel values ​​of the area to be determined on the circumference of the concentric circle, and average the pixel values ​​of each sampling point to determine the pixel average value of the concentric circle.

[0061] Step 203: Determine whether the area to be determined satisfies the annular coding according to the average pixel values ​​of the four concentric circles.

[0062] Specifically, whether the area to be determined satisfies the circular coding is determined in the following manner:

[0063] Step 2031, when it is simultaneously satisfied that the pixel average value of the 0.5r concentric circles is displayed as white, the pixel average value of the 1.5r concentric circles is displayed as black, the pixel average value of the 2.5r concentric circles is not displayed as black or white, and the pixel average value of the 3.5r concentric circles is displayed as black, it is determined that the area to be determined satisfies the said circular coding; wherein, when the pixel value is 255, it is displayed as white, and when the pixel value is 0, it is displayed as black.

[0064] Step 204: determine the to-be-determined area that satisfies the annular coding as the landmark point.

[0065] Step 1025: extract the two-dimensional pixel coordinates of each marker point in the camera coordinate system; wherein the two-dimensional pixel coordinates are used to represent the encoding order of each marker point.

[0066] In specific implementation, the two-dimensional pixel coordinates of each marker point in the camera coordinate system can be extracted in the following way:

[0067] Step 301: For each marker point, sample the 12-bit binary number of the marker point starting from any position on the circumference of the 2.5r concentric circle.

[0068] Step 302: Convert each group of 12-bit binary numbers into decimal numbers.

[0069] Step 303: determine a group of 12-bit binary numbers corresponding to the minimum decimal number as the binary code value of the marker point, and determine the minimum decimal number as the two-dimensional pixel coordinates of the marker point.

[0070] After determining the binary code value of the marker point, the method further includes: transposing the binary code value of the marker point bit by bit to obtain multiple groups of 12-bit binary numbers, and calculating the binary number average of each binary bit, and converting the binary number average into a decimal number according to the binary bit sequence of the marker point to verify whether the binary code value of the marker point is correct.

[0071] S103 , sorting the two-dimensional pixel coordinates, and obtaining a corresponding relationship between the two-dimensional pixel coordinates and the three-dimensional coordinates based on the recording order of the three-dimensional coordinates.

[0072] In this step, the two-dimensional pixel coordinates are sorted according to the coding order of each marker point represented by the two-dimensional pixel coordinates to make them consistent with the recording order of the three-dimensional coordinates, so that a one-to-one correspondence between the two-dimensional pixel coordinates and the three-dimensional coordinates can be obtained.

[0073] S104, using a projector to project the preset Gray code and phase shift code, and using a camera to capture an image, so as to convert the two-dimensional pixel coordinates into two-dimensional coordinates in a projector coordinate system.

[0074] In specific implementation, the two-dimensional pixel coordinates are converted into two-dimensional coordinates in the projector coordinate system by the following method:

[0075] Step 1041: Use a projector to project the preset Gray code and phase-shift code, and use a camera to capture the projection images of the Gray code and phase-shift code.

[0076] Step 1042: Decode the projected image to obtain the relative position relationship between the camera and the projector.

[0077] Step 1043: Based on the two-dimensional pixel coordinates and according to the relative position relationship between the camera and the projector, obtain the two-dimensional coordinates of each marker point in the projector coordinate system.

[0078] S105 , calculating a relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates to obtain two-dimensional projection coordinates, and projecting the two-dimensional projection coordinates onto the surface of the workpiece to be assembled.

[0079] In specific implementation, projection is performed on the surface of the workpiece to be assembled in the following manner:

[0080] Step 1051: Utilize the PNP algorithm to calculate the relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates, and obtain the relative position relationship between the workpiece to be assembled and the projector.

[0081] Step 1052: determine the points of interest of the workpiece to be assembled and the corresponding three-dimensional coordinates, and obtain the two-dimensional projection coordinates of the workpiece to be assembled based on the three-dimensional coordinates corresponding to the points of interest and the relative position relationship between the workpiece to be assembled and the projector.

[0082] Step 1053 : Projecting onto the surface of the workpiece to be assembled according to the two-dimensional projection coordinates to obtain an augmented reality projection result of the workpiece to be assembled.

[0083] The above contents are only preferred embodiments of the present invention. For ordinary technicians in this field, many changes can be made in the specific implementation methods and application scopes based on the ideas of the present technical content. As long as these changes do not deviate from the concept of the present invention, they all fall within the scope of protection of this patent.

Claims

1. An augmented reality projection method based on ring-coded markers, characterized in that: The method comprises: Use a projector to evenly arrange circular coded marking points on the surface of the workpiece to be assembled, and record the three-dimensional coordinates of each marking point; The camera is used to capture the image of the workpiece to be assembled containing each mark point, so as to obtain the two-dimensional pixel coordinates of each mark point in the camera coordinate system; Sorting the two-dimensional pixel coordinates, and obtaining a corresponding relationship between the two-dimensional pixel coordinates and the three-dimensional coordinates based on the recording order of the three-dimensional coordinates; Using a projector to project the preset Gray code and phase shift code, and using a camera to capture an image, so as to convert the two-dimensional pixel coordinates into two-dimensional coordinates in the projector coordinate system; A relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates is calculated to obtain two-dimensional projection coordinates, and projection is performed on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates.

2. The method according to claim 1, characterized in that The annular code is encoded by a 12-bit binary number, and the marking point is composed of three concentric rings, and the concentric rings are divided into a white area and a black area; wherein the white area is represented by 1 and the black area is represented by 0.

3. The method according to claim 1, characterized in that The method of capturing an image of the workpiece to be assembled containing each mark point by using a camera to obtain a two-dimensional pixel coordinate of each mark point in a camera coordinate system includes: Use a camera to capture an image of the workpiece to be assembled containing each mark point; After preprocessing the image of the workpiece to be assembled, using the Canny edge detection method, detecting the area with concentric ring characteristics in the preprocessed image of the workpiece to be assembled, so as to identify each initial ROI; Performing perspective correction on the image of the workpiece to be assembled containing the initial ROI to map the ellipse into a standard circle to identify each clear ROI; The OTSU algorithm is used to perform binarization processing on the image of the workpiece to be assembled containing a clear ROI, and the clear ROI is divided into a plurality of regions to be determined, so as to identify each landmark point; The two-dimensional pixel coordinates of each marker point in the camera coordinate system are extracted; wherein the two-dimensional pixel coordinates are used to characterize the encoding order of each marker point.

4. The method according to claim 3, characterized in that Each landmark is identified by: For each area to be determined, four pre-set concentric circles are used to cover the area to be determined; wherein the radii of the concentric circles are 0.5r, 1.5r, 2.5r, and 3.5r, respectively, and r is the minimum radius of the concentric rings of the marking points; For each concentric circle, the pixel values ​​of the area to be determined are uniformly sampled on the circumference of the concentric circle, and the pixel values ​​of each sampling point are averaged to determine the pixel average value of the concentric circle; Determining whether the area to be determined satisfies the annular coding according to the average values ​​of the pixels of the four concentric circles; The to-be-determined area satisfying the annular coding is determined as the landmark point.

5. The method according to claim 4, characterized in that Whether the area to be determined satisfies the annular coding is determined in the following manner: When the pixel average value of the 0.5r concentric circles is displayed as white, the pixel average value of the 1.5r concentric circles is displayed as black, the pixel average value of the 2.5r concentric circles is not displayed as black or white, and the pixel average value of the 3.5r concentric circles is displayed as black, it is determined that the area to be determined satisfies the said circular coding; wherein, when the pixel value is 255, it is displayed as white, and when the pixel value is 0, it is displayed as black.

6. The method according to claim 4, characterized in that The step of extracting the two-dimensional pixel coordinates of each marker point in the camera coordinate system includes: For each marker point, sample the 12-bit binary number of the marker point from any position on the circumference of the 2.5r concentric circle; Convert each group of 12 binary numbers obtained into decimal numbers; A group of 12-bit binary numbers corresponding to the minimum decimal number is determined as the binary code value of the mark point, and the minimum decimal number is determined as the two-dimensional pixel coordinates of the mark point.

7. The method according to claim 6, characterized in that The method of projecting the preset Gray code and phase shift code by a projector and capturing an image by a camera to convert the two-dimensional pixel coordinates into two-dimensional coordinates in a projector coordinate system includes: Using a projector to project the preset Gray code and phase-shift code, and using a camera to capture the projection images of the Gray code and phase-shift code; Decoding the projected image to obtain a relative position relationship between the camera and the projector; Based on the two-dimensional pixel coordinates and according to the relative position relationship between the camera and the projector, the two-dimensional coordinates of each marker point in the projector coordinate system are obtained.

8. The method according to claim 3, characterized in that The step of calculating the relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates to obtain the two-dimensional projection coordinates, and projecting the two-dimensional projection coordinates on the surface of the workpiece to be assembled includes: Using the PNP algorithm, the relative relationship matrix between the two-dimensional coordinates and the three-dimensional coordinates is calculated to obtain the relative position relationship between the workpiece to be assembled and the projector; Determine the points of interest of the workpiece to be assembled and the corresponding three-dimensional coordinates, and obtain the two-dimensional projection coordinates of the workpiece to be assembled based on the three-dimensional coordinates corresponding to the points of interest and the relative position relationship between the workpiece to be assembled and the projector; Projection is performed on the surface of the workpiece to be assembled according to the two-dimensional projection coordinates to obtain an augmented reality projection result of the workpiece to be assembled.

9. The method according to claim 1, characterized in that The lenses of the projector and the camera are both facing the workpiece to be assembled, and the field of view of the projector can cover the field of view of the camera.

Images