A map creating method and system based on long-distance two-dimensional code

Abstract

The application provides a map creation method and system based on a long-distance two-dimensional code, wherein the method steps include: setting the long-distance two-dimensional code at a look-up position of the top of a mapping area relative to a camera; acquiring an image containing the long-distance two-dimensional code through the camera to extract the spatial pose and ID information of the long-distance two-dimensional code in a camera coordinate system; loading the spatial position relationship between the first long-distance two-dimensional code and the camera into a map, and taking the position of the current camera as the origin of the map coordinate system; when the image observed by the camera contains the first observed long-distance two-dimensional code B, calculating the relative attitude between the first observed long-distance two-dimensional code and the recorded long-distance two-dimensional code A in the image to obtain the position of the first observed long-distance two-dimensional code B in the map, and adding the first observed long-distance two-dimensional code B into the map. Thus, the indoor map creation is realized based on the long-distance two-dimensional code.

Description

Technical Field

[0001] This invention relates to indoor spatial mapping and positioning technology, and more particularly to a map creation method and system based on long-distance QR codes. Background Technology

[0002] Currently, the mainstream indoor mobile robot mapping and localization solutions typically employ laser mapping technology. However, in some special industrial scenarios, such as stacking, destacking, narrow passages, and dynamic environments, this technology is easily affected by the environment and stacked objects, which can obstruct laser radar scanning and thus affect the quality of laser mapping and localization.

[0003] Therefore, considering the unique characteristics of long-range QR code positioning, the inventors suggest that long-range QR codes can be placed in an unobstructed and open environment such as an indoor ceiling. In this case, the camera will not be easily affected by the ground environment when capturing the long-range QR code. Therefore, if long-range QR code positioning technology can be used for indoor mapping, it will have significant advantages in the aforementioned special industrial scenarios. Summary of the Invention

[0004] Therefore, the main objective of this invention is to provide a map creation method and system based on long-distance QR codes, so as to realize indoor map creation based on long-distance QR codes.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a map creation method based on long-range QR codes is provided, comprising the following steps:

[0006] Step S100: Set the remote QR code at the top of the mapping area in an upward position relative to the camera; acquire an image containing the remote QR code through the camera to extract the spatial pose and ID information of the remote QR code in the camera coordinate system;

[0007] Step S200: Load the spatial relationship between the first long-distance QR code and the camera into the map, and use the current location of the camera as the origin of the map coordinate system;

[0008] Step S300: When the image observed by the camera contains the first observed long-range QR code B, calculate the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image, so as to obtain the position of the first observed long-range QR code B in the map and add it to the map.

[0009] In a possible preferred embodiment, step S300, calculating the relative attitude between B and A, includes:

[0010] Step S310 fuses the spatial pose of the long-range QR code extracted from each frame of image in the form of a factor graph.

[0011] Step S320 optimizes the spatial pose of each long-distance QR code through incremental updates;

[0012] Step S330 calculates the relative pose based on the optimized long-distance QR code spatial pose.

[0013] In a possible preferred embodiment, step S310, which involves fusing each frame of images to extract the spatial pose of the long-range QR code, includes:

[0014] Step S311 uses the measured value Constraining the variables to be optimized Using Bayesian networks to model The optimized state variables are solved using the maximum a posteriori probability:

[0015]

[0016] in For the spatial pose of all distant QR codes on the map, the measured values A set of measurements, including:

[0017] Indicates that the camera is in the first position. At each position, the camera... Observations;

[0018] Represents the location code in a long-range QR code The distance between the centers;

[0019] Represents the first in a long-distance QR code The pixel coordinates of the center of each positioning code circle;

[0020] Represents the coordinate system of the long-distance QR code. The spatial location of the center of the positioning code;

[0021] This indicates the camera's position on the distant QR code map. One location;

[0022] This indicates the camera's intrinsic parameters.

[0023] In a possible preferred embodiment, the long-range QR code includes: a positioning code and an information code, wherein the positioning code is formed by several directional points arranged around a center point in several non-intersecting directions, and several quadrant regions are divided at the angle between each adjacent directional point, and the information code is arranged in the corresponding quadrant region according to a preset code table to form a two-dimensional dot matrix with the positioning code, wherein the surfaces of the positioning code and the information code are provided with a reflective layer.

[0024] In a possible preferred embodiment, step S100, the step of extracting the spatial pose and ID information of the long-range QR code in the camera coordinate system, includes:

[0025] Step S110: After acquiring the image containing the long-distance QR code information, perform binarization processing and then edge extraction.

[0026] Step S120 performs ellipse fitting detection based on the extracted edge information to obtain the original code and its center coordinates;

[0027] Step S120: Based on the geometric relationship of the positioning codes, locate the distribution position of the positioning codes of each remote QR code in the original code, so as to filter out the information codes in the original code, and obtain the ID information of the remote QR code according to the code table.

[0028] Step S140: When it is determined that the long-range QR code under the ID appears for the first time, calculate the spatial pose of the corresponding long-range QR code in the camera coordinate system according to the center coordinates of the positioning code in the image and bind it to the ID information.

[0029] In a possible preferred embodiment, step S140, which involves calculating the spatial pose of the long-range QR code in the camera coordinate system, includes:

[0030] Step S141 sets the center coordinates of the bit code as follows: Establish a homogeneous matrix

[0031]

[0032] The homography matrix H is solved using the SVD method, where u and v represent the pixel coordinates of the center of the location code. This represents the coordinates of a point in the long-distance QR code coordinate system, where s is the equivalent distance scaling factor.

[0033] Step S142: Based on the relationship between the homography matrix and the transformation matrix of the long-range QR code in the camera coordinate system.

[0034]

[0035] The rotation and translation matrices are obtained, where P is the camera projection matrix and E is the truncation extrinsic parameter matrix. , Cameras direction and Focal length of direction, , Here are the coordinates of the camera center point. These are the first two columns of the rotation matrix. , , These represent the positions of the far-distance QR code center in the camera coordinate system. It is a 3×3 homography projection matrix.

[0036] In a possible preferred embodiment, step S140 further includes:

[0037] Step 143 solves for minimizing the error function using an iterative optimization method.

[0038]

[0039] To optimize the rotation and translation matrices, where R represents rotation and t represents translation.

[0040]

[0041]

[0042] This represents the i-th element in the translation vector. Represents the i-th column in the rotation matrix; where These represent the first and second parts of the image, respectively. The coordinates of the center of a distant QR code projected onto the camera's normalized coordinate system. ; Represents the center coordinates of all the location codes; These represent the first and second parts of the long-distance QR code. The spatial coordinates of the center of each positioning code, transformed into the coordinates of a point in the camera's normalized plane using a rotation and translation matrix. ; This represents the spatial coordinates of the center of each location barcode in the long-range QR code coordinate system. ;

[0043] In the solution process,

[0044]

[0045] As a constraint.

[0046] In a possible preferred embodiment, the steps of the map creation method based on long-distance QR codes further include:

[0047] Step S150 constrains the spatial pose obtained in step S140 according to time filtering and prior pose constraints. The time filtering step includes filtering out when there is a sudden change in the spatial pose result. The prior pose constraint step includes filtering out when the coordinate system of the far-field QR code and the camera coordinate system in the spatial pose result are not parallel.

[0048] In a possible preferred embodiment, step S120 further includes: determining whether the ratio of the major axis to the minor axis of each original code meets the threshold according to the formulas for the major and minor axes of the ellipse equation, and filtering out the code when it does not meet the threshold.

[0049] To achieve the above objectives, according to a second aspect of the present invention, a map creation method system based on long-range QR codes is also provided, comprising:

[0050] The storage unit is used to store a program including the steps of the map creation method based on long-distance QR codes as described above, for the control unit and processing unit to retrieve and execute as needed.

[0051] The control unit is used to control the infrared camera to capture images containing long-range QR codes;

[0052] The processing unit performs binarization on the image to be processed and then performs edge extraction. Based on the extracted edge information, it performs ellipse fitting detection to obtain the original code and its center coordinates. Then, based on the geometric relationship of the preset positioning codes of the long-range QR codes, it locates the distribution positions of the positioning codes corresponding to each long-range QR code in the original code to filter out the information codes in the original code. It also obtains the ID information of the long-range QR codes according to the passcode table. When a long-range QR code under a given ID appears for the first time, it calculates the spatial pose of the long-range QR code in the camera coordinate system based on the center coordinates of the positioning code in the image and stores the corresponding ID information in the storage unit. Then, it loads the spatial position relationship between the first long-range QR code and the camera into the map, using the current camera position as the origin of the map coordinate system. When the image observed by the camera contains the first observed long-range QR code B, it calculates the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image to obtain the position of the first observed long-range QR code B in the map and adds it to the map.

[0053] The map creation method and system based on long-distance QR codes provided by this invention cleverly utilizes the characteristic that a circular geometric shape, even with changes in camera angle and affine transformation, will only become an ellipse and still retain its center. It also leverages the robust characteristic that a circle remains circular / elliptical even when detection at long distances is prone to blurring. This allows for the design of QR codes suitable for long-distance detection. Furthermore, corresponding to the aforementioned characteristics of long-distance QR codes, ellipse detection can accurately obtain the center of the ellipse to establish the pose transformation matrix with the camera. This solves the problem that traditional QR codes often appear as blurry clumps in the detection image when the distance and angle of the detection camera are large, leading to inaccurate recognition. Based on this, by calculating the relative poses between long-distance QR codes in the image, the position of each code on the map can be obtained sequentially, thus realizing the creation of indoor QR code maps. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0055] Figure 1 This is a schematic diagram illustrating the use of a long-range QR code of the present invention affixed to the ceiling in an indoor environment for indoor positioning and mapping.

[0056] Figure 2 This is a schematic diagram illustrating the steps of the map creation method based on long-distance QR codes of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating the breakdown steps of the long-distance QR code-based map creation method of the present invention, which involves continuously detecting long-distance QR codes to calculate the relative positions between codes for positioning and mapping.

[0058] Figure 4 This is a schematic diagram illustrating the process logic of the map creation method based on long-distance QR codes of the present invention.

[0059] Figure 5 This is a schematic diagram of the long-range QR code dot matrix structure of the present invention, wherein the circular part indicates the possible placement position of the information code;

[0060] Figure 6 This is a schematic diagram of the long-distance QR code representation of the present invention;

[0061] Figure 7 This is a schematic diagram of the geometric positional relationship of the long-distance QR code according to the present invention;

[0062] Figure 8 A schematic diagram of the long-range QR code of the present invention, captured by an infrared camera;

[0063] Figure 9 This is a schematic diagram illustrating the steps of calculating the spatial pose of an object based on a long-range QR code in the map creation method based on a long-range QR code of the present invention.

[0064] Figure 10 This is a schematic diagram of the binarized image in the map creation method based on long-distance QR codes of the present invention;

[0065] Figure 11 This is a schematic diagram of a pose singularity phenomenon.

[0066] Figure 12 This is a schematic diagram of the map creation system based on long-distance QR codes according to the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with embodiments, so as to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this application are merely some embodiments of the present invention, and not all embodiments. It should be noted that, for those skilled in the art, the embodiments and features in the embodiments of this application can be combined with each other without departing from the concept of the present invention and without conflict. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the disclosure and protection scope of the present invention.

[0068] Furthermore, the terms "first," "second," "S100," "S200," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those described herein. At the same time, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arranged," "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this case according to the specific circumstances and in conjunction with existing technology.

[0069] To create indoor maps based on long-range QR codes, please refer to [link / reference]. Figures 1 to 4 As shown, this invention provides a map creation method based on long-distance QR codes, the steps of which include:

[0070] Step S100: Set the remote QR code at the top of the mapping area in an upward position relative to the camera; acquire an image containing the remote QR code through the camera to extract the spatial pose and ID information of the remote QR code in the camera coordinate system.

[0071] Please see Figure 1 As shown, when establishing an indoor QR code positioning scheme, if the QR code is affixed to an upper space far from the ground, such as the indoor ceiling, the camera of the moving robot will encounter a technical defect where the QR code cannot be accurately recognized due to factors such as distance and angle changes when it takes a picture of the QR code.

[0072] For this purpose, please refer to Figures 5 to 8As shown, in order to ensure the recognition accuracy of QR codes under long-distance and multi-angle detection conditions, the present invention provides a long-distance QR code, which includes: a positioning code and an information code. The positioning code is formed by several directional points arranged around a center point in several non-intersecting directions, and several quadrant regions are divided at the angle between each adjacent directional point. The information code is arranged in the corresponding quadrant region according to a preset code table to form a two-dimensional dot matrix with the positioning code.

[0073] Specifically, the inventors discovered that circles (including ellipses, elliptical rings, and circular rings) undergo affine transformations when the viewing angle changes, appearing as ellipses in the camera plane. Therefore, their ability to adapt to changes in viewing angle is relatively weak. While traditional square pixel QR codes are more robust to corner points under different viewing angles, their weakness lies in the tendency for corner points to shift at long distances. Based on this discovery, the inventors designed a dot matrix-style QR code.

[0074] like Figures 5 to 7 As shown in this example, in this long-range QR code, the positioning code is composed of several directional points arranged around the central point according to the four cardinal directions (north, south, east, and west), so as to divide the area at the angle between each adjacent directional point. Figure 7 The four quadrant regions are shown, and the information code is as follows: Figure 6 As shown, codes can be deployed in the corresponding quadrant areas according to a preset code table to form a combination with the positioning code, as shown below. Figure 5 The two-dimensional dot matrix shown is preferably in the form of a circle, an ellipse, an elliptical ring, or a circular ring in this example.

[0075] With this configuration, the long-range QR code uses only one basic pattern (dots), which reduces the difficulty of processing and producing the QR code. Furthermore, the arrangement of any two QR codes in this dot matrix design can vary significantly, thus reducing the chance of false detection and supporting different height distances.

[0076] In addition, to enhance the layout characteristics of the positioning code, such as Figure 7 As shown in this preferred example, at least one main direction point exists among the plurality of direction points (e.g., Figure 7 The distance between point A (in the code) and the center point B differs from that of other secondary directional points; for example, the distance between A and B is half the distance between C, D, and F and B. However, the distances between each secondary directional point and the center point are equal. Using this as one of the constraints for long-distance QR code recognition can further improve recognition accuracy.

[0077] Furthermore, considering the combined effects of ambient light, the background color of the indoor ceiling, and suspended obstructions, which can interfere with the image captured by the camera and affect the recognition rate of long-distance QR codes, this example preferably uses a reflective layer, such as bright silver reflective cloth, on the surface of the positioning and information codes of the long-distance QR code. Correspondingly, the camera is preferably set as an infrared camera, with an infrared filter installed between the camera lens and the CMOS sensor to allow infrared light within a certain wavelength to pass through while absorbing or reflecting visible light and ultraviolet light. An infrared lamp can also be optionally added to the camera. After this setup, as shown... Figure 8 As shown, the infrared camera can easily capture images that clearly contain long-range QR code information.

[0078] On the other hand, corresponding to the long-distance QR code in the above example, such as Figure 9 As shown, the steps for extracting the spatial pose and ID information of a distant QR code in the camera coordinate system include:

[0079] Step S110: After acquiring the image containing the long-distance QR code information, perform binarization processing and then perform edge extraction.

[0080] Specifically, based on the example of long-range QR codes above, when an infrared camera captures an image containing a long-range QR code, the acquired image is as follows: Figure 8 As shown, except for the white dots representing the location code and information code, the rest of the image is black. In this case, the pixel values ​​of the image can be inverted. The inverted image is then binarized to convert the grayscale image into a black and white image, as shown below. Figure 10 As shown.

[0081] Next, the edges of the black and white image are extracted using the Canny edge detection algorithm. Due to the special characteristics of the aforementioned infrared camera, there are fewer edges in the black and white image, which greatly speeds up the edge extraction process.

[0082] Step S120 performs ellipse fitting detection based on the extracted edge information to obtain the original code and its center coordinates.

[0083] Specifically, a circle is a special shape of an ellipse. Due to different viewing angles, points in a distant QR code generally appear as an approximation of an ellipse in the imaging plane. Ellipse fitting is commonly used in problems such as feature extraction, scene modeling, and camera calibration, given a set of image measurement data from a certain ellipse. The set of all edge points, which contains all the edge points that form an ellipse. This represents each edge point, whose coordinates consist of pixel positions. Here, N represents the number of pixels that make up the edge. Image point edge information is used here.

[0084]

[0085] Since measurement data inevitably contains errors, the problem becomes recovering the corresponding ellipse information from the erroneous data. The equation of the ellipse is... It can be written as: where The set of parameters representing an ellipse. There are a total of 6 parameters for an ellipse.

[0086]

[0087] Since every edge point of the ellipse lies on the ellipse, all edge points lie on the ellipse. Ellipse fitting is transformed into a linear least squares problem:

[0088]

[0089] To ensure that the least squares solution is always an ellipse, constraints must be added. , , ,

[0090]

[0091] in:

[0092]

[0093] This leads to the construction of a constrained nonlinear optimization problem:

[0094]

[0095] For this type of nonlinear optimization problem, the Leweinberg-Marquardt algorithm can be used to solve for the corresponding elliptic parameter information.

[0096] Based on the above ellipse detection scheme, after extracting all edge information, each edge information is fitted with an ellipse using least squares. This yields the fitted information for each black ellipse, i.e., the original code, where the general equation of the ellipse is as follows: Where A, B, C, D, and E are the parameters of the ellipse equation, obtained during the ellipse fitting process.

[0097] By using the ellipse parameters, the coordinates of the geometric center of the ellipse can be obtained as follows:

[0098]

[0099] The major axis of the ellipse The lengths of the minor axis b are respectively:

[0100]

[0101] Furthermore, since there may be noise in the image or the influence of other reflective materials, it is necessary to filter each ellipse information to find the equation of the most round ellipse.

[0102] Therefore, step S120 further includes: determining whether the ratio of the major and minor axes of each original code meets a threshold based on the formulas for the major and minor axes of the ellipse equation, and filtering out codes that do not meet the threshold. For example, if the ratio between the major and minor axes is less than 1.3, the ellipse (original code) is considered not to conform to the shape of a circle and is thus filtered out. Simultaneously, to eliminate the influence of other noise points, the radii of all dots belonging to the same long-range QR code should be within the same range. Therefore, this condition can be used to further filter all original codes to quickly remove unqualified original codes.

[0103] Step S130: Based on the geometric relationship of the positioning codes, locate the distribution position of the positioning codes corresponding to each remote QR code in the original code.

[0104] Specifically, after obtaining all the possible raw codes that make up a long-range QR code, since a long-range QR code, as described in the example above, is composed of a positioning code and an information code. The positioning code consists of five dots, which is a fixed template, such as... Figure 7 As shown, the distance between a circle B at the center and its nearest circle A is exactly equal to the distances between the other three circles C, D, and F and circle B. Based on the geometric relationships described above, the location information of each long-range QR code in the image can be quickly determined.

[0105] Therefore, each long-range QR code is grouped. This classification method can greatly speed up the localization process of long-range QR codes in an image. At the same time, similar to template matching, multiple long-range QR codes in an image can be distinguished without the need for additional clustering.

[0106] Furthermore, by grouping all the long-range QR codes in an image, the location of the positioning code for each long-range QR code within the image is obtained. For further verification, the roundness of all positioning codes is checked to ensure that each circle has the same major and minor axis lengths. Through the spatial position of the positioning codes, the distribution of information codes within the long-range QR codes can be quickly determined.

[0107] Because the information code carries the ID information of the long-range QR code, each long-range QR code has a unique information code. Once the location of the positioning code is determined, all the information codes of the long-range QR code can be obtained by calculating the black and white status of the pixel blocks in each quadrant. The previously established code table determined the spatial position information of the dots in each long-range QR code; therefore, by referring to this code table, the ID information of each long-range QR code can be determined.

[0108] This is used to determine whether the long-range QR code under this ID is appearing for the first time. If it is not the first time, it means that there is a pre-calculated spatial pose matrix of the long-range QR code in the camera coordinate system, which can be directly called. If it is the first time, the next step can be executed to calculate the spatial pose of the long-range QR code in the camera coordinate system. This saves computing power. At the same time, if the position of the long-range QR code in the environment has been recorded, the recognition of this ID can also be used as the positioning of the mobile robot on the map.

[0109] Step S140 calculates the spatial pose of the corresponding long-range QR code in the camera coordinate system based on the center coordinates of the positioning code in the image.

[0110] Specifically, the first step is to determine the coordinate system coordinates and pixel (image) coordinates of the input label (long-range QR code).

[0111] For example: to calculate a 3×3 homography matrix, the 2D dots in the homogeneous coordinates are moved from the label coordinate system (where...). Located at the center of the long-range QR code, the label is and Projecting (one unit in the direction) onto a 2D image coordinate system. Homography is calculated using the Direct Linear Transform (DLT) algorithm. Note that since the homography projection points are in homogeneous coordinates, it is only calculated by scale. definition.

[0112] The second step is to establish the relationship between the homography matrix and the transformation matrix.

[0113] Calculating the position and orientation of a long-range QR code, i.e., the extrinsic parameter matrix, requires additional information: the camera's focal length and the tag's physical size. The 3 × 3 homography matrix (calculated by DLT) can be written as the product of a 3 × 4 camera projection matrix P (assuming it is known) and a 4 × 3 truncated extrinsic parameter matrix E.

[0114] The extrinsic parameter matrix is ​​typically 4 × 4, but each position on the label is at z = 0 in the label coordinate system. Therefore, each label coordinate can be rewritten as a two-dimensional homogeneous point with z implicitly zero, and the third column of the extrinsic parameter matrix can be deleted to form a truncated extrinsic parameter matrix.

[0115] The relationship between the homography matrix and the transformation matrix after the establishment is as follows:

[0116]

[0117] Where P is the camera projection matrix, and its rotation component is expressed as: E is the truncated extrinsic parameter matrix, and the translation components are expressed as... s is the equivalent distance scaling factor.

[0118] The third step is to solve for the homography matrix.

[0119] First, homography of a plane is defined as the projective mapping from one plane to another. Mathematically, this is described by multiplying a point on one plane by the projection matrix, resulting in the corresponding point on the other plane. The coordinates are the center point of the detected ellipse, where u and v represent the pixel coordinates of the ellipse center, respectively. This represents the coordinates of a point in the label coordinate system. It can be expressed as a homogeneous matrix:

[0120]

[0121] The projection matrix H is a 3×3 square matrix.

[0122]

[0123] Expand equation (10)

[0124]

[0125] The above equation is in the form of a ratio of the coefficients in each row of matrix H, meaning that the numerator and denominator on the right side of the equation can be multiplied by a scaling factor.

[0126]

[0127] Without affecting the correctness of the result, the projection matrix is ​​not unique, and the matrix elements can be scaled proportionally. Therefore, when solving matrix H, there are only 8 unknowns instead of 9. As long as a definite value is assigned to any non-zero element in the matrix, the other elements will also get definite values ​​according to the proportion.

[0128] Typically, the matrix used is... The value obtained by setting 1 is denoted as . To ensure that elements h3 and h6 are the values ​​of u and v when both x and y are 0, this has specific significance in certain applications. As can be seen from the preceding derivation, for each pair of points on the plane, we can simultaneously establish equations based on and The coordinates have two equations, and finding the projection matrix H requires eight constraints, so four pairs of points are needed to find H.

[0129] In this example, an additional point is added to ensure the accuracy of pose calculation. As shown in the ellipse detection example in step S200, each long-range QR code can provide at least five sets of positioning points. Five sets of positioning points provide more sufficient constraints compared to four pairs of positioning points, thereby ensuring the accuracy of pose calculation.

[0130] The fourth step is to solve the homography matrix based on SVD.

[0131] Equation (13) can be written in the following form:

[0132]

[0133] That is

[0134]

[0135] in

[0136]

[0137]

[0138] When there are five pairs of points on two planes Then, we can obtain the following system of equations.

[0139]

[0140] in

[0141]

[0142] To make it more intuitive, expand the following formula.

[0143]

[0144] Next, we solve this equation. The solution to this matrix is ​​generally done using singular value decomposition.

[0145] The fifth step is to solve for the transformation matrix using the homography matrix.

[0146]

[0147] After obtaining the H matrix through SVD decomposition in the fourth step, since the camera's intrinsic parameters can be obtained through camera calibration, the vectors can be calculated using the obtained homography matrix and the camera position relationship, i.e., equation (21). , , Due to the orthogonality of the rotation matrices, we can obtain... Obtain the transformation matrix of each long-range QR code relative to the camera coordinate system. This leads to the rotation and translation matrices.

[0148] Furthermore, due to the influence of noise, the rotation and translation matrices calculated by this method are not the global optimal solution. Therefore, it is necessary to construct a nonlinear optimization problem to optimize the result.

[0149] The sixth step is to solve for the minimum error function through iterative optimization to optimize the rotation and translation matrices.

[0150] Since the results obtained by the direct method are greatly affected by image noise, iterative optimization methods are needed to optimize the results based on the distribution of pixels and spatial points to obtain more accurate long-range QR code pose information. This optimization is based on the spatial point set. and its corresponding image point set Determine the camera attitude information.

[0151] The rotation of the camera relative to the QR code coordinate system is represented as: Translation matrix

[0152]

[0153] Under ideal conditions with no noise or other factors affecting the location, and with the camera already calibrated, the spatial point The points projected onto the camera's normalized plane coordinate system satisfy the following equation.

[0154]

[0155] in These represent the first and second parts of the long-distance QR code. The coordinates of the center of each positioning code, transformed into the coordinates of a point in the camera's normalized plane using a rotation and translation matrix. , This represents the spatial coordinates of the center of each location barcode in the long-range QR code coordinate system. .

[0156] Due to the influence of errors, spatial points and pixels do not actually satisfy the above problems, and the problem becomes finding the optimal parameters. Minimize error function , where R represents rotation and t represents translation. This represents the i-th element in the translation vector. Let represent the i-th column in the rotation matrix, where These represent the first and second parts of the image, respectively. The center of a long-distance QR code is projected onto the coordinates in the camera's normalized coordinate system. Let represent the center coordinates of all the locating codes. Then the error minimization function is:

[0157]

[0158] During the minimization process, to ensure that R is a rotation matrix, the following constraints apply:

[0159]

[0160] Its optimization function becomes

[0161]

[0162] For this type of nonlinear optimization problem, the Leweinberg-Marquardt algorithm can be used to solve for its corresponding accurate rotation and translation matrices.

[0163] Furthermore, such as Figure 11 As shown, in practical applications, if the QR code is very small at a distance or viewed from a great distance, positional singularities sometimes occur. Positional singularities are a fundamental characteristic of the problem. Geometrically, these two poses roughly correspond to the object's rotation around a plane whose normal passes through the line of sight from the camera center to the object center. In such cases, there are usually two poses that can be solved using the above method. The reprojection errors of these two solutions are similar, therefore the reprojection error cannot be used to select the correct pose.

[0164] Currently, there is no reliable algorithm to resolve pose ambiguity, as it is a natural property of the problem of recovering 3D pose from a planar object. Solving this problem requires more information to constrain the pose of the QR code.

[0165] Therefore, the method in this example further includes: Step S150, constraining the spatial pose obtained in step S140 according to time filtering and prior pose constraints. Time filtering involves determining whether the mobile robot's pose changes abruptly when calculating its current pose after acquiring the pose information of the distant QR code. If an incorrect positioning occurs, the pose will jump, and if this is detected, the current position information of the distant QR code can be filtered out. Prior pose constraints consider that the distant QR code is affixed to the ceiling, and its coordinate system is in a parallel plane with the ground (i.e., the vehicle body). Therefore, if the distant QR code has an incorrect pose, this constraint can be used to filter it out.

[0166] Step S200 loads the spatial relationship between the first long-distance QR code and the camera into the map, and uses the current location of the camera as the origin of the map coordinate system.

[0167] Specifically, once the QR code recognition problem is solved, a map can be gradually created based on the spatial relationship between the camera and the distant QR code. However, the origin of the map coordinate system must be established first. Therefore, in this example, when the camera captures the first distant QR code, the spatial relationship between the current distant QR code and the camera is obtained. At this time, the location of the current camera (or mobile robot) is taken as the origin of the map coordinate system, and the captured first distant QR code is loaded into the map according to the spatial relationship, thus completing the establishment of the first map location.

[0168] Step S300: When the image observed by the camera contains the first observed long-range QR code B, calculate the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image, so as to obtain the position of the first observed long-range QR code B in the map and add it to the map.

[0169] Specifically, such as Figure 3 As shown, a mobile robot vehicle equipped with a top-view infrared camera moves through the room, sequentially recording the spatial location of each distant QR code. Figure 3 In (a), the car is located below code 1. After the algorithm detects the distant QR code, it obtains the pose of the distant QR code in the camera coordinate system and creates the map origin based on this pose. Figure 3 In (b), the car moves below code 1 and code 2. At this point, since the camera can simultaneously observe code 1 and code 2, it can simultaneously obtain the attitude information of code 1 and code 2 in the camera coordinate system. Since code 1 has been identified by its ID, its spatial position relative to the camera is known. Therefore, it is only necessary to calculate the spatial position of code 2. Then, the relative attitude of code 2 can be calculated from code 1, thereby obtaining the position of code 2 on the map. Figure 3 (c), Figure 3 As shown in (d), the algorithm of the present invention will calculate the pose of code 3 according to the above logic and add it to the map.

[0170] For example, since the spatial range within which a single, fixed, long-range QR code can be observed is limited, multiple long-range QR codes will be deployed in the scene in this example. When a camera can simultaneously capture multiple long-range QR codes at a certain location, it can extract observations from multiple long-range QR codes. At this time, observation factors for different long-range QR codes will be created (i.e., measurements of the QR codes by the camera. These measurements include the real-time pose information of the QR code in the camera coordinate system, the pixel distance information between the QR code positioning points, and the size of the QR code. Because the platform carrying the camera is in motion, it can take pictures in real time during movement and extract QR code information. Therefore, the QR code observation factor increases continuously with the number of times the camera takes pictures).

[0171] Since the spatial location information of multiple long-range QR codes has been extracted, and the rigid body relationship between the camera and these QR codes, along with the location information of previously existing long-range QR codes on the map, allows the creation of QR code factors (i.e., the mathematical expression of the QR code, including its spatial location relationship, the positions of all positioning points within the QR code, and the geometric relationships between these points) for long-range QR codes not currently on the map. This completes the creation of new long-range QR codes on the map, thereby expanding the map.

[0172] Furthermore, since the position of a long-range QR code calculated in a single instance is affected by conditions such as lighting, there will be a certain error in the posture of the long-range QR code. In order to ensure the accuracy of QR code mapping, this invention preferably fuses the long-range QR code information extracted from each frame of image in the form of a factor graph, and optimizes the posture of each long-range QR code through incremental updates, thereby ensuring the real-time performance and robustness of long-range QR code mapping.

[0173] Specifically, to integrate the long-range QR code information obtained at all times, this invention constructs a maximum a posteriori probability problem using a factor graph to solve for the position of each long-range QR code. The position information of all long-range QR codes on the map is determined by... This indicates the use of a series of measurements. Constraining the variables to be optimized And modeled using Bayesian networks The optimized state variables are obtained through the maximum a posteriori probability. This refers to the most accurate spatial location of the long-range QR code. To estimate the location where the maximum a posteriori probability occurs, we need to find a state variable that maximizes the probability value of the product of all factors; this is equivalent to solving for:

[0174]

[0175] Among them, the measured value A set of measurements includes the following aspects:

[0176] Indicates that the camera is in the first position. At each position, the camera... Observations;

[0177] Represents the location code in a long-range QR code The distance between the centers;

[0178] Represents the first in a long-distance QR code The pixel coordinates of the center of each positioning code circle;

[0179] Represents the coordinate system of the long-distance QR code. The spatial location of the center of the positioning code;

[0180] This indicates the camera's position on the distant QR code map. One location;

[0181] This indicates the camera's intrinsic parameters.

[0182] in, It contains the pose information of each long-range QR code in the scene. and the current camera's pose information on the map. ,exist In the middle, it can be found It is obtained by multiplying the conditional probabilities of a series of observation models. Therefore, all factors in the chain of products are separated from the state variables and analyzed as a separate class. This is the origin of factor graphs, in which different modules are divided into two parts: factors and variables. Each factor corresponds to a probability in the Bayesian network above, and each factor is only connected to nodes that are related to it. Therefore, the Bayesian network above is transformed into a factor graph, as shown below:

[0183]

[0184] To ensure that the factors are computable, following traditional methods, a Gaussian model is used as the basic model for the QR code attitude sensor:

[0185]

[0186] In the above formula, Represents the optimization variable. This represents the mean of the Gaussian function. This represents the Mahalanobis distance. Note. It is an operator, when When it is a vector, it represents subtraction between vectors. If Represented as When it is time, it is calculated according to the following formula:

[0187]

[0188] In the above formula, and They represent The rotation and translation components. This represents mapping a matrix to a Lie algebra. This represents a Lie algebra mapping that transforms a Lie algebra into a rotation matrix.

[0189] Absolute pose prior: Represented as a unary factor, it can be used to specify poses with noise. Prior posture

[0190]

[0191] Relative pose prior: Represented as a binary factor, if the transformation between two poses is known. ,

[0192]

[0193] First, the problem is modeled using a Bayesian network. The problem describes a mobile robot moving in a scenario where distant QR codes are deployed, and performing real-time mapping and localization. The modeling process considers not only the pose measurements of the distant QR codes and the camera's orientation, but also, to ensure mapping accuracy and eliminate the influence of singular QR code positions on incorrect localization, constraints such as pixel measurement factors for the distant QR codes, QR code geometric size constraints, and simultaneous camera capture of multiple QR codes are added to the model.

[0194]

[0195] Indicates the camera pose as And the observed first Under the condition of the code, the first The probability density function of the pose of a code in the map, where, Indicates that the camera is in the first position. Location Time camera The observation. This refers to the observation of a distant QR code in the camera coordinate system on a map.

[0196] It represents the probability density function of the current camera's pose, given the known pose of the distant QR code on the map and the effective observation of the distant QR code.

[0197] This represents the camera factor based on the pinhole camera model (i.e., the mathematical model of the camera, including the camera projection model and distortion model, and also including the camera's spatial location information on the map), based on the long-distance QR code. Location on the map The spatial center point of each positioning code circle in the long-distance QR code coordinate system. Transform the point to the map coordinate system. Locate the distant QR code pixel position as observed by the current camera. The observation function is established under the given conditions. Let represent the probability density function of the camera attitude under the above conditions.

[0198] This represents the geometric dimensional constraints of each long-range QR code in the map coordinate system. It describes the distance between the centers of the positioning codes of the long-range QR codes observed in the map coordinate system. Constraints between the actual size of the long-distance QR code and the actual size of the QR code.

[0199] Given two distant QR codes in an image, this represents the probability density function of the relative positional relationship between the two distant QR codes.

[0200] This indicates a priori information: when the camera sees the first distant QR code, the spatial position of the first distant QR code seen is initialized to the map origin.

[0201] The next step is to estimate the spatial location of the long-range QR code where the maximum posterior probability occurs, and to solve for all camera poses, i.e., to find a state variable that maximizes the probability value of the product of all factors.

[0202]

[0203] Therefore, the maximum a posteriori probability problem is transformed into a least squares problem, which can be solved using least squares methods. Common methods include gradient descent, Gauss-Newton method, Leweenberg-Marquardt method, and DogLeg method.

[0204] In actual deployment, not all constraints are used. As the mobile robot moves and the scene of long-distance QR code placement changes, multiple factors are added to the optimization function in turn to ensure the accuracy of mapping and the current camera pose.

[0205] Therefore, as the number of observation factors increases, the spatial location information of all QR codes and camera information is optimized using a factor map. This yields accurate spatial locations of the QR codes and camera positions, allowing the map update process described above to continue.

[0206] To further ensure the accuracy and real-time performance of the optimization, this example preferably uses the iSAM2 method to incrementally update the integrated map, which ensures that the map optimization and update are completed at the camera frame rate.

[0207] On the other hand, such as Figure 12 As shown, corresponding to the above method, the present invention also provides a map creation method system based on long-distance QR codes, which includes:

[0208] The storage unit is used to store a program including the steps of the map creation method based on long-distance QR codes as described above, for the control unit and processing unit to retrieve and execute as needed.

[0209] The control unit is used to control the infrared camera to capture images containing long-range QR codes;

[0210] The processing unit performs binarization on the image to be processed and then performs edge extraction. Based on the extracted edge information, it performs ellipse fitting detection to obtain the original code and its center coordinates. Then, based on the geometric relationship of the preset positioning codes of the long-range QR codes, it locates the distribution positions of the positioning codes corresponding to each long-range QR code in the original code to filter out the information codes in the original code. It also obtains the ID information of the long-range QR codes according to the passcode table. When a long-range QR code under a given ID appears for the first time, it calculates the spatial pose of the long-range QR code in the camera coordinate system based on the center coordinates of the positioning code in the image and stores the corresponding ID information in the storage unit. Then, it loads the spatial position relationship between the first long-range QR code and the camera into the map, using the current camera position as the origin of the map coordinate system. When the image observed by the camera contains the first observed long-range QR code B, it calculates the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image to obtain the position of the first observed long-range QR code B in the map and adds it to the map.

[0211] In summary, the map creation method and system based on long-distance QR codes provided by this invention cleverly utilizes the characteristic that a circular geometric shape, even with changes in camera angle, undergoes affine transformations in space, only becoming an ellipse while still retaining its center. Furthermore, it leverages the robust characteristic of a circle remaining circular / elliptical even when detection at long distances is prone to blurring. This allows for the design of QR codes suitable for long-distance detection. Corresponding to the aforementioned characteristics of long-distance QR codes, ellipse detection can accurately obtain the ellipse center to establish the pose transformation matrix with the camera. This solves the problem that traditional QR codes often appear as blurry clumps in the detection images when the distance and angle of the detection camera are far, leading to inaccurate recognition. Building upon this, this invention fuses the long-distance QR code information extracted from each frame of image using a factor graph and optimizes the pose of each long-distance QR code through incremental updates. This allows for more accurate calculation of the relative poses between long-distance QR codes in the image, obtaining the position of each code on the map, thereby realizing the creation of indoor QR code maps and ensuring the real-time performance and robustness of long-distance QR code mapping.

[0212] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

[0213] Those skilled in the art will understand that, besides implementing the system, apparatus, unit, and its modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and its modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0214] Furthermore, all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0215] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A method for creating a map based on long-range QR codes, comprising the following steps: Step S100: Set the remote QR code at the top of the mapping area, at an upward position relative to the camera; An image containing a long-range QR code is acquired by a camera to extract the spatial pose and ID information of the long-range QR code in the camera coordinate system; wherein the long-range QR code includes: a positioning code and an information code, wherein the positioning code is formed by several directional points arranged around a center point in several non-intersecting directions, and several quadrant regions are divided at the angle between each adjacent directional point; the information code is arranged in the corresponding quadrant region according to a preset code table to form a two-dimensional dot matrix with the positioning code; wherein the surfaces of the positioning code and the information code are provided with a reflective layer. Step S200: Load the spatial relationship between the first long-distance QR code and the camera into the map, and use the current location of the camera as the origin of the map coordinate system; Step S300: When the image observed by the camera contains the first observed long-range QR code B, calculate the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image, so as to obtain the position of the first observed long-range QR code B in the map and add it to the map.

2. The map creation method based on long-range QR codes according to claim 1, wherein step S300, calculating the relative pose between B and A, includes: Step S310 fuses the spatial pose of the long-range QR code extracted from each frame of image in the form of a factor graph. Step S320 optimizes the spatial pose of each long-distance QR code through incremental updates; Step S330 calculates the relative pose based on the optimized long-distance QR code spatial pose.

3. The map creation method based on long-range QR codes according to claim 2, wherein step S310, the step of fusing each frame of image to extract the spatial pose of the long-range QR code, includes: Step S311 uses the measured value Constraining the variables to be optimized Using Bayesian networks to model The optimized state variables are solved using the maximum a posteriori probability: ； in For the spatial pose of all distant QR codes on the map, the measured values A set of measurements, including: Indicates that the camera is in the first position. At each position, the camera... Observations; Represents the location code in a long-range QR code The distance between the centers; Represents the first in a long-distance QR code The pixel coordinates of the center of each positioning code circle; Represents the coordinate system of the long-distance QR code. The spatial location of the center of the positioning code; This indicates the camera's position on the distant QR code map. One location; This indicates the camera's intrinsic parameters.

4. The map creation method based on long-range QR codes according to claim 1, wherein step S100, the step of extracting the spatial pose and ID information of the long-range QR code in the camera coordinate system, includes: Step S110: After acquiring the image containing the long-distance QR code information, perform binarization processing and then edge extraction. Step S120 performs ellipse fitting detection based on the extracted edge information to obtain the original code and its center coordinates; Step S120: Based on the geometric relationship of the positioning codes, locate the distribution position of the positioning codes of each remote QR code in the original code, so as to filter out the information codes in the original code, and obtain the ID information of the remote QR code according to the code table. Step S140: When it is determined that the long-range QR code under the ID appears for the first time, calculate the spatial pose of the corresponding long-range QR code in the camera coordinate system according to the center coordinates of the positioning code in the image and bind it to the ID information.

5. The map creation method based on long-range QR codes according to claim 4, wherein step S140, calculating the spatial pose of the long-range QR code in the camera coordinate system, includes: Step S141 sets the center coordinates of the bit code as follows: Establish a homogeneous matrix ； The homography matrix H is solved using the SVD method, where u and v represent the pixel coordinates of the center of the location code. This represents the coordinates of a point in the long-distance QR code coordinate system, where s is the equivalent distance scaling factor. Step S142: Based on the relationship between the homography matrix and the transformation matrix of the long-range QR code in the camera coordinate system. ； The rotation and translation matrices are obtained, where P is the camera projection matrix and E is the truncation extrinsic parameter matrix. , Cameras Focal length in the y-direction and the x-direction. , Here are the coordinates of the camera center point. These are the first two columns of the rotation matrix. , , These represent the positions of the far-distance QR code center in the camera coordinate system. It is a 3×3 homography projection matrix.

6. The map creation method based on long-distance QR codes according to claim 5, wherein step S140 further includes: Step 143 solves for minimizing the error function using an iterative optimization method. ； To optimize the rotation and translation matrices, where R represents rotation and t represents translation. ； ； This represents the i-th element in the translation vector. Represents the i-th column in the rotation matrix; where These represent the first and second parts of the image, respectively. The coordinates of the center of a distant QR code projected onto the camera's normalized coordinate system. ; Represents the center coordinates of all the location codes; These represent the first and second parts of the long-distance QR code. The spatial coordinates of the center of each positioning code, transformed into the coordinates of a point in the camera's normalized plane using a rotation and translation matrix. ; This represents the spatial coordinates of the center of each location barcode in the long-range QR code coordinate system. ; In the solution process, As a constraint.

7. The map creation method based on long-distance QR codes according to claim 4, wherein the steps further include: Step S150 constrains the spatial pose obtained in step S140 according to time filtering and prior pose constraints. The time filtering step includes filtering out when there is a sudden change in the spatial pose result. The prior pose constraint step includes filtering out when the coordinate system of the far-field QR code and the camera coordinate system in the spatial pose result are not parallel.

8. The map creation method based on long-distance QR codes according to claim 4, wherein step S120 further includes: Based on the formulas for the major and minor axes of the ellipse equation, determine whether the ratio of the major and minor axes of each original code meets the threshold, and filter out those that do not.

9. A map creation method system based on long-distance QR codes, comprising: A storage unit is used to store a program including the steps of the map creation method based on long-distance QR codes as described in any one of claims 1 to 8, for the control unit and processing unit to retrieve and execute as needed. The control unit is used to control the infrared camera to capture images containing long-range QR codes; The processing unit performs binarization on the image to be processed and then performs edge extraction. Based on the extracted edge information, it performs ellipse fitting detection to obtain the original code and its center coordinates. Then, based on the geometric relationship of the preset positioning codes of the long-range QR codes, it locates the distribution positions of the positioning codes corresponding to each long-range QR code in the original code to filter out the information codes in the original code. It also obtains the ID information of the long-range QR codes according to the passcode table. When a long-range QR code under a given ID appears for the first time, it calculates the spatial pose of the long-range QR code in the camera coordinate system based on the center coordinates of the positioning code in the image and stores the corresponding ID information in the storage unit. Then, it loads the spatial position relationship between the first long-range QR code and the camera into the map, using the current camera position as the origin of the map coordinate system. When the image observed by the camera contains the first observed long-range QR code B, it calculates the relative pose between the first observed long-range QR code and the long-range QR code A already recorded in the image to obtain the position of the first observed long-range QR code B in the map and adds it to the map.

Images