A Mobile Robot Localization Method Based on CycleGAN Network and QR Code

Abstract

The present invention discloses a mobile robot positioning method based on CycleGAN network and two-dimensional code, including: calibrating a camera and an inertial measurement unit; acquiring a sequence of images; determining whether there is a two-dimensional code in the application scenario. If there is no two-dimensional code, input the sequence of images into the CycleGAN network to obtain an image with consistent brightness and extract key points, then perform optical flow tracking on the key points and remove mis-matched key points, and estimate the pose of the camera by using an extended Kalman filter algorithm according to the data collected by the inertial measurement unit to obtain the pose of the mobile robot; if there is a two-dimensional code, calculate the rotation matrix and translation matrix between the two-dimensional code and the camera based on the PNP algorithm according to the two-dimensional code, estimate the pose of the camera, and obtain the pose of the mobile robot according to the pose of the camera. This method helps to ensure that the mobile robot can work normally in a dark environment or an overexposed environment, and eliminates the cumulative error after long-term movement, ensuring the working stability and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot positioning, and particularly relates to a mobile robot positioning method based on the CycleGAN network and two-dimensional codes. Background Art

[0002] With the maturity of related technologies, mobile robots have been applied in a rich variety of scenarios. Due to the advantages of low cost and inexpensive price, cameras are increasingly widely used in the field of mobile robot positioning. However, visual positioning highly depends on light, and cameras cannot work in dark places or some textureless areas. A large change in light intensity during work will also have a relatively large impact on the pose estimation of mobile robots. Moreover, the positioning method of visual-inertial combination will also produce cumulative errors and is not suitable for use in long-term positioning scenarios. Therefore, this application proposes a mobile robot positioning method based on the CycleGAN network and two-dimensional codes. Summary of the Invention

[0003] The purpose of the present invention is to propose a mobile robot positioning method based on the CycleGAN network and two-dimensional codes for the above problems, which helps to ensure that the mobile robot can work normally in a dark environment or an exposure environment, eliminate cumulative errors after the mobile robot moves for a long time, and ensure the working stability and accuracy.

[0004] To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0005] A mobile robot positioning method based on the CycleGAN network and two-dimensional codes proposed by the present invention. The mobile robot includes a camera and an inertial measurement unit. The mobile robot positioning method based on the CycleGAN network and two-dimensional codes includes the following steps:

[0006] S1. Calibrate the camera and the inertial measurement unit to correct the measurement data;

[0007] S2. Use the calibrated camera to collect the application scenario video and convert it into a sequence of images;

[0008] S3. Based on the two-dimensional code corner detection algorithm, judge whether two-dimensional codes are deployed in the application scenario according to the sequence of images. If not, execute step S4; otherwise, record the absolute three-dimensional coordinates of the two-dimensional codes and execute step S7;

[0009] S4. Input the sequence of images into the CycleGAN network to obtain images with consistent brightness, and extract the key points of the images. The total loss function L(G AB , G BA , D A , D B ) of the CycleGAN network is as follows:

[0010] L(G AB ,G BA ,D A ,D B )=L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)+γ cyc L cyc (G AB ,G BA )+γ t L t (G AB ,G BA )

[0011] in,

[0012]

[0013] Where, L GAN To counter the loss function, L cyc is the cycle consistency loss function, L t is the temporal consistency loss function, G AB Denotes the first generator, G BA Denotes the second generator, D A represents the first discriminator, D B represents the second discriminator, A represents the image under normal lighting, B represents the image under abnormal lighting, γ cyc is a constant that controls the relative importance of the cycle consistency loss function, γ t is a constant that controls the relative importance of the temporal consistency loss function, a t-1 is the original image of frame t-1, a t is the original image of frame t, is the original image a t-1 to a t The optical flow function, is the image obtained after one cycle of the original image of the t-th frame, The image obtained after one cycle of the original image of frame t-1 is: For images arrive The optical flow function, F(*) represents the L1 norm;

[0014] S5. Use the FlowNet network to perform optical flow tracking on the key points of the image, and use the RANSAC algorithm to remove the mismatched key points in adjacent images;

[0015] S6. Based on the data collected by the calibrated inertial measurement unit, use the extended Kalman filter algorithm to estimate the pose of the camera, obtain the pose of the mobile robot according to the pose of the camera, and return to execute step S2 until the mobile robot finishes running;

[0016] S7. Scan the QR code through the camera to obtain the position of the QR code, and calculate the rotation matrix and translation matrix between the QR code and the camera based on the size and position of the QR code using the PNP algorithm, estimate the pose of the camera, obtain the pose of the mobile robot according to the pose of the camera, and return to execute step S2 until the mobile robot finishes running.

[0017] Preferably, calibrate the camera and the inertial measurement unit as follows:

[0018] S11. Use the checkerboard calibration board and the camera_calibration calibration package to calibrate the camera to obtain the internal parameters and distortion parameters of the camera;

[0019] S12. Calibrate the inertial measurement unit using deterministic errors and random errors. The random errors are obtained through Allan variance analysis using the IMU_UTILS tool, and the deterministic errors are obtained by querying the factory parameters;

[0020] S13. Start the camera and synchronously rotate the inertial measurement unit to collect data packets, start the Kalibr calibration package, and use the QR code calibration board, data packets, internal parameters, distortion parameters, deterministic errors and random errors to jointly calibrate the camera and the inertial measurement unit.

[0021] Preferably, the pattern array size of the checkerboard calibration board is 7*9.

[0022] Preferably, the random errors include gyroscope noise, gyroscope drift, accelerometer noise and accelerometer drift.

[0023] Preferably, the sequence of images consists of images extracted from the video every 10 ms.

[0024] Preferably, the key points of the image are Harris corners extracted using the Harris corner detection algorithm.

[0025] Preferably, the adversarial loss function L GAN Adopt the cross-entropy loss function, and the cycle consistency loss function L cyc Adopt the L1 norm.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: This method can perform mobile robot positioning based on the CycleGAN network and two-dimensional codes. The CycleGAN network is used to unify the image brightness of the camera at a certain level, and a temporal consistency loss function is introduced into the CycleGAN network to ensure the temporal continuity of the image sequence. Moreover, it can ensure that the mobile robot can work properly in dark environments or overexposed environments, improving the robustness of positioning, solving the cumulative positioning error generated after the long-term movement of the mobile robot, and using two-dimensional codes to assist in eliminating the cumulative error to ensure the positioning accuracy, enabling the mobile robot to work more stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flowchart of the mobile robot positioning method based on the CycleGAN network and two-dimensional codes of the present invention;

[0028] Figure 2 is a schematic structural diagram of the CycleGAN network of the present invention;

[0029] Figure 3 is a schematic structural diagram of the checkerboard calibration board of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0031] It should be noted that unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present application. The terms used in the specification of the present application herein are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

[0032] The present application can solve the problems that the pose estimation of the mobile robot is inaccurate in the dark or under large light intensity changes, and the cumulative error generated after long-term movement affects the positioning accuracy.

[0033] As Figures 1 - 3 shown, a mobile robot positioning method based on the CycleGAN network and two-dimensional codes, the mobile robot includes a camera and an inertial measurement unit, and the mobile robot positioning method based on the CycleGAN network and two-dimensional codes includes the following steps:

[0034] S1. Calibrate the camera and the inertial measurement unit to correct the measurement data.

[0035] In one embodiment, the camera and the inertial measurement unit are calibrated as follows:

[0036] S11. Calibrate the camera using a checkerboard calibration board and the camera_calibration calibration package to obtain the internal parameters and distortion parameters of the camera;

[0037] S12. Calibrate the inertial measurement unit using deterministic errors and random errors. The random errors are obtained through Allan variance analysis using the IMU_UTILS tool, and the deterministic errors are obtained by querying the factory parameters;

[0038] S13. Start the camera and synchronously rotate the inertial measurement unit to collect data packets. Start the Kalibr calibration package, and use the QR code calibration board, data packets, internal parameters, distortion parameters, deterministic errors and random errors to perform joint calibration of the camera and the inertial measurement unit.

[0039] In one embodiment, the pattern array size of the checkerboard calibration board is 7*9.

[0040] In one embodiment, the random errors include gyroscope noise, gyroscope drift, accelerometer noise and accelerometer drift.

[0041] S2. Use the calibrated camera to collect the application scenario video and convert it into a sequence of images.

[0042] In one embodiment, the sequence of images is composed of images extracted from the video every 10 ms. Or it can be adjusted according to actual needs.

[0043] S3. Based on the QR code corner detection algorithm, determine whether a QR code is deployed in the application scenario according to the sequence of images. If not, execute step S4; otherwise, record the absolute three-dimensional coordinates of the QR code and execute step S7.

[0044] S4. Input the sequence of images into the CycleGAN network to obtain images with consistent brightness, and extract the key points of the images. The total loss function L(G AB ,G BA ,D A ,D B ) of the CycleGAN network is as follows:

[0045] L(G AB ,G BA ,D A ,D B ) = L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A, B, A) + γ cyc L cyc (G AB , G BA ) + γ t L t (G AB , G BA )

[0046] Among them,

[0047]

[0048] In the formula, L GAN is the adversarial loss function, L cyc is the cycle consistency loss function, L t is the temporal consistency loss function, G AB represents the first generator, G BA represents the second generator, D A represents the first discriminator, D B represents the second discriminator, A represents the image under normal illumination, B represents the image under abnormal illumination, γ cyc is a constant that controls the relative importance of the cycle consistency loss function, γ t is a constant that controls the relative importance of the temporal consistency loss function, a t-1 is the original image of the (t - 1)-th frame, a t is the original image of the t-th frame, is the original image a t-1 to a t 's optical flow function, is the image obtained by one cycle of the original image of the t-th frame, is the image obtained by one cycle of the original image of the (t - 1)-th frame, is the image to 's optical flow function, F(*) represents the L1 norm.

[0049] In one embodiment, the key points of the image are Harris corners extracted by using the Harris corner detection algorithm.

[0050] In one embodiment, the adversarial loss function L GAN adopts the cross-entropy loss function, and the cycle consistency loss function L cyc adopts the L1 norm.

[0051] S5. Use the FlowNet network to perform optical flow tracking on the key points of the image, and adopt the RANSAC algorithm to remove the mis-matched key points in adjacent images.

[0052] S6. Based on the data collected by the calibrated inertial measurement unit, use the extended Kalman filter algorithm to estimate the pose of the camera. Obtain the pose of the mobile robot according to the pose of the camera, and return to execute step S2 until the mobile robot finishes running.

[0053] S7. Scan the QR code through the camera to obtain the position of the QR code, and calculate the rotation matrix and translation matrix between the QR code and the camera based on the size and position of the QR code using the PNP algorithm. Estimate the pose of the camera, obtain the pose of the mobile robot according to the pose of the camera, and return to execute step S2 until the mobile robot finishes running.

[0054] The following is a detailed description through specific embodiments.

[0055] The specific steps of step 1 include:

[0056] Step 1.1: Use a 7*9 checkerboard calibration board, as Figure 3 shown. After starting the camera, complete the camera calibration using the camera_calibration calibration package to obtain the internal parameters and distortion parameters of the camera, where:

[0057] The internal parameters are shown in Table 1:

[0058] Table 1

[0059] <![CDATA[f x > <![CDATA[f y > <![CDATA[c x > <![CDATA[c y > 430.7047 430.6445 293.4685 256.0733

[0060] In Table 1, f x represents the product of the camera focal length and each physical dimension in the horizontal axis direction, f y represents the product of the camera focal length and each physical dimension in the vertical axis direction, c x represents the lateral drift of the image origin relative to the imaging point of the optical center, c y represents the longitudinal drift of the image origin relative to the imaging point of the optical center.

[0061] The distortion parameters are shown in Table 2:

[0062] Table 2

[0063] <![CDATA[K1]]> <![CDATA[K2]]> <![CDATA[K3]]> <![CDATA[P1]]> <![CDATA[P2]]> -0.0209 -0.1260 -0.0050 -0.0003 -0.1636

[0064] In Table 2, K1, K2, and K3 represent radial distortion, and P1 and P2 represent tangential distortion.

[0065] Step 1.2: The calibration parameters of the inertial measurement unit consist of deterministic errors and random errors. The deterministic errors can be found from the factory parameters. Use the IMU_UTILS tool to perform Allan variance analysis to obtain the parameters of the random errors, as shown in Table 3:

[0066] Table 3

[0067] Gyroscope noise Gyroscope drift Accelerometer noise Accelerometer drift 0.0078 0.0036 0.016 0.015

[0068] Step 1.3: Print the aprilgrid image, make a QR code calibration board, start the camera and fully rotate the inertial measurement unit so that it rotates along three axes, and collect the corresponding images or measurement data to form a data packet. Start the Kalibr calibration package, input the calibration files of the camera and the inertial measurement unit obtained in Step 1.1 and Step 1.2, perform calibration, and obtain the rotation matrix R and translation matrix t of the IMU relative to the camera:

[0069]

[0070] t = [0.0039 0.046 0.022]

[0071] Step 2: Use the calibrated camera to collect the application scenario video and extract images every 10 ms to form a sequence of images.

[0072] Step 3: As Figure 1 shown, based on the sequence of images, use the QR code corner detection algorithm to determine whether there is a QR code deployed in the application scenario. If there is no QR code in the application scenario, execute Step 4. If there is a QR code in the application scenario, such as making a QR code containing absolute position information and deploying it in the application scenario, record the absolute three-dimensional coordinates of the QR code, and execute Step 7.

[0073] The specific steps of Step 4 include:

[0074] Step 4.1: Input the sequence of images into a pre-trained CycleGAN network to obtain images with consistent brightness. The CycleGAN network framework is as Figure 2 shown, where G AB and G BA are both generators, D A and D B are both discriminators. The input to generator G AB is the image A under normal light, that is, a pre-set image, and the input to generator G BA is the image B under abnormal light, that is, the image captured by the camera during movement. Continuously input the camera images to obtain images with consistent brightness. In this application, the CycleGAN network introduces a temporal consistency loss to ensure the temporal continuity of the image sequence, and uses the L1 norm loss function as the measurement value. The overall loss of the finally improved CycleGAN network consists of the adversarial loss L GAN , the cycle consistency loss L cyc and the temporal consistency loss L t .

[0075] Step 4.2: After obtaining images with the same brightness information, extract Harris corners as key points.

[0076] The specific steps of Step 5 include:

[0077] Step 5.1: Since real-time performance is required and there are small displacements in the input image sequence, the FlowNet network (such as the FlowNet2-css-ft-sd optical flow model) is used to solve the small displacement problem.

[0078] Step 5.2: Remove the mismatches of the tracked harris corners through the RANSAC algorithm.

[0079] Step 6: According to the data collected by the calibrated inertial measurement unit, the extended Kalman filter algorithm is used to estimate the pose of the camera. Based on the pose of the camera, the pose of the mobile robot is obtained, and Step 2 is returned to execute until the mobile robot finishes running. During the running of the mobile robot, given the position relationship between the camera and the mobile robot, the pose of the mobile robot can be obtained according to the pose of the camera, and the cumulative error during the running process is repaired.

[0080] Step 7: As Figure 1 shown, if there is a QR code in the application scenario, use the camera to scan the QR code to obtain the absolute three-dimensional coordinates of the QR code. Since the size of the QR code is known, the rotation matrix and translation matrix between the QR code and the camera can be calculated through the size and position of the QR code, and then the pose of the camera can be obtained. During the running of the mobile robot, given the position relationship between the camera and the mobile robot, the pose of the mobile robot can be obtained according to the pose of the camera, and the cumulative error during the running process is repaired.

[0081] This method can perform mobile robot positioning based on the CycleGAN network and QR codes. The CycleGAN network is used to unify the image brightness of the camera at a certain level, and a temporal consistency loss function is introduced into the CycleGAN network to ensure the temporal continuity of the image sequence, and it can ensure that the mobile robot can work normally in dark or overexposed environments, improving the robustness of positioning, solving the positioning cumulative error generated after the long-term movement of the mobile robot, and using the QR code to assist in eliminating the cumulative error to ensure the positioning accuracy, enabling the mobile robot to work more stably.

[0082] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0083] The above-described embodiments merely represent relatively specific and detailed embodiments of the present application, but should not be construed as limiting the scope of the patented application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A mobile robot positioning method based on a CycleGAN network and a QR code, wherein the mobile robot includes a camera and an inertial measurement unit, characterized in that: The mobile robot positioning method based on the CycleGAN network and the QR code comprises the following steps: S1. Calibrate the camera and inertial measurement unit to correct the measurement data; S2, using the calibrated camera to capture the application scene video and convert it into a sequence of images; S3. Determine whether a QR code is deployed in the application scene based on the QR code corner detection algorithm according to the sequence image. If not, execute step S4. Otherwise, record the absolute three-dimensional coordinates of the QR code and execute step S7. S4. Input the sequence image into the CycleGAN network to obtain an image with consistent brightness, and extract the key points of the image. The total loss function L(G AB ,G BA ,D A ,D B )The formula is as follows: L(G AB ,G BA ,D A ,D B )=L GAN (G AB ,D B ,A,B)+L GAN (G BA ,D A ,B,A)+γ cyc L cyc (G AB ,G BA )+γ t L t (G AB ,G BA ) in, Where, L GAN To counter the loss function, L cyc is the cycle consistency loss function, L t is the temporal consistency loss function, G AB Denotes the first generator, G BA Denotes the second generator, D A represents the first discriminator, D B represents the second discriminator, A represents the image under normal lighting, B represents the image under abnormal lighting, γ cyc is a constant that controls the relative importance of the cycle consistency loss function, γ t is a constant that controls the relative importance of the temporal consistency loss function, a t-1 is the original image of frame t-1, a t is the original image of frame t, is the original image a t-1 to a t The optical flow function, is the image obtained after one cycle of the original image of the t-th frame, The image obtained after one cycle of the original image of frame t-1 is: For images arrive The optical flow function, F(*) represents the L1 norm; S5. Use the FlowNet network to perform optical flow tracking on the key points of the image, and use the RANSAC algorithm to remove the mismatched key points in adjacent images; S6. Estimate the camera's pose using the extended Kalman filter algorithm based on the data collected by the calibrated inertial measurement unit, obtain the pose of the mobile robot based on the camera's pose, and return to step S2 until the mobile robot finishes running. S7. Scan the QR code with a camera to obtain the position of the QR code, and calculate the rotation matrix and translation matrix between the QR code and the camera based on the PNP algorithm according to the size and position of the QR code, estimate the position of the camera, and obtain the position of the mobile robot according to the position of the camera. Return to step S2 and execute until the mobile robot finishes running.

2. The mobile robot positioning method based on the CycleGAN network and the QR code according to claim 1, characterized in that: The process of calibrating the camera and inertial measurement unit is as follows: S11. Calibrate the camera using the checkerboard calibration plate and the camera_calibration calibration package to obtain the camera's intrinsic parameters and distortion parameters. S12. Calibrate the inertial measurement unit using deterministic error and random error, where the random error is obtained by performing Allen variance analysis using the IMU_UTILS tool, and the deterministic error is obtained by querying factory parameters; S13. Start the camera and synchronously rotate the inertial measurement unit to collect data packets, start the Kalibr calibration package, and use the QR code calibration plate, data packets, internal parameters, distortion parameters, deterministic errors, and random errors to jointly calibrate the camera and inertial measurement unit.

3. The mobile robot positioning method based on the CycleGAN network and the QR code according to claim 2, characterized in that: The pattern array size of the checkerboard calibration plate is 7*9.

4. The mobile robot positioning method based on the CycleGAN network and the QR code according to claim 2, characterized in that: The random errors include gyroscope noise, gyroscope drift, accelerometer noise and accelerometer drift.

5. The mobile robot positioning method based on CycleGAN network and QR code according to claim 1, characterized in that: The sequence of images is composed of images extracted from the video every 10ms.

6. The mobile robot positioning method based on the CycleGAN network and the QR code according to claim 1, characterized in that: The key points of the image are Harris corner points extracted using the Harris corner detection algorithm.

7. The mobile robot positioning method based on CycleGAN network and QR code according to claim 1, characterized in that: The adversarial loss function L GAN Using the cross entropy loss function, the cycle consistency loss function L cyc The L1 norm is used.

Images