A calibration method, device and electronic device

Abstract

An embodiment of the present invention provides a calibration method, device, and electronic device, which are applied to the field of robotics. The method includes: obtaining the image positions and global positions of the first visual beacons included in each first image frame, and obtaining the first odometry information collected by an odometer carried by a mobile robot during the movement of the mobile robot within the first visual beacon area, and determining a homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the obtained first odometry information, the image positions and global positions of the first visual beacons. Through this solution, the efficiency of parameter calibration of the mobile robot can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and particularly to a calibration method, device and electronic device. Background Art

[0002] In mobile robots, parameter calibration is a very crucial link, and the accuracy of the calibration result directly affects the accuracy of the work of the mobile robot. Therefore, doing a good job in the parameter calibration of the mobile robot is the prerequisite for doing subsequent work and the key to improving the accuracy of the work of the mobile robot.

[0003] For a mobile robot, its parameter calibration mainly includes the calibration of the camera internal parameters and the external parameters between the camera and the odometer. In related calibration methods, only one parameter can be calibrated each time. For example, only the calibration of the camera internal parameters or the external parameters between the camera and the odometer can be carried out each time, making the calibration work cumbersome and the efficiency low. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a calibration method, device and electronic device to improve the efficiency of parameter calibration of mobile robots. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present invention provide a calibration method, and the method includes:

[0006] Obtain the image positions and global positions of the first visual beacons included in each first image frame, where each first image frame is: an image frame collected by a camera carried by the mobile robot during the movement of the mobile robot in the first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the image frame to which it belongs; the global position of each visual beacon is the position of the visual beacon in the world coordinate system;

[0007] Obtain the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area;

[0008] Based on the obtained first odometry information, the image positions and global positions of each first visual beacon, determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0009] Optionally, the determining the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the obtained first odometry information, the image positions and global positions of each first visual beacon includes:

[0010] For the first visual beacon, based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon, determine the reprojection error of the first visual beacon; wherein, the target transformation function is: a function of the parameters to be calibrated, which indicates the coordinate transformation relationship between the pixel plane of the camera and the body coordinate system plane of the odometer;

[0011] If the preset iteration end condition is not satisfied, then based on the reprojection errors of the first visual beacons, adjust the parameters in the target transformation function, and return to execute the step of determining the reprojection error of the first visual beacon based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon;

[0012] If the iteration end condition is satisfied, then use the target transformation function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0013] Optionally, the determining the reprojection error of the first visual beacon based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon includes:

[0014] Based on the target transformation function, project the image position of the first visual beacon onto the body coordinate system plane to obtain a first projection position;

[0015] Based on the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, project the first projection position into the global coordinate system to obtain a second projection position;

[0016] Take the difference information between the second projection position and the global position of the first visual beacon as the reprojection error of the first visual beacon.

[0017] Optionally, the preset iteration end condition includes at least one of the following conditions:

[0018] The total reprojection error is less than the first error threshold; the total reprojection error is the sum of the reprojection errors of the first visual beacons;

[0019] The total reprojection error reaches the first minimum value, where the first minimum value is the minimum value that the total reprojection error can reach after continuous iteration;

[0020] The number of iterations performed is greater than the first number threshold.

[0021] Optionally, obtaining the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area includes:

[0022] Obtaining the original encoded data of the odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area; wherein, the original encoded data is the data collected by the encoder within the odometer;

[0023] Based on the internal parameters of the odometer, converting the original encoded data into odometry information as the first odometry information.

[0024] Optionally, the internal parameters of the odometer are internal parameters to be calibrated;

[0025] Before adjusting the parameters in the target conversion function based on the reprojection errors of the first visual beacons, the method further includes:

[0026] For each image frame, based on the image position and the global position of the first visual beacon in the image frame, determining the pose of the camera when collecting the image frame as the first pose of the odometer at the acquisition moment of the image frame, and based on the first odometry information, determining the second pose of the odometer at the acquisition moment of the image frame;

[0027] For each image frame, calculating the difference information between the first pose and the second pose of the odometer at the acquisition moment of the image frame as the motion error of the odometer at the acquisition moment of the image frame;

[0028] The adjusting the parameters in the target conversion function based on the reprojection errors of the first visual beacons includes:

[0029] Based on the reprojection errors of the first visual beacons and the motion errors of the odometer at the acquisition moments of the image frames, adjusting the parameters in the target conversion function and the internal parameters of the odometer.

[0030] Optionally, the preset iteration end condition includes at least one of the following conditions:

[0031] The sum of the total reprojection error and the total motion error is less than a second error threshold, where the total reprojection error is the sum of the reprojection errors of the first visual beacons, and the total motion error is the sum of the motion errors of the odometer at the acquisition moments of the image frames;

[0032] The sum of the reprojection total error and the motion total error reaches a second minimum value, where the second minimum value is the minimum value that the sum of the reprojection total error and the motion total error can reach after continuous iteration;

[0033] The number of iterations is greater than a second number threshold.

[0034] Optionally, after determining the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first odometry information, the image positions and global positions of the first visual beacons, the method further includes:

[0035] Based on the homography matrix, controlling the mobile robot to move in a second visual beacon area according to a preset motion mode; wherein, the accuracy of each second visual beacon in the second visual beacon area is greater than the accuracy of each first visual beacon in the first visual beacon area;

[0036] Obtaining the image positions and global positions of the second visual beacons included in the second image frame collected by the camera, and the second odometry information collected by the odometer, during the movement of the mobile robot in the second visual beacon area;

[0037] Optimizing the homography matrix based on the acquired second odometry information, the image positions and global positions of the second visual beacons to obtain an optimized homography matrix.

[0038] In a second aspect, an embodiment of the present invention provides a calibration device, and the device includes:

[0039] A position acquisition module, configured to acquire the image positions and global positions of the first visual beacons included in each first image frame, where each first image frame is: an image frame collected by a camera carried by the mobile robot during the movement of the mobile robot in a first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the image frame to which it belongs; the global position of each visual beacon is the position of the visual beacon in the world coordinate system;

[0040] An information acquisition module, configured to acquire the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area;

[0041] A matrix determination module, configured to determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first odometry information, the image positions and global positions of the first visual beacons.

[0042] Optionally, the matrix determination module includes:

[0043] An information determination sub-module, configured to determine a reprojection error of the first visual beacon based on a target conversion function, first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon; wherein, the target conversion function is: a function of the parameters to be calibrated, which is used to indicate the coordinate conversion relationship between the pixel plane of the camera and the body coordinate system plane of the odometer;

[0044] A parameter adjustment sub-module, configured to, if a preset iteration end condition is not satisfied, adjust each parameter in the target conversion function based on the reprojection errors of the first visual beacons, and call the information determination sub-module;

[0045] A matrix determination sub-module, configured to, if the iteration end condition is satisfied, use the target conversion function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0046] Optionally, the information determination sub-module is specifically configured to project the image position of the first visual beacon onto the body coordinate system plane based on the target conversion function to obtain a first projection position; project the first projection position into the global coordinate system based on the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs to obtain a second projection position; use the difference information between the second projection position and the global position of the first visual beacon as the reprojection error of the first visual beacon.

[0047] Optionally, the preset iteration end condition includes at least one of the following conditions: the total reprojection error is less than a first error threshold; the total reprojection error is the sum of the reprojection errors of the first visual beacons; the total reprojection error reaches a first minimum value, where the first minimum value is the minimum value that the total reprojection error can reach after continuous iteration; the number of iterations is greater than a first number threshold.

[0048] Optionally, the information acquisition module includes:

[0049] A data acquisition sub-module, configured to acquire the original encoded data of the odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area; wherein, the original encoded data is the data collected by the encoder in the odometer;

[0050] An information conversion sub-module, configured to convert the original encoded data into odometry information based on the internal parameters of the odometer as the first odometry information.

[0051] Optionally, the internal parameters of the odometer are the internal parameters to be calibrated; the device further includes:

[0052] A motion error determination module, configured to, before the parameter adjustment sub-module adjusts each parameter in the target conversion function based on the reprojection errors of the first visual beacons, for each image frame, determine the pose of the camera when collecting this image frame based on the image position and the global position of the first visual beacon in this image frame, as the first pose of the odometer at the collection moment of this image frame, and determine the second pose of the odometer at the collection moment of this image frame based on the first odometry information; for each image frame, calculate the difference information between the first pose and the second pose of the odometer at the collection moment of this image frame, as the motion error of the odometer at the collection moment of this image frame.

[0053] The parameter adjustment sub-module is specifically configured to adjust each parameter in the target conversion function and the internal parameters of the odometer based on the reprojection errors of the first visual beacons and the motion errors of the odometer at the collection moments of each image frame.

[0054] Optionally, the preset iteration end condition includes at least one of the following conditions: the sum of the total reprojection error and the total motion error is less than a second error threshold, where the total reprojection error is the sum of the reprojection errors of the first visual beacons, and the total motion error is the sum of the motion errors of the odometer at the collection moments of each image frame; the sum of the total reprojection error and the total motion error reaches a second minimum value, and the second minimum value is the minimum value that the sum of the total reprojection error and the total motion error can reach after continuous iteration; the number of iterations is greater than a second number threshold.

[0055] Optionally, the device further includes: a matrix optimization module, configured to, after the matrix determination module determines the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first odometry information, the image positions and the global positions of the first visual beacons, control the mobile robot to move in a preset motion manner within the second visual beacon area based on the homography matrix; where the accuracy of each second visual beacon in the second visual beacon area is greater than the accuracy of each first visual beacon in the first visual beacon area; obtain the image positions and the global positions of the second visual beacons included in the second image frame collected by the camera and the second odometry information collected by the odometer during the movement of the mobile robot within the second visual beacon area; optimize the homography matrix based on the acquired second odometry information, the image positions and the global positions of the second visual beacons to obtain an optimized homography matrix.

[0056] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0057] The memory is used to store a computer program;

[0058] The processor is configured to implement the method steps described in any one of the first aspects when executing the program stored on the memory.

[0059] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0060] Beneficial effects of the embodiments of the present invention:

[0061] In the calibration method provided by the embodiments of the present invention, the image positions and global positions of the first visual beacons included in each first image frame can be obtained, and the first odometry information collected by the odometer carried by the mobile robot during the movement in the first visual beacon area can be obtained. And based on the obtained first odometry information, the image positions and global positions of each first visual beacon, a homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer is determined. Since the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer includes the internal parameters of the camera and the external parameters between the camera and the odometer, the process of determining the homography matrix based on the obtained first odometry information, the image positions and global positions of each first visual beacon can be understood as a process of calibrating the internal parameters of the camera and the external parameters between the camera and the odometer at the same time. It can be seen that through this solution, it is no longer necessary to separately calibrate the internal parameters of the camera and the external parameters between the camera and the odometer, thereby simplifying the process of calibrating the parameters of the mobile robot and improving the efficiency of calibrating the parameters of the mobile robot.

[0062] Of course, when implementing any product or method of the present invention, it is not necessarily required to achieve all the above-mentioned advantages at the same time. Description of the drawings

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative efforts.

[0064] Figure 1 It is a flowchart of the calibration method provided by the embodiments of the present invention;

[0065] Figure 2 Schematic diagram of the first visual beacon area provided by an embodiment of the present invention;

[0066] Figure 3 Another flowchart of the calibration method provided by an embodiment of the present invention;

[0067] Figure 4 Another flowchart of the calibration method provided by an embodiment of the present invention;

[0068] Figure 5 Schematic diagram of the second visual beacon area provided by an embodiment of the present invention;

[0069] Figure 6 Schematic structural diagram of the calibration device provided by an embodiment of the present invention;

[0070] Figure 7 Schematic structural diagram of the electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0071] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0072] In a mobile robot, parameter calibration is a very crucial link, and the accuracy of its calibration result directly affects the working accuracy of the mobile robot. Therefore, doing a good job in mobile robot parameter calibration is the premise for doing subsequent work well, and improving the calibration accuracy is the focus of scientific research work.

[0073] For a mobile robot, its parameter calibration mainly includes the calibration of the camera internal parameters and the external parameters between the camera and the odometer. In related calibration methods, only one parameter can be calibrated each time. For example, only the calibration of the camera internal parameters or the external parameters between the camera and the odometer can be performed each time, making the calibration work cumbersome and inefficient.

[0074] To improve the efficiency of mobile robot parameter calibration, embodiments of the present invention provide a calibration method, device, and electronic device.

[0075] It should be noted that in specific applications, the embodiments of the present invention can be applied to various electronic devices. For example, personal computers, servers, mobile phones, and other devices with data processing capabilities. And the calibration method provided by the embodiments of the present invention can be implemented in a software, hardware, or software-hardware combination manner

[0076] In one embodiment, the embodiment of the present invention can be applied to a mobile robot, such as an AGV (Automated Guided Vehicle), a two-wheel differential mobile robot, etc. The mobile robot may include a camera and an odometer, and the odometer may include an encoder, such as a photoelectric encoder, etc. For a mobile robot, the working principle of the odometer is to detect the arc of the wheel within a certain period of time based on the encoders installed on the left and right driving wheel motors, and then calculate the change in the relative posture of the mobile robot.

[0077] The calibration method provided by the embodiment of the present invention may include:

[0078] Acquire the image position and global position of the first visual beacon contained in each first image frame, wherein each first image frame is: an image frame captured by a camera carried by the mobile robot during the movement of the mobile robot in the first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the corresponding image frame; and the global position of each visual beacon is the position of the visual beacon in the world coordinate system;

[0079] Acquire first mileage information collected by an odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area;

[0080] Based on the acquired first mileage information, the image position and the global position of each first visual beacon, a homography matrix between a pixel plane of the camera and a body coordinate system plane of the odometer is determined.

[0081] In the above solution provided by the embodiment of the present invention, since the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer contains the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer, the process of determining the homography matrix based on the acquired first mileage information, the image position of each first visual beacon and the global position can be understood as the process of calibrating the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer at the same time. It can be seen that through this solution, it is no longer necessary to calibrate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer separately, thereby simplifying the process of mobile robot parameter calibration and improving the efficiency of mobile robot parameter calibration.

[0082] A calibration method provided by an embodiment of the present invention is introduced below with reference to the accompanying drawings.

[0083] like Figure 1 As shown, a calibration method provided by an embodiment of the present invention may include the following steps:

[0084] S101, obtaining an image position and a global position of a first visual beacon included in each first image frame;

[0085] Among them, each first image frame is: an image frame collected by a camera carried by a mobile robot during the movement of the mobile robot within a first visual beacon area. The image position of each visual beacon is the position of the visual beacon in the image frame to which it belongs; the global position of each visual beacon is the position of the visual beacon in the world coordinate system. Optionally, the layout requirement of the first visual beacon area is that the code spacing is smaller than the size of the camera's field of view, so that when the mobile robot moves within the first visual beacon area, the camera can at least collect an image frame containing at least one visual beacon.

[0086] Among them, as Figure 2 shown, an embodiment of the present invention provides a schematic diagram of a first visual beacon area. Figure 2 Each rectangular block in it represents a visual beacon, and the visual beacons form a visual beacon matrix as the first visual beacon area. Each visual beacon contains a two-dimensional code, and the position information of the visual beacon in the world coordinate system is recorded in the two-dimensional code information.

[0087] When the mobile robot moves within the first visual beacon area, the camera carried by the mobile robot can perform image acquisition on the first visual beacon area, so that an image frame containing at least one visual beacon can be collected. For the sake of distinction, in the embodiment of the present invention, the image frame collected by the camera carried by the mobile robot during the movement of the mobile robot within the first visual beacon area is referred to as the first image frame.

[0088] For any visual beacon, the image position of the visual beacon is the position of the visual beacon in the image frame to which it belongs, that is, the position of the visual beacon in the pixel coordinate system of the camera. The position of the visual beacon in the image frame to which it belongs can be the pixel coordinates of the center point of the visual beacon corresponding to the pixel point in the image frame, or the position of the visual beacon in the image frame to which it belongs can also be the pixel coordinates of at least one of the four corner points of the visual beacon corresponding to the pixel point in the image frame, which is all acceptable.

[0089] For any visual beacon, the global position of the visual beacon is the position of the visual beacon in the world coordinate system. Among them, the world coordinate system can be a global coordinate system constructed in advance for the scene where the mobile robot is located, and its origin position, X-axis direction, Y-axis direction and Z-axis direction can all be determined based on experience and needs. Of course, it can also be determined by a specific global coordinate system determination method, and the embodiment of the present invention does not specifically limit this. The global position of each visual beacon can be recorded in the visual beacon. For example, if the visual beacon contains a QR code, the global position of the visual beacon can be written into its QR code, so that the global position of the visual beacon can be read by reading the QR code in the visual beacon. Specifically, similar to the image position, the global position of the visual beacon can be the position of the center point or at least one corner point of the visual beacon in the world coordinate system. In order to improve the accuracy of the unit matrix, the image position of the visual beacon can be matched with the full text position of the visual beacon. For example, if the image position is the pixel coordinates of the pixel corresponding to the center point, the global position should be the coordinates of the center point in the world coordinate system.

[0090] In this step, the image frame captured by the camera can be obtained first, and then for any visual beacon in the image frame, the position of the visual beacon in the image frame can be determined as the image position of the visual beacon, and the position of the visual beacon in the world coordinate system carried by the visual beacon can be obtained by reading the information of the visual beacon, as the global position of the visual beacon.

[0091] S102, obtaining first mileage information collected by an odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area;

[0092] The first mileage information collected by the odometer carried by the mobile robot may be the position and posture of the mobile robot at each sampling moment during the movement of the mobile robot, as recorded by the odometer in the mobile robot.

[0093] In one implementation, the mileage information recorded by the mileage meter during the movement of the mobile robot in the first visual beacon area can be directly read from the mileage meter as the first mileage information.

[0094] Alternatively, in another implementation, the first mileage information may also be determined based on the original coded data collected by the encoder in the mileage meter and the internal parameters of the mileage meter. Optionally, the original coded data of the mileage meter carried by the mobile robot may be obtained during the movement of the mobile robot in the first visual beacon area, and then the original coded data may be converted into mileage information based on the internal parameters of the mileage meter as the first mileage information. The original coded data is the data collected by the encoder in the mileage meter.

[0095] Among them, when the odometer is a differential wheel odometer, the data collected by the encoder in the above odometer are the rotation radians of the left and right wheels. That is, the original encoded data are the rotation radians of the left and right wheels of the mobile robot during the movement of the mobile robot in the first visual beacon area. Furthermore, in combination with the internal parameters of the odometer, the real-time pose of the odometer relative to the starting pose at each sampling moment is determined.

[0096] S103. Based on the obtained first odometry information, the image positions and global positions of the first visual beacons, determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer;

[0097] The homography matrix between the pixel plane of the above camera and the body coordinate system plane of the odometer is used to describe the projection transformation relationship from the coordinate points on the pixel plane of the camera to the coordinate points on the body coordinate system plane of the odometer. The pixel plane of the camera is the plane in the pixel coordinate system of the camera, and the body coordinate system plane of the odometer is the plane in the body coordinate system of the odometer. The homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer contains the internal and external parameters of the camera; among them, the internal and external parameters of the camera indicate the internal parameters of the camera and the external parameters between the camera and the odometer.

[0098] In short, for the embodiments of the present invention, the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer indicates the first conversion relationship from the pixel coordinate system of the camera to the camera coordinate system, and the second conversion relationship from the camera coordinate system to the body coordinate system of the odometer. The above first conversion relationship contains the internal parameters of the camera, the above second conversion relationship contains the external parameters between the camera and the odometer, and the homography matrix between the pixel plane and the body coordinate system plane of the odometer is essentially the product of the first conversion relationship and the second conversion relationship. Therefore, the homography matrix between the pixel plane and the body coordinate system plane of the odometer contains the internal and external parameters of the camera.

[0099] For any first feature point in the pixel plane of the camera, in order to determine the position of the first feature point in the body coordinate system plane of the odometer, it is necessary to project the first feature point from the pixel plane of the camera to the camera coordinate system of the camera based on the first conversion relationship to obtain a second feature point. Furthermore, based on the second conversion relationship, project the second feature point from the camera coordinate system of the camera to the body coordinate system of the odometer to obtain a third feature point. The position of the third feature point is the position of the above first feature point in the body coordinate system plane of the odometer. It can be seen from this that the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer can essentially be immediately the coordinate conversion relationship from the pixel plane of the camera to the body coordinate system plane of the odometer.

[0100] In order to determine the coordinate conversion relationship between the pixel plane of the camera and the body coordinate system plane of the odometer, in the embodiments of the present invention, the image positions of the first visual beacons obtained can be understood as the positions before conversion that need to be converted using the above coordinate conversion relationship.

[0101] Since the first odometry information records the pose of the odometer at each sampling moment, and this pose is essentially the conversion parameter between the body coordinate system of the odometer and the world coordinate system. That is to say, during the movement of the mobile robot, the conversion relationship between the body coordinate system of the odometer and the world coordinate system is constantly changing, and the conversion parameter of this conversion relationship is the real-time pose of the odometer in the world coordinate system. Thus, at each sampling moment, the conversion parameter in the conversion relationship between the body coordinate system of the odometer and the world coordinate system at this sampling moment can be understood as the pose of the odometer at this sampling moment.

[0102] Based on this, the global positions of the first visual beacons can be converted into the body coordinate system of the odometer using the first odometry information, and the positions of the first visual beacons in the body coordinate system of the odometer are obtained. This position can be understood as the position after conversion that needs to be converted using the above coordinate conversion relationship.

[0103] After determining the position before conversion and the position after conversion, the conversion relationship between the position before conversion and the position after conversion can be determined using the position before conversion and the position after conversion, that is, the coordinate conversion relationship between the pixel plane of the camera and the body coordinate system plane of the odometer, which is also the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0104] In the above solution provided by the embodiments of the present invention, since the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer contains the internal parameters of the camera and the external parameters between the camera and the odometer, the process of determining the homography matrix based on the obtained first odometry information, the image positions and global positions of the first visual beacons can be understood as the process of calibrating the internal parameters of the camera and the external parameters between the camera and the odometer simultaneously. It can be seen that through this solution, there is no need to separately calibrate the internal parameters of the camera and the external parameters between the camera and the odometer, thus simplifying the process of parameter calibration of the mobile robot and improving the efficiency of parameter calibration of the mobile robot.

[0105] Based on Figure 1 the method shown, as Figure 3 shown, the embodiments of the present invention also provide another calibration method. The above step S103 may include steps S301 - S304, where:

[0106] S301. For the first visual beacon, based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon, determine the reprojection error of the first visual beacon.

[0107] Among them, the target transformation function is a function of the parameters to be calibrated, which is used to indicate the coordinate transformation relationship between the pixel plane of the camera and the body coordinate system plane of the odometer. Exemplarily, the above target transformation relationship can be expressed as:

[0108]

[0109] Among them, (u, v) are the coordinates in the pixel coordinate system, s represents the scale factor, (x b , y b ) are the coordinates in the body coordinate system of the odometer, where h 11 ~h 32 are the parameters to be calibrated included in the homography matrix.

[0110] In one implementation, the above step S301 may include steps A1 - A3, where:

[0111] Step A1. Based on the target transformation function, project the image position of the first visual beacon onto the body coordinate system plane to obtain the first projection position.

[0112] Since the target transformation function is a function used to indicate the coordinate transformation relationship between the pixel plane of the camera and the body coordinate system plane of the odometer, the image position of the first visual beacon can be substituted into the target transformation function to obtain the position of the first visual beacon in the body coordinate system plane as the first projection position.

[0113] Of course, it should be noted that since the parameters included in the target transformation function are the parameters to be calibrated, the determined first projection position is actually the position estimated by using the target transformation function, and there may be a difference between this first projection position and the actual projection position of the first visual beacon in the body coordinate system.

[0114] Step A2. Based on the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, project the first projection position into the global coordinate system to obtain the second projection position.

[0115] In this step, based on the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, the pose of the odometer at the acquisition moment of the image frame to which the first visual beacon belongs can be determined. As described above, this pose is essentially the conversion parameter between the body coordinate system of the odometer and the world coordinate system. Therefore, using this pose, the conversion relationship between the body coordinate system of the odometer and the world coordinate system at the acquisition moment of the image frame to which the first visual beacon belongs can be determined. Furthermore, the first projection position can be projected into the global coordinate system using this conversion relationship to obtain the second projection position.

[0116] Since the first projection position is essentially the position estimated using the target conversion function, there is also a difference between the second projection position determined based on the first projection position and the actual position of the first visual beacon in the global coordinate system.

[0117] Step A3: Use the difference information between the second projection position and the global position of the first visual beacon as the reprojection error of the first visual beacon.

[0118] As can be seen from the above, there is also a difference between the second projection position and the actual position of the first visual beacon in the global coordinate system, and the actual position of the first visual beacon in the global coordinate system is essentially the global position of the first visual beacon. Therefore, the difference information between the second projection position and the global position of the first visual beacon can be calculated as the reprojection error of the first visual beacon.

[0119] The above difference information can be the distance between the second projection position and the global position, such as the Euclidean distance. That is to say, the reprojection error is the distance between the second projection position and the global position.

[0120] S302: Determine whether the preset iteration end condition is satisfied. If the preset iteration end condition is not satisfied, execute step S303. If the iteration end condition is satisfied, execute step S304.

[0121] In one implementation, the above preset iteration end condition includes at least one of the following conditions:

[0122] 1. The total reprojection error is less than the first error threshold;

[0123] The above total reprojection error is the sum of the reprojection errors of each first visual beacon. The magnitude of the total reprojection error reflects the magnitude of the errors of the parameters to be calibrated in the target conversion function. The larger the total reprojection error, the larger the errors of the parameters to be calibrated in the target conversion function. The smaller the total reprojection error, the smaller the errors of the parameters to be calibrated in the target conversion function.

[0124] When the total reprojection error is less than the first error threshold, it can be considered that the errors of the calibration parameters to be determined in the target conversion function are small enough, so there is no need to continue adjusting the calibration parameters to be determined, that is, the iteration can be ended. The above first error threshold can be determined based on actual requirements and experience.

[0125] 2. The total reprojection error reaches the first minimum value;

[0126] Among them, the above first minimum value is the minimum value that the total reprojection error can reach after continuous iteration.

[0127] When the total reprojection error reaches the first minimum value, there is no need to perform iteration anymore, so the iteration can be ended.

[0128] 3. The number of iterations is greater than the first number threshold.

[0129] To avoid falling into an infinite iteration loop, a first number threshold can be set. When the number of iterations is greater than the first number threshold, it is considered that the iteration can be ended. In this case, each time step A1 is executed, the number of iterations can be updated and the iteration count is increased by one.

[0130] If the preset iteration end condition is not satisfied, step S303 is executed. If the iteration end condition is satisfied, step S304 is executed.

[0131] S303: Based on the reprojection errors of the first visual beacons, adjust the parameters in the target conversion function and return to execute step S301;

[0132] This step can be executed when the preset iteration end condition is not satisfied.

[0133] If the preset iteration end condition is not satisfied, it means that the calibration parameters to be determined in the target conversion function do not meet the requirements and need to be adjusted. Therefore, the parameters in the target conversion function can be adjusted based on the reprojection errors of the first visual beacons.

[0134] S304: Use the target conversion function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0135] This step can be executed when the preset iteration end condition is satisfied.

[0136] If the preset iteration end condition is satisfied, it means that the calibration parameters to be determined in the target conversion function already meet the requirements. Therefore, the target conversion function is used as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0137] The above solution provided by the embodiments of the present invention can improve the efficiency of parameter calibration of the mobile robot. Further, through iterative optimization of the target transformation function, the parameters within the target transformation function are continuously adjusted, so that a target transformation function that meets the preset iteration end condition can be obtained, serving as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer, thereby providing an implementation basis for improving the efficiency of parameter calibration of the mobile robot.

[0138] When converting the original encoded data into odometry information as the first odometry information based on the internal parameters of the odometer, there are various situations for the internal parameters of the above odometer. In one case, the internal parameters of the above odometer can be known in advance, that is, the internal parameters of the odometer are accurately known in advance, so that the internal memory of the odometer can be directly utilized to convert the original encoded data into odometry information. In this case, the odometry information of the odometer is also accurate.

[0139] In another case, the internal parameters of the above odometer can be unknown quantities to be calibrated, that is, the internal parameters of the odometer are internal parameters to be calibrated. At this time, the first odometry information is estimated using the internal parameters to be calibrated, which means that the odometer pose determined by the first odometry information is inaccurate and there is a difference from the actual pose of the odometer.

[0140] When the internal parameters of the odometer are unknown quantities to be calibrated, in order to further improve the accuracy of the unit matrix, a joint optimization method can be adopted to determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0141] In one embodiment, constraints can be constructed based on the first odometry information and the actual pose of the odometer, and combined with the aforementioned reprojection error, the internal parameters of the odometer and the internal and external parameters of the camera are jointly optimized.

[0142] In one implementation, before adjusting the parameters within the target transformation function based on the reprojection error of each first visual beacon, the motion error can be determined first, that is, the difference between the actual pose of the odometer and the pose determined based on the internal parameters to be calibrated and the original encoded data.

[0143] Optionally, the following steps can be used to determine the motion error, including steps B1 - B2, where:

[0144] Step B1, for each image frame, based on the image position and global position of the first visual beacon in the image frame, determine the pose of the camera when collecting the image frame as the first pose of the odometer at the collection moment of the image frame, and based on the first odometry information, determine the second pose of the odometer at the collection moment of the image frame;

[0145] In one implementation, the image position of the first visual beacon may include the pixel coordinates of at least three of the four corner points of the first visual beacon in the image frame, while the world position of the first visual beacon includes the three-dimensional coordinates of at least three of the four corner points of the first visual beacon in the world coordinate system, so that the pose of the camera when collecting the image frame can be determined by using the N-point perspective pose solution method.

[0146] When the pose of the camera when collecting the image frame is determined, the pose of the camera when collecting the image frame can be used as the first pose of the odometer when the camera collects the image frame, that is, the pose of the camera is used instead of the pose of the odometer.

[0147] Furthermore, the pose at the acquisition moment of the image frame can be determined from the poses at each acquisition moment included in the first odometry information and used as the second pose of the odometer at the acquisition moment of the image frame.

[0148] Step B2, for each image frame, calculate the difference information between the first pose and the second pose of the odometer at the acquisition moment of the image frame as the motion error of the odometer at the acquisition moment of the image frame.

[0149] Since the internal parameters of the odometer are the internal parameters to be calibrated, the second pose of the odometer is actually the pose estimated based on the internal parameters to be calibrated. The first pose of the odometer can be approximately considered as the actual pose of the odometer. When the internal parameters of the odometer are accurate, the difference between the first pose and the second pose should be small. The larger the difference, the more inaccurate the internal parameters to be calibrated.

[0150] After determining the first pose and the second pose of the odometer, the difference information between the first pose and the second pose of the odometer at the acquisition moment of the image frame can be calculated as the motion error of the odometer at the acquisition moment of the image frame. This difference information can be the Euclidean distance between the poses. The motion error of the odometer reflects the accuracy of the internal parameters of the odometer, so the internal parameters of the odometer can be calibrated using the motion error.

[0151] In this case, when adjusting the parameters in the target transformation function, a joint optimization method can be adopted, including:

[0152] Based on the reprojection errors of the first visual beacons and the motion errors of the odometer at the acquisition moments of the image frames, adjust the parameters in the target transformation function and the internal parameters of the odometer.

[0153] In one implementation, a graph optimization objective function for the homography matrix can be constructed based on the reprojection error of the visual beacon and the motion error of the odometer. For example:

[0154] min{∑‖r o ‖2 + ∑‖r v ‖ 2}

[0155] where r o is the motion error, and r v is the reprojection error of the visual beacon.

[0156] The objective optimization function aims to adjust the parameters of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer at least. In the above formula, the objective of adjusting the parameters of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer is to minimize the sum of the total reprojection error and the total motion error.

[0157] Of course, there can also be other optimization objectives. In this case, the above preset iteration end conditions include at least one of the following conditions:

[0158] 1. The sum of the total reprojection error and the total motion error is less than the second error threshold;

[0159] Among them, the total reprojection error is the sum of the reprojection errors of each first visual beacon, and the total motion error is the sum of the motion errors of the odometer at the acquisition moments of each image frame. Its magnitude reflects the magnitude of the errors of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer. The larger the sum of the total reprojection error and the total motion error, the larger the errors of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer; the smaller the sum of the total reprojection error and the total motion error, the smaller the errors of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer.

[0160] When the sum of the total reprojection error and the total motion error is less than the second error threshold, it can be considered that the errors of each parameter to be calibrated in the target transformation function and the internal parameters of the odometer are small enough, so there is no need to continue adjusting each parameter to be calibrated, that is, the iteration can be ended. The above second error threshold can be determined based on actual requirements and experience.

[0161] 2. The sum of the total reprojection error and the total motion error reaches the second minimum value;

[0162] Among them, the second minimum value is the minimum value that the sum of the total reprojection error and the total motion error can reach after continuous iteration.

[0163] When the sum of the total reprojection error and the total motion error reaches the second minimum value, it means that there is no need to continue the iteration, so the iteration can be ended.

[0164] 3. The number of iterations is greater than the second number threshold.

[0165] To avoid falling into an infinite iteration loop, a second number threshold can be set. When the number of iterations performed is greater than the second number threshold, it is considered that the iteration can end. In this case, each time step A1 is executed, the number of iterations performed can be updated, increasing the iteration count by one.

[0166] During the above iteration process, it can be implemented using a graph optimization algorithm. At this time, the parameters to be calibrated of the unit matrix and the internal parameters of the odometer serve as the nodes of the graph constructed by the graph optimization, and the total reprojection error and the total motion error serve as the constraint edges between the nodes of the constructed graph.

[0167] The above solution provided by the embodiments of the present invention can improve the efficiency of parameter calibration of the mobile robot. Further, by using the reprojection error and the motion error, the parameters in the target transfer function and the internal parameters of the odometer are jointly optimized. Since there are more constraints, the accuracy of the determined homography matrix is higher.

[0168] To further improve the accuracy of the unit matrix, after the unit matrix is determined, the unit matrix can be further optimized. Based on Figure 1 the solution shown, as Figure 4 shown, the embodiments of the present invention further provide a calibration method. After the above step S103, steps S401 - S403 can further be included, where:

[0169] S401, based on the homography matrix, control the mobile robot to move in the second visual beacon area according to a preset motion mode; wherein, the accuracy of each second visual beacon in the second visual beacon area is greater than the accuracy of each first visual beacon in the first visual beacon area;

[0170] After obtaining the homography matrix, the mobile robot can be controlled to move based on the homography matrix. To improve the accuracy of the unit matrix, the mobile robot can be controlled to move in the second visual beacon area according to a preset motion mode.

[0171] Among them, the preset motion mode can be any motion mode set according to requirements, such as circular motion, random motion, etc. To improve the optimization efficiency, the above preset motion mode can be to reciprocate several times between two preset endpoints in two dimensions and perform a rotation in place at the endpoints.

[0172] As Figure 5 shown, a schematic diagram of a second visual beacon area provided by the embodiments of the present invention Figure 5Among them, the middle rectangle marks the mobile robot. The small rectangles on both sides of the middle rectangle represent the left and right wheels of the mobile robot, and the arrow direction indicates the movement mode of the mobile robot. The visual beacons in the second visual beacon area can be linearly distributed at equal intervals, and the code distance between adjacent second visual beacons is the second code distance, which is less than the first code distance. For example, the second code distance is between 0.5 m and 1.0 m, such as 0.75 m.

[0173] In order to improve the accuracy of the unit matrix, the accuracy of each second visual beacon in the second visual beacon area is greater than the accuracy of each first visual beacon in the first visual beacon area. This means that the accuracy of each second visual beacon in the second visual beacon area is higher, the difference between the global position recorded by its visual information and the actual position is smaller, and / or the code distance between adjacent second visual beacons is less than the code distance between adjacent first visual beacons.

[0174] S402: Obtain the image position and global position of the second visual beacon included in the second image frame collected by the camera during the movement of the mobile robot in the second visual beacon area, and the second odometry information collected by the odometer.

[0175] When the mobile robot is moving in the second visual beacon area, the camera in the mobile robot can collect images of the second visual beacon area, so that an image frame containing at least one visual beacon can be collected. For the sake of distinction, in the embodiments of the present invention, the image frame collected by the camera carried by the mobile robot during the movement of the mobile robot in the second visual beacon area is referred to as the second image frame.

[0176] In order to reduce the amount of calculation, the second image frame can be a key frame among the collected image frames. For any second visual beacon included in each second, the image position of the second visual beacon in the image frame can be the pixel coordinates of the corresponding pixel points of the center point of the second visual beacon in the second image frame. Alternatively, the image position of the second visual beacon in the second image frame can also be the pixel coordinates of the corresponding pixel points of at least one corner point among the four corner points of the second visual beacon in the image frame.

[0177] S403: Optimize the homography matrix based on the obtained second odometry information, the image positions and global positions of the second visual beacons to obtain an optimized homography matrix.

[0178] After obtaining the second odometry information, the image positions and global positions of the second visual beacons, the homography matrix can be optimized to obtain an optimized homography matrix.

[0179] The specific optimization method is similar to the process of determining the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first mileage information, the image positions and global positions of the first visual beacons. The difference is that in the process of determining the unit matrix, the to-be-calibrated parameters in the target conversion function are constrained by the first mileage information, the image positions and global positions of the first visual beacons, while in this step, the second mileage information, the image positions and global positions of the second visual beacons are used to constrain the calibrated parameters of the unit matrix, so as to achieve the purpose of optimizing the calibrated parameters of the unit matrix. For the specific implementation method, reference can be made to the implementation process of determining the unit matrix, which will not be elaborated in this embodiment of the present invention.

[0180] The above solution provided by the embodiment of the present invention can improve the efficiency of parameter calibration of the mobile robot. Further, by using the homography matrix, the mobile robot is controlled to move in the second visual beacon area according to a preset motion mode, and the homography matrix is further optimized based on the collected data, thereby improving the accuracy of the homography matrix.

[0181] On the basis of the above embodiment, the embodiment of the present invention further provides another calibration method combined with graph optimization, which may include:

[0182] Step 1: Coarse calibration of the homography matrix and odometer parameters.

[0183] In this step, a camera and an odometer are carried in the AGV, and the odometer can be a differential wheel encoder. The AGV is controlled to complete small-range translation and rotation movements in the first visual beacon area to obtain data frames such as visual beacons collected by the camera and odometer information. Among them, the translation and rotation movements do not need to be very precise, and an open-loop control method can be adopted. Each frame of data includes, but is not limited to, the camera timestamp, the identification of valid identifiers, the global coordinates of the visual beacon and the corner image coordinates of the visual beacon, etc., as well as the differential wheel encoder timestamp and encoder readings, etc. The layout requirement of the first visual beacon area is that the code spacing is less than the size of the lower camera's field of view, so that when the AGV moves in the first visual beacon area, the camera can observe at least one ground code.

[0184] Using the encoder readings, the code reading results (global positions of the visual beacons), and the corner local positions (image positions of the visual beacons) as inputs, directly output the initial value of the odometer internal parameter calibration without the user providing any prior knowledge about the odometer. Of course, if the odometer internal parameter design parameters can be obtained or the internal parameter calibration of the odometer has been completed, there is no need to determine the initial value of the internal parameter calibration.

[0185] After coarsely calibrating the odometer internal parameters, the homography matrix of the camera is coarsely calibrated according to the odometer and the code reading results.

[0186] Step 2: Write the coarsely calibrated camera and odometer parameters into the vehicle body, and control the AGV to move back and forth multiple times along the preset path in the area where the ground codes are accurately arranged (the position accuracy of the ground codes is within 1 mm), that is, the second visual beacon area, and collect the information of the visual beacons collected by each camera and the encoder data.

[0187] In this step, after the rough calibration of the homography matrix, write the rough calibration result into the AGV. Then, on the flat ground where several visual beacons are arranged, control the AGV to perform reciprocating movements between the two endpoint visual beacons. Rotate in place at the endpoint visual beacons on the way, and collect the left and right wheel encoder readings and the code reading results of the camera for each frame.

[0188] Step 3: After selecting the corresponding key frames according to the collected sensor information such as cameras and encoders, initialize the poses of the visual beacons, the poses of the key frames, and the homography matrices of each camera.

[0189] In this step, a high-dimensional non-linear graph optimization problem needs to be constructed, and the optimal state solution including the parameters to be calibrated is obtained through iterative solution. In addition to the parameters of the cameras and odometers to be calibrated, the global poses of the key frames and the visual beacons are also the states of the optimization problem. In order to obtain better convergence effects, we need to obtain relatively accurate initial state values.

[0190] Key frames are some relatively representative frames selected by the calibration algorithm from all frame data. If computing resources permit, it is also entirely possible to use all frames as key frames. In this case, the dimension of the optimization problem will become very high, and the memory occupation and computing time consumption of the optimization algorithm will become unacceptable. Therefore, by screening out a certain number of key frames, the dimension of the optimization problem is controlled within an acceptable range.

[0191] The pose of the key frame is the pose of the AGV at the moment when the key frame is collected. After obtaining the internal parameters of the rough calibration of the odometer, directly use the odometer pose measurement as the initial value of the key frame pose. After completing the screening and pose initialization of the key frames, the global pose of the visual beacon can be initialized by using the code reading results of the visual beacon in the key frames.

[0192] Step 4: Use the observation constraints of the key frame corner points by the visual beacon and the inter-frame constraints of the encoder odometer, etc., to construct a global graph optimization problem. By solving this optimization problem, finally output the refined calibration results of parameters such as the camera homography matrix and the odometer internal parameters.

[0193] Among them, the graph optimization problem consists of nodes and constraints. Each node represents a set of tightly coupled states, and each constraint represents the observations (i.e., errors) of several nodes. The ultimate optimization goal in graph optimization is to minimize the constraint edges (i.e., observations or errors).

[0194] In the embodiment of the present invention, the global pose of the visual beacon, the global pose of the key frame, the camera homography matrix, and the odometer internal parameters are used as nodes, and the encoder odometer motion observations and the visual beacon corner reprojections, etc. constitute constraint edges. Among them, the odometer constraint is associated with the key frame pose and the odometer internal parameters, and the visual beacon corner reprojection constraint is associated with nodes such as the key frame pose, the visual beacon pose, and the camera homography matrix. At the same time, we can choose to add an additional sampling delay node (indicating the difference between the sampling times of the encoder and the camera for the same data frame) for overall optimization to overcome the problem of asynchronous sensor data acquisition.

[0195] Using the odometer internal parameters, the camera homography matrix after rough calibration, and the node state parameters such as the initialized key frame and visual beacon, as well as the constraint edges such as the encoder odometer motion observations and the visual beacon corner reprojections, construct the objective function of graph optimization. Then, by solving this optimization problem, the fine calibration result of the AGV system parameters can be finally output.

[0196] The above solution provided by the embodiment of the present invention can at least have the following beneficial effects:

[0197] (1) It is convenient to collect calibration data and calibrate the parameters of sensors such as cameras and encoders at one time, including the odometer internal parameters and the homography matrix (including the internal and external parameters of the camera).

[0198] (2) Directly use the original calibration data to generate initial values without the need to provide additional prior information such as mechanical parameters, which improves the usability of the method.

[0199] (3) By calibrating and compensating the sampling delay of the camera and the encoder, the problem of mismatched sampling times of multiple sensors is overcome, and the calibration accuracy is effectively improved.

[0200] Corresponding to the annotation method provided by the above embodiment of the present invention, as Figure 6 shown, the embodiment of the present invention also provides a calibration device, and the device includes:

[0201] A position acquisition module 601, configured to acquire the image position and the global position of the first visual beacon included in each first image frame, where each first image frame is: an image frame acquired by a camera carried by the mobile robot during the movement of the mobile robot within the first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the corresponding image frame; the global position of each visual beacon is the position of the visual beacon in the world coordinate system;

[0202] An information acquisition module 602, configured to acquire the first odometry information acquired by an odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area;

[0203] A matrix determination module 603, configured to determine a homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first odometry information, the image positions and global positions of the first visual beacons.

[0204] Optionally, the matrix determination module includes:

[0205] An information determination sub-module, configured to, for a first visual beacon, determine a reprojection error of the first visual beacon based on a target transformation function, the first odometry information acquired by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon; wherein the target transformation function is a function of the parameters to be calibrated and used to indicate the coordinate transformation relationship between the pixel plane of the camera and the body coordinate system plane of the odometer;

[0206] A parameter adjustment sub-module, configured to, if a preset iteration end condition is not satisfied, adjust the parameters in the target transformation function based on the reprojection errors of the first visual beacons, and call the information determination sub-module;

[0207] A matrix determination sub-module, configured to, if the iteration end condition is satisfied, use the target transformation function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0208] Optionally, the information determination sub-module is specifically configured to project the image position of the first visual beacon onto the body coordinate system plane based on the target transformation function to obtain a first projection position; project the first projection position into the global coordinate system based on the first odometry information acquired by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs to obtain a second projection position; and use the difference information between the second projection position and the global position of the first visual beacon as the reprojection error of the first visual beacon.

[0209] Optionally, the preset iteration end condition includes at least one of the following conditions: the total reprojection error is less than a first error threshold; the total reprojection error is the sum of the reprojection errors of the first visual beacons; the total reprojection error reaches a first minimum value, where the first minimum value is the minimum value that the total reprojection error can reach after continuous iteration; the number of iterations is greater than a first number threshold.

[0210] Optionally, the information acquisition module includes:

[0211] A data acquisition sub-module, configured to acquire the original encoded data of the odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area; wherein, the original encoded data is the data collected by the encoder within the odometer;

[0212] An information conversion sub-module, configured to convert the original encoded data into odometry information based on the internal parameters of the odometer, as the first odometry information.

[0213] Optionally, the internal parameters of the odometer are internal parameters to be calibrated; the apparatus further includes:

[0214] A motion error determination module, configured to, before the parameter adjustment sub-module adjusts each parameter in the target conversion function based on the reprojection error of each first visual beacon, for each image frame, determine the pose of the camera when collecting this image frame based on the image position and the global position of the first visual beacon in this image frame, as the first pose of the odometer at the acquisition moment of this image frame, and determine the second pose of the odometer at the acquisition moment of this image frame based on the first odometry information; for each image frame, calculate the difference information between the first pose and the second pose of the odometer at the acquisition moment of this image frame, as the motion error of the odometer at the acquisition moment of this image frame;

[0215] The parameter adjustment sub-module is specifically configured to adjust each parameter in the target conversion function and the internal parameters of the odometer based on the reprojection error of each first visual beacon and the motion error of the odometer at the acquisition moment of each image frame.

[0216] Optionally, the preset iteration end condition includes at least one of the following conditions: the sum of the total reprojection error and the total motion error is less than a second error threshold, where the total reprojection error is the sum of the reprojection errors of each first visual beacon, and the total motion error is the sum of the motion errors of the odometer at the acquisition moment of each image frame; the sum of the total reprojection error and the total motion error reaches a second minimum value, and the second minimum value is the minimum value that the sum of the total reprojection error and the total motion error can reach after continuous iteration; the number of iterations is greater than a second number threshold.

[0217] Optionally, the device also includes: a matrix optimization module, which is used to control the mobile robot to move in a preset movement mode within the second visual beacon area based on the homography matrix after the matrix determination module executes the determination of the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the acquired first mileage information, the image position and the global position of each first visual beacon, and the mobile robot moves in a second visual beacon area according to a preset movement mode based on the homography matrix; wherein the accuracy of each second visual beacon within the second visual beacon area is greater than the accuracy of each first visual beacon within the first visual beacon area; the image position and the global position of the second visual beacon contained in the second image frame captured by the camera during the movement of the mobile robot within the second visual beacon area, and the second mileage information collected by the odometer; based on the acquired second mileage information, the image position and the global position of each second visual beacon, the homography matrix is ​​optimized to obtain an optimized homography matrix.

[0218] In the above scheme provided by this embodiment, since the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer contains the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer, the process of determining the homography matrix based on the acquired first mileage information, the image position of each first visual beacon and the global position can be understood as the process of calibrating the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer at the same time. It can be seen that through this scheme, it is no longer necessary to calibrate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the odometer separately, thereby simplifying the process of mobile robot parameter calibration and improving the efficiency of mobile robot parameter calibration.

[0219] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0220] Memory 703, used for storing computer programs;

[0221] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0222] Acquire the image position and global position of the first visual beacon contained in each first image frame, wherein each first image frame is: an image frame captured by a camera carried by the mobile robot during the movement of the mobile robot in the first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the corresponding image frame; and the global position of each visual beacon is the position of the visual beacon in the world coordinate system;

[0223] Obtain the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area;

[0224] Based on the obtained first odometry information, the image positions and global positions of the first visual beacons, determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

[0225] In the above solution provided by the embodiments of the present invention, since the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer contains the internal parameters of the camera and the external parameters between the camera and the odometer, the process of determining the homography matrix based on the obtained first odometry information, the image positions and global positions of the first visual beacons can be understood as a process of calibrating the internal parameters of the camera and the external parameters between the camera and the odometer simultaneously. It can be seen that through this solution, it is no longer necessary to separately calibrate the internal parameters of the camera and the external parameters between the camera and the odometer, thereby simplifying the process of calibrating the parameters of the mobile robot and improving the efficiency of calibrating the parameters of the mobile robot.

[0226] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0227] The communication interface is used for communication between the above electronic device and other devices.

[0228] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0229] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0230] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above calibration methods are implemented.

[0231] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the calibration methods in the above embodiments.

[0232] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).

[0233] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0234] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for device, equipment, and system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0235] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A calibration method, characterized in that, the method includes: Obtaining the image positions and global positions of the first visual beacons included in each first image frame, where each first image frame is: an image frame collected by a camera carried by the mobile robot during the movement of the mobile robot within the first visual beacon area; the image position of each visual beacon is the position of the visual beacon in the image frame to which it belongs; the global position of each visual beacon is the position of the visual beacon in the world coordinate system; Obtaining first odometry information collected by an odometer carried by the mobile robot during the movement of the mobile robot within the first visual beacon area; Based on the obtained first odometry information, the image positions and global positions of each first visual beacon, determining a homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer, including: For a first visual beacon, based on a target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon, determining the reprojection error of the first visual beacon; where the target transformation function is a function of the parameters to be calibrated and used to indicate the coordinate transformation relationship between the pixel plane of the camera and the body coordinate system plane of the odometer; If the preset iteration end condition is not satisfied, then based on the reprojection errors of the first visual beacons, adjusting the parameters in the target transformation function, and returning to execute the step of determining the reprojection error of the first visual beacon for the first visual beacon, based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon; If the iteration end condition is satisfied, then taking the target transformation function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

2. The method according to claim 1, characterized in that, the determining the reprojection error of the first visual beacon based on the target transformation function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, and the image position and global position of the first visual beacon includes: Based on the target transformation function, projecting the image position of the first visual beacon onto the body coordinate system plane to obtain a first projection position; Based on the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first visual beacon belongs, projecting the first projection position into the global coordinate system to obtain a second projection position; Taking the difference information between the second projection position and the global position of the first visual beacon as the reprojection error of the first visual beacon.

3. The method according to claim 1, characterized in that, the preset iteration end condition includes at least one of the following conditions: The total reprojection error is less than a first error threshold; the total reprojection error is the sum of the reprojection errors of the first visual beacons; The total reprojection error reaches a first minimum value, where the first minimum value is the minimum value that the total reprojection error can reach after continuous iteration. The number of iterations is greater than a first number threshold.

4. The method according to claim 1, wherein, when obtaining the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area, it includes: obtaining the original encoded data of the odometer carried by the mobile robot during the movement of the mobile robot in the first visual beacon area; wherein the original encoded data is the data collected by the encoder in the odometer; based on the internal parameters of the odometer, converting the original encoded data into odometry information as the first odometry information.

5. The method according to claim 4, wherein, the internal parameters of the odometer are internal parameters to be calibrated; before adjusting the parameters in the target conversion function based on the reprojection errors of the first visual beacons, the method further includes: for each image frame, based on the image position and the global position of the first visual beacon in the image frame, determining the pose of the camera when collecting the image frame as the first pose of the odometer at the acquisition moment of the image frame, and based on the first odometry information, determining the second pose of the odometer at the acquisition moment of the image frame; for each image frame, calculating the difference information between the first pose and the second pose of the odometer at the acquisition moment of the image frame as the motion error of the odometer at the acquisition moment of the image frame; the adjusting the parameters in the target conversion function based on the reprojection errors of the first visual beacons includes: adjusting the parameters in the target conversion function and the internal parameters of the odometer based on the reprojection errors of the first visual beacons and the motion errors of the odometer at the acquisition moments of the image frames.

6. The method according to claim 5, wherein, the preset iteration end condition includes at least one of the following conditions: the sum of the total reprojection error and the total motion error is less than a second error threshold, where the total reprojection error is the sum of the reprojection errors of the first visual beacons, and the total motion error is the sum of the motion errors of the odometer at the acquisition moments of the image frames; the sum of the total reprojection error and the total motion error reaches a second minimum value, where the second minimum value is the minimum value that the sum of the total reprojection error and the total motion error can reach after continuous iteration; the number of iterations is greater than a second number threshold.

7. The method according to claim 1, wherein, after determining the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the obtained first odometry information, the image positions and the global positions of the first visual beacons, the method further includes: Based on the homography matrix, control the mobile robot to move in the second vision beacon area according to a preset motion mode; wherein, the accuracy of each second vision beacon in the second vision beacon area is greater than the accuracy of each first vision beacon in the first vision beacon area; Obtain the image position and global position of the second vision beacon included in the second image frame collected by the camera during the movement of the mobile robot in the second vision beacon area, and the second odometry information collected by the odometer. Based on the obtained second odometry information, the image position and global position of each second vision beacon, optimize the homography matrix to obtain an optimized homography matrix.

8. A calibration device, Characterized in that, The device includes: A position acquisition module, configured to acquire the image position and global position of each first vision beacon included in each first image frame, where each first image frame is: an image frame collected by a camera carried by the mobile robot during the movement of the mobile robot in the first vision beacon area; the image position of each vision beacon is the position of the vision beacon in the image frame to which it belongs; the global position of each vision beacon is the position of the vision beacon in the world coordinate system; An information acquisition module, configured to acquire the first odometry information collected by the odometer carried by the mobile robot during the movement of the mobile robot in the first vision beacon area; A matrix determination module, configured to determine the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer based on the obtained first odometry information, the image position and global position of each first vision beacon; The matrix determination module includes: An information determination sub-module, configured to, for a first vision beacon, determine the reprojection error of the first vision beacon based on a target conversion function, the first odometry information collected by the odometer at the acquisition moment of the image frame to which the first vision beacon belongs, and the image position and global position of the first vision beacon; wherein, the target conversion function is: a function of the parameters to be calibrated, which is used to indicate the coordinate conversion relationship between the pixel plane of the camera and the body coordinate system plane of the odometer; A parameter adjustment sub-module, configured to, if the preset iteration end condition is not satisfied, adjust the parameters in the target conversion function based on the reprojection errors of the first vision beacons, and call the information determination sub-module; A matrix determination sub-module, configured to, if the iteration end condition is satisfied, use the target conversion function as the homography matrix between the pixel plane of the camera and the body coordinate system plane of the odometer.

9. An electronic device, Characterized in that, It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor, when executing the program stored on the memory, implements the method steps described in any one of claims 1-7.

10. A computer-readable storage medium, Characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-7 are implemented.

Images