External reference calibration method and apparatus, and autonomous mobile device

Abstract

The application provides an external parameter calibration method and device and an autonomous mobile device, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring at least one calibration image collected by a camera in a calibration scene and at least one frame of calibration point cloud frame collected by a laser radar in the calibration scene; determining the pose of the camera according to the calibration image and the point cloud contour of the calibration scene, and determining the pose of the laser radar according to the calibration point cloud frame and the point cloud contour; and determining the calibration external parameters of the camera and the laser radar according to the pose of the camera and the pose of the laser radar. The external parameter calibration method provided by the application can directly determine the pose of each sensor by using the point cloud contour of the calibration scene constructed in advance and combining the optical information collected by each sensor in the calibration scene, and then calibrate the external parameters of multiple sensors. In this way, the external parameters of multiple sensors without a common viewing area can be calibrated, and the operation accuracy of the autonomous mobile device is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method and device for calibrating extrinsic parameters and an autonomous mobile device. BACKGROUND

[0002] Camera and lidar are two very important sensors on autonomous mobile devices (such as autonomous vehicles, robots, etc.). By fusing the information of the two sensors, the accuracy and robustness of algorithms in key links such as map creation, positioning, and environment perception can be effectively improved. In order to realize the information fusion of the two sensors, good extrinsic parameter calibration is one of the important prerequisites.

[0003] Generally speaking, the extrinsic parameter calibration between multiple sensors can be realized by setting a calibration board in the common view area of the multiple sensors. However, when the multiple sensors do not have a common view area, the above-mentioned method cannot be used for extrinsic parameter calibration. Since the multiple sensors on real autonomous mobile devices are likely to not have a common view area, there is an urgent need for a solution that can realize extrinsic parameter calibration without relying on a common view area. SUMMARY

[0004] In view of the above, embodiments of the present application provide a method and device for calibrating extrinsic parameters, an autonomous mobile device, an electronic device, and a computer-readable storage medium, aiming to provide a solution for calibrating extrinsic parameters without relying on a common view area.

[0005] The first aspect of the present application provides a method for calibrating extrinsic parameters, comprising: obtaining at least one calibration image collected by a camera in a calibration scene and at least one frame of calibration point cloud frame collected by a lidar in the calibration scene, wherein the camera and the lidar are fixedly arranged on an autonomous mobile device; determining a pose of the camera according to a point cloud contour of the calibration scene and the calibration image, and determining a pose of the lidar according to the point cloud contour and the calibration point cloud frame; and determining a calibration extrinsic parameter of the camera and the lidar according to the pose of the camera and the pose of the lidar.

[0006] In some implementations, determining the pose of the camera according to the point cloud contour of the calibration scene and the calibration image comprises: determining a plurality of spatial points in the point cloud contour corresponding to a plurality of feature points of the calibration image, and obtaining position information of the plurality of spatial points; and calculating the pose of the camera according to the position information of the plurality of spatial points.

[0007] Further, in some implementations, determining the plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image comprises: determining, in a plurality of scene images of the calibration scene captured in different ranges in advance, a plurality of candidate scene images satisfying a preset condition in similarity between the calibration image and the plurality of candidate scene images; determining a candidate scene image containing the plurality of feature points as a standard scene image; and determining, according to a correspondence relationship between feature points of the plurality of scene images determined in advance and the spatial points in the point cloud profile, a plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the standard scene image as the plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image.

[0008] In some implementations, determining the pose of the LiDAR according to the calibration point cloud frame and the point cloud profile comprises: estimating m first candidate poses of the LiDAR according to the point cloud profile, and determining m first candidate point cloud frames corresponding to the m first candidate poses; registering the m first candidate point cloud frames with the calibration point cloud frame to determine n first candidate poses corresponding to n first candidate point cloud frames closest to the calibration point cloud frame as second candidate poses; and determining a second candidate pose with a minimum matching error in the n second candidate poses as the pose of the LiDAR.

[0009] In some implementations, the point cloud profile comprises a global point cloud profile of the calibration scene.

[0010] In some implementations, the point cloud profile comprises a point cloud profile of a first scene range in the calibration scene and a point cloud profile of a second scene range in the calibration scene, wherein when the autonomous mobile device is in a first device pose in the calibration scene, a capture range of the camera is located within the first scene range, and a capture range of the LiDAR is located within the second scene range.

[0011] Further, in some implementations, the point cloud profile of the first scene range comprises a point cloud profile of a first calibration object, and the point cloud profile of the second scene range comprises a point cloud profile of a second calibration object.

[0012] Optionally, in some implementations, the first scene range and the second scene range do not overlap with each other.

[0013] In some implementations, before determining the pose of the camera according to the calibration image and the point cloud profile of the calibration scene and determining the pose of the LiDAR according to the calibration point cloud frame and the point cloud profile, the method further comprises: collecting information of the calibration scene to construct the point cloud profile.

[0014] Further, in some implementations, the information of the calibration scene is collected to construct the point cloud profile includes: taking pictures of the calibration scene at two different positions in the calibration scene to obtain two sets of scene images, wherein each set of scene images includes a plurality of scene images; and calculating the point cloud profile according to the two sets of scene images.

[0015] In some implementations, the at least one calibration image includes a plurality of calibration image sets, and the at least one frame of calibration point cloud frame includes a plurality of calibration point cloud frame sets, wherein the plurality of calibration image sets and the plurality of calibration point cloud frame sets are respectively collected by the camera and the lidar when the autonomous mobile device is at a plurality of device poses in the calibration scene, wherein determining the pose of the camera according to the calibration image and the point cloud profile of the calibration scene includes: determining a plurality of first camera poses corresponding to the plurality of device poses of the camera according to the plurality of calibration image sets and the point cloud profile; and performing optimization calculation on the plurality of first camera poses to obtain the pose of the camera, and wherein determining the pose of the lidar according to the calibration point cloud frame and the point cloud profile of the calibration scene includes: determining a plurality of first lidar poses corresponding to the plurality of device poses of the lidar according to the plurality of calibration point cloud frame sets and the point cloud profile; and performing optimization calculation on the plurality of first lidar poses to obtain the pose of the lidar.

[0016] Further, in some implementations, each calibration image set includes a plurality of calibration images, and each calibration point cloud frame set includes a plurality of frames of calibration point cloud frames, wherein determining the plurality of first camera poses corresponding to the plurality of device poses of the camera according to the plurality of calibration image sets and the point cloud profile includes: determining a plurality of second camera poses corresponding to each device pose of the camera according to the plurality of calibration images and the point cloud profile; and performing optimization calculation on the plurality of second camera poses to obtain each first camera pose, and wherein determining the plurality of first lidar poses corresponding to the plurality of device poses of the lidar according to the plurality of calibration point cloud frame sets and the point cloud profile includes: determining a plurality of second lidar poses corresponding to each device pose of the lidar according to the plurality of frames of calibration point cloud frames and the point cloud profile; and performing optimization calculation on the plurality of second lidar poses to obtain each first lidar pose.

[0017] A second aspect of the present application provides an external parameter calibration device, comprising: an acquisition module configured to acquire at least one calibration image collected by a camera in a calibration scene and at least one frame of calibration point cloud frame collected by a lidar in the calibration scene, wherein the camera and the lidar are fixedly arranged on an autonomous mobile device; a determination module configured to determine a pose of the camera according to the calibration image and a point cloud profile of the calibration scene, and determine a pose of the lidar according to the calibration point cloud frame and the point cloud profile; and a calibration module configured to determine calibration external parameters of the camera and the lidar according to the pose of the camera and the pose of the lidar.

[0018] Optionally, the external parameter calibration device further comprises a construction module configured to collect information of the calibration scene to construct the point cloud profile.

[0019] A third aspect of the present application provides an autonomous mobile device, comprising: at least one camera configured to collect images; at least one lidar configured to collect point cloud frames; and a processor configured to perform the external parameter calibration method provided in the first aspect of the present application to calibrate the at least one camera and the at least one lidar.

[0020] A fourth aspect of the present application provides an electronic device, comprising: a memory configured to store computer instructions; and a processor configured to execute the computer instructions to implement the external parameter calibration method provided in the first aspect of the present application.

[0021] A fifth aspect of the present application provides a computer readable storage medium storing computer instructions, which, when executed by a processor, implement the external parameter calibration method provided in the first aspect of the present application.

[0022] The external parameter calibration method, device, autonomous mobile device, electronic device and computer readable storage medium provided by the present application can directly determine the pose of each sensor by using the point cloud profile of the calibration scene constructed in advance and combining the optical information collected by each sensor in the calibration scene, and further obtain the calibration external parameters between the multiple sensors. In this way, the external parameters of the multiple sensors without a common view area can be calibrated, thereby effectively improving the operation accuracy of the autonomous mobile device.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot constitute a limitation on the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the accompanying drawings constitute a part of the specification, and are used to explain the embodiments of the present application together with the embodiments of the present application, and do not constitute a limitation on the present application. Unless otherwise stated, the same symbols and numbers in the drawings generally represent the same steps or components.

[0025] Figure 1 Fig. 1 shows a schematic diagram of an exemplary external parameter calibration system provided by an embodiment of the present application.

[0026] Figure 2 Fig. 2 shows a flowchart of an external parameter calibration method provided by an embodiment of the present application.

[0027] Figure 3 Fig. 3 shows a flowchart of an exemplary external parameter calibration method provided by an embodiment of the present application.

[0028] Figure 4 Fig. 6 shows a schematic diagram of another exemplary extrinsic parameter calibration method provided by an embodiment of the present application.

[0029] Figure 5 Fig. 7 shows a schematic diagram of an extrinsic parameter calibration method provided by another embodiment of the present application.

[0030] Figure 6 Fig. 8 shows a schematic diagram of a process of an exemplary extrinsic parameter calibration method provided by an embodiment of the present application.

[0031] Figure 7a Fig. 9 shows a schematic diagram of a state of an autonomous mobile device in a calibration scene in an embodiment of the present application.

[0032] Figure 7b Fig. 10 shows a schematic diagram of a state of an autonomous mobile device in a calibration scene in an embodiment of the present application.

[0033] Figure 7c Fig. 11 shows a schematic diagram of a state of an autonomous mobile device in a calibration scene in an embodiment of the present application.

[0034] Figure 8 Fig. 12 shows a schematic diagram of an extrinsic parameter calibration apparatus provided by an embodiment of the present application.

[0035] Figure 9 Fig. 13 shows a schematic diagram of an extrinsic parameter calibration apparatus provided by another embodiment of the present application.

[0036] Figure 10 Fig. 14 shows a schematic diagram of an autonomous mobile device provided by an embodiment of the present application.

[0037] Figure 11 Fig. 15 shows a schematic diagram of an exemplary electronic device provided by an embodiment of the present application.DETAILED DESCRIPTION

[0038] In related technologies, in order to calibrate extrinsic parameters of different sensors, a calibration board is usually placed in a common view area of a camera and a lidar to be calibrated, and then an image and a point cloud frame containing the common view area are respectively acquired by the camera and the lidar. Then, the positions of specific positions (for example, four vertices of the calibration board, or a preset special mark, etc.) on the calibration board in a pixel coordinate system and a point cloud coordinate system are respectively obtained from the image and the point cloud frame. Then, a transformation relationship between the two coordinate systems is solved, and a calibration extrinsic parameter between the camera and the lidar is obtained.

[0039] However, it can be seen from the principle of this calibration manner that the precondition for its implementation is that the camera and the lidar to be calibrated have a common view area. That is, if the camera and the lidar to be calibrated do not have a common view area, the extrinsic parameter calibration between the two sensors cannot be directly implemented in this manner. This problem limits the installation positions of the sensors on the autonomous mobile device and may not be able to collect data in the best combination.

[0040] Exemplarily, in some implementations, in order to implement the extrinsic parameter calibration between two sensors that do not have a common view area, an intermediate sensor having a common view area with the two sensors respectively can be arranged between the two sensors. In this way, the calibration extrinsic parameter between any one of the two sensors and the intermediate sensor can be obtained by using the above-mentioned calibration manner, and then the calibration extrinsic parameter between the two sensors can be calculated based on the coordinate system of the intermediate sensor.

[0041] This manner solves the above-mentioned problem, but it needs to use another sensor, which increases the device cost, operation complexity and calculation amount, and may introduce more errors in the calculation process, which affects the accuracy of the finally obtained calibration extrinsic parameter.

[0042] Therefore, embodiments of the present application provide an extrinsic parameter calibration method, which can implement the extrinsic parameter calibration for multiple sensors that do not have a common view area, so that the combination design of the sensors of the autonomous mobile device is more flexible. Meanwhile, the extrinsic parameter calibration method provided by the present application does not need to increase the number of sensors, effectively avoiding the increase of cost and the expansion of calculation error.

[0043] Exemplary system

[0044] Figure 1 An exemplary extrinsic parameter calibration system 100 provided by embodiments of the present application is shown in the figure. The extrinsic parameter calibration system 100 can include a scene reconstruction system 110 and a calibration execution system 120, wherein the scene reconstruction system 110 can include a first sensor 111 and a first computing device 112, and the calibration execution system 120 can include an autonomous mobile device 121 and a second computing device 122, wherein the camera 1211 and the lidar 1212 are fixedly arranged on the autonomous mobile device 121.

[0045] The scene reconstruction system 110 can be used to determine the point cloud profile of the calibration scene, wherein the first sensor 111 can be used to collect optical information in the calibration scene and transmit the optical information to the first computing device 112, so that the first computing device 112 can reconstruct the point cloud profile of the calibration scene based on the optical information. The point cloud profile of the calibration scene can be the point cloud profile of a specific object in the calibration scene, can be the point cloud profile of a local range in the calibration scene, or can be the point cloud profile of the whole calibration scene.

[0046] For example, the first sensor 111 can be a binocular vision sensor (hereinafter referred to as a binocular camera) configured to capture two sets of images corresponding to each other in the calibration scene, so that the first computing device 112 can calculate the depth information of the structural object appearing in the images according to the two sets of images, and further reconstruct a sparse point cloud profile of the calibration scene. Optionally, according to actual requirements, the first computing device 112 can further calculate a dense point cloud profile of the calibration scene according to the sparse point cloud profile.

[0047] In some other examples, the first sensor 111 can also be a monocular vision sensor (hereinafter referred to as a monocular camera) configured to capture two sets of images corresponding to each other at two preset positions in the calibration scene, so that the first computing device 112 can reconstruct a point cloud profile of the calibration scene.

[0048] In addition, the first sensor 111 can also be a structured light sensor (for example, an RGB-D camera) configured to capture the calibration scene and obtain the depth information of the calibration scene at the same time. Alternatively, the first sensor 111 can also be a combination of multiple sensors, such as a combination of a camera and a laser radar, or a combination of a camera, a laser radar, and an inertial sensor (for example, an inertial measurement unit IMU).

[0049] It should be understood that the scene reconstruction system 110 can only realize the point cloud profile reconstruction of the calibration scene, and the specific implementation form and function division of the first sensor 111 and the first computing device 112 in the embodiments of the present application are not limited.

[0050] The calibration execution system 120 can be used to execute the process of extrinsic calibration. The autonomous mobile device 121 can be a vehicle or a robot with an automatic driving mode. In order to realize the function of autonomous movement, the autonomous mobile device 121 needs to be provided with sensors for perceiving the surrounding environment. In the embodiments of the present application, the autonomous mobile device 121 is provided with at least two sensors, i.e., a camera 1211 and a laser radar 1212. In order to fuse the different types of information collected by the camera 1211 and the laser radar 1212 at different poses, the camera 1211 and the laser radar 1212 need to be extrinsically calibrated. It should be understood that the autonomous mobile device 121 can be provided with at least one camera 1211 and at least one laser radar 1212, and the number and installation pose of each sensor are usually designed according to the actual needs of the autonomous mobile device. The extrinsic calibration method provided by the embodiments of the present application can realize extrinsic calibration for any number of cameras 1211 and any number of laser radars 1212, that is, unless otherwise specified, the "camera" and "laser radar" described in the present application are generic terms and can be used to refer to any one or more cameras and laser radars on the autonomous mobile device.

[0051] In the process of extrinsic calibration, the autonomous mobile device 121 can be placed in a calibration scene, the calibration scene can be photographed by the camera 1211 to obtain a calibration image, and the calibration scene can be scanned by the laser radar 1212 to obtain a calibration point cloud frame.

[0052] The second computing device 122 can be communicatively connected with the camera 1211, the laser radar 1212 and the first computing device 112 respectively, so as to obtain the calibration image, the calibration point cloud frame and the point cloud contour of the calibration scene. According to these data, the second computing device 122 can calculate the extrinsic calibration parameters between the camera 1211 and the laser radar 1212 by executing the extrinsic calibration method provided by the embodiments of the present application.

[0053] Here, the first computing device 112 and the second computing device 122 can be a processor or a server, and can also be a computer, a mobile phone, a tablet computer, a vehicle-mounted computer or other electronic devices with processing functions. In some implementations, the first computing device 112 and the second computing device 122 can be two independent devices; in other implementations, the first computing device 112 and the second computing device 122 can be the same device or two processing modules integrated in the same device. It should be understood that the embodiments of the present application do not limit the specific implementation forms of the first computing device 112 and the second computing device 122.

[0054] Exemplary method

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0056] Figure 2 Fig. 1 shows a flowchart of an external parameter calibration method provided by an embodiment of the present application. The method can be executed by, for example, a first computing device 121 shown in Fig. 1. Figure 1 The method can include the following contents, as shown in Fig. 2. Figure 2 The method can include the following contents, as shown in Fig. 2.

[0057] S210: Obtain at least one calibration image collected by a camera in a calibration scene and at least one calibration point cloud frame collected by a lidar in the calibration scene.

[0058] The camera and the lidar are fixedly arranged on the autonomous mobile device.

[0059] In order to calibrate the camera and the lidar on the autonomous mobile device, the autonomous mobile device can be located in a calibration scene, and the camera and the lidar fixedly arranged on the autonomous mobile device are controlled to collect information of the calibration scene.

[0060] The camera can collect an image of the calibration scene, and the image will present optical information in the calibration scene within the collection range of the camera. Here, the image can be a color image or a grayscale image. The lidar can collect depth information in the calibration scene to obtain a point cloud frame, and the point cloud frame will contain a point cloud profile of the spatial structure in the calibration scene within the collection range of the lidar.

[0061] It should be understood that an image can uniquely reflect the pose of the camera when the camera collects the image, and similarly, a point cloud frame can uniquely reflect the pose of the lidar when the lidar collects the point cloud frame. According to this principle, based on the calibration image collected by the camera in the calibration scene, the pose of the camera when each collection is performed can be calculated; similarly, based on the calibration point cloud frame collected by the lidar in the calibration scene, the pose of the lidar when each collection is performed can also be calculated.

[0062] S220: a. Determine the pose of the camera according to the calibration image and the point cloud profile of the calibration scene, and b. Determine the pose of the lidar according to the calibration point cloud frame and the point cloud profile of the calibration scene.

[0063] It should be understood that the steps a and b in the step S220 can be executed simultaneously or in any order, and the embodiments of the present application do not limit the specific execution time of the steps a and b.

[0064] Here, the point cloud profile of the calibration scene can be predetermined. For example, the information of the calibration scene can be pre-acquired and the point cloud profile of the calibration scene can be constructed according to the information by the scene reconstruction system 110 shown in the figure. Figure 1 The scene reconstruction system 110 shown in the figure pre-acquires the information of the calibration scene and constructs the point cloud profile of the calibration scene according to the information.

[0065] In an exemplary implementation as shown in the figure, Figure 3 In an exemplary implementation as shown in the figure, the step a can include:

[0066] S3211: determining a plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image, and obtaining the position information of the plurality of spatial points.

[0067] S3212: calculating the pose of the camera according to the position information of the plurality of spatial points.

[0068] The point cloud profile of the calibration scene can include a sparse point cloud profile, the sparse point cloud profile can contain a plurality of spatial points in the calibration scene, and the feature information and the position information of the plurality of spatial points. According to the feature information of each spatial point in the sparse point cloud profile, the feature points in the calibration image and the spatial points in the sparse point cloud profile can be matched to obtain a plurality of matching spatial points and a plurality of matching feature points in the calibration image corresponding to the plurality of matching spatial points. Therefore, the position information of the plurality of matching spatial points contained in the sparse point cloud profile is the position information of the plurality of matching feature points in the calibration image in space, and then according to the position information, the pose of the camera when shooting the calibration image can be calculated.

[0069] In a preferred implementation of the embodiments of the present application, when the point cloud profile of the calibration scene is constructed in advance, two groups of scene images of the calibration scene can be shot by using a visible light camera (for example, shot by using a binocular camera or shot at two acquisition points by using a monocular camera), and the depth information in the calibration scene can be calculated by using the two groups of scene images, and then the sparse point cloud profile and / or the dense point cloud profile of the calibration scene can be reconstructed. At the same time, the image information of each scene image can also be retained.

[0070] Based on this, in an exemplary implementation as shown in the figure, Figure 4 In an exemplary implementation as shown in the figure, the above step S3211 can specifically include:

[0071] S4211: determining a plurality of candidate scene images in which the similarity between the calibration image and the calibration scene satisfies a preset condition from a plurality of scene images shot in different ranges of the calibration scene in advance;

[0072] S4212: determining a candidate scene image containing the plurality of feature points in the plurality of candidate scene images as a standard scene image;

[0073] S4213: determining point cloud information of the standard scene image according to the point cloud profile of the calibration scene;

[0074] S4214: determining a plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the standard scene image as the plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image according to the correspondence between the feature points of the plurality of scene images and the spatial points in the point cloud profile.

[0075] Here, the point cloud profile of the calibration scene can be a sparse point cloud profile or a dense point cloud profile. Exemplarily, the image information retained in the process of pre-reconstructing the point cloud profile of the calibration scene can include: a word vector V of each scene image, a descriptor set D of the feature points in each scene image, and a correspondence set H between each spatial point in the point cloud profile and the feature points in each scene image.

[0076] In determining the plurality of candidate scene images with a similarity to the calibration image satisfying a preset condition, the word vector V of the calibration image can be extracted first C Then, according to the pre-retained word vector V of each scene image, a plurality of scene images with a higher similarity to the calibration image (for example, the top 5 scene images) are selected as candidate scene images.

[0077] Further, the descriptor set D of the calibration image can also be extracted C And according to the pre-retained descriptor set D of each candidate scene image, a candidate scene image with the most matching feature points to the calibration image is found from the plurality of candidate scene images as a standard scene image. According to the pre-retained correspondence H between each spatial point in the point cloud profile and the feature points in each scene image, the corresponding spatial points of each feature point in the standard scene image in the point cloud profile can be determined, thereby obtaining the point cloud information of the standard scene image.

[0078] Since the calibration image and the standard scene image have a plurality of matching feature points, the spatial points corresponding to these feature points in the standard scene image are the spatial points corresponding to the matching feature points in the calibration image. That is, according to the point cloud information of the standard scene image and the correspondence between the calibration image and the standard scene image, the plurality of spatial points corresponding to the plurality of feature points in the calibration image can be determined, and the position information of these spatial points can be obtained.

[0079] On this basis, in step S3212, when the pose of the camera is calculated according to the position information of the plurality of spatial points, the initial pose of the camera can be calculated according to the correspondence between the plurality of feature points in the calibration image and the plurality of spatial points in the calibration scene, and the position information of the plurality of spatial points determined in step S3211 (for example, the PnP algorithm can be used), and based on the initial pose, the pose of the camera is obtained through iterative calculation by an optimization algorithm (for example, BA (Bundle Adjustment) optimization).

[0080] The external parameter calibration method provided by the embodiments of the present application can find a scene image with high similarity to the calibration image in the plurality of scene images based on image information by using the plurality of scene images retained in the point cloud contour reconstruction process of the calibration scene, and then find the spatial points corresponding to the feature points in the calibration image in the point cloud contour by means of the scene image, so that the pose can be calculated according to the position information of the spatial points. Compared with the way of matching the calibration image only by the point cloud contour, the matching accuracy and calculation efficiency can be effectively improved by introducing image information for matching.

[0081] Optionally, in an implementation manner, the step b in the step S220 can include:

[0082] S4221: estimating m first candidate poses of the laser radar according to the point cloud contour, and determining m first candidate point cloud frames corresponding to the m first candidate poses;

[0083] S4222: registering the m first candidate point cloud frames with the calibration point cloud frame, and determining n first candidate poses corresponding to n first candidate point cloud frames closest to the calibration point cloud frame as second candidate poses;

[0084] S4223: determining a second candidate pose with the minimum matching error in the n second candidate poses as the pose of the laser radar.

[0085] Here, the point cloud contour can be a sparse point cloud contour of the calibration scene, or a dense point cloud contour of the calibration scene.

[0086] Specifically, when the calibration scene is reconstructed in advance, sparse point cloud information can be directly collected in the calibration scene to construct a sparse point cloud contour of the calibration scene, or dense point cloud information can be directly collected in the calibration scene to construct a dense point cloud contour of the calibration scene, or a corresponding dense point cloud contour can be further recovered based on the sparse point cloud contour after the sparse point cloud contour of the calibration scene is obtained, so that the sparse point cloud contour and the dense point cloud contour can be stored for easy selection and use in the calibration process according to requirements.

[0087] In a preferred embodiment, the dense point cloud profile of the calibration scene can be used in step b. When determining the pose of the lidar, n first candidate poses of the lidar and n first candidate point cloud frames corresponding to the n first candidate poses can be estimated according to the dense point cloud profile of the calibration scene. In order to reduce the computational burden, a larger area in the calibration scene that can cover the actual acquisition range of the lidar (for example, in an implementation, the area can be the second scene range in the calibration scene) can be determined in advance according to the approximate pose of the autonomous mobile device when the calibration point cloud frame is acquired and the approximate installation pose of the lidar on the autonomous mobile device, so that the first candidate poses of the lidar can be estimated only according to the dense point cloud information in the area.

[0088] After obtaining the m first candidate point cloud frames, the matching error between each first candidate point cloud frame and the calibration point cloud frame can be determined by respectively registering each first candidate point cloud frame with the calibration point cloud frame. According to the matching error, n first candidate point cloud frames that are closer to the calibration point cloud frame can be selected from the m first candidate point cloud frames as second candidate point cloud frames, and n first candidate poses corresponding to the n second candidate point cloud frames can be selected as second candidate poses.

[0089] Further, in order to improve the accuracy of the result, after obtaining the n second candidate poses, the n second candidate poses can be further screened by fine processing to obtain a second candidate pose with the smallest matching error as the pose of the lidar.

[0090] Compared with the method of determining the pose of the lidar by using the sparse point cloud profile, by using the dense point cloud profile of the calibration scene as the registration basis of the calibration point cloud frame, the matching accuracy can be improved, and thus a more accurate pose of the lidar can be obtained. Meanwhile, the external parameter calibration method provided in the embodiments of the present application can generate multiple candidate poses by using the dense point cloud profile of the calibration scene as the registration object, which can not only greatly improve the accuracy of the calculation of the pose of the lidar, but also effectively improve the matching efficiency.

[0091] It should be understood that in the present implementation, the point cloud profile of the calibration scene in the foregoing step a can also include a dense point cloud profile. That is, when calculating the pose of the camera according to the calibration image, the dense point cloud profile of the calibration scene can also be used for calculation, so as to further improve the calibration accuracy.

[0092] S230: determining the calibration external parameters of the camera and the lidar according to the pose of the camera and the pose of the lidar.

[0093] After obtaining the pose of each sensor, one sensor can be selected as a master sensor from the plurality of sensors, and the poses of the other sensors are converted into the coordinate system of the master sensor, so as to obtain the calibration extrinsic parameters of all sensors. In the embodiments of the present application, the plurality of sensors can include at least one camera and at least one lidar. For example, when the plurality of sensors to be calibrated include a first camera, a second camera, a first lidar, and a second lidar, the first camera can be selected as the master sensor, and the poses of the second camera, the first lidar, and the second lidar are converted into the coordinate system of the first camera, so as to obtain the calibration extrinsic parameters of each sensor, and complete the overall process of extrinsic parameter calibration.

[0094] It should be understood that the master sensor can also be pre-selected, and those skilled in the art can select it according to actual needs, and the embodiments of the present application do not limit the selection logic of the master sensor.

[0095] The extrinsic parameter calibration method provided by the embodiments of the present application can directly determine the pose of each sensor by using the point cloud profile of the calibration scene pre-constructed in combination with the optical information collected by each sensor on the autonomous mobile device in the calibration scene, and then calibrate the extrinsic parameters of the plurality of sensors. In this way, the extrinsic parameters of the plurality of sensors without a common view area can be calibrated, the multi-modal data can be fused with high precision, and the running accuracy of the autonomous mobile device can be significantly improved.

[0096] Preferably, in an implementation, in order to improve the accuracy of the extrinsic parameter calibration, after the autonomous mobile device is located at the first position and completes the steps S210 and S220 in the first attitude, the pose of the autonomous mobile device can be further changed (for the sake of distinction, the pose of the autonomous mobile device in the calibration scene is referred to as the device pose), and after the device pose of the autonomous mobile device is changed each time, the steps S210 and S220 are repeatedly executed, so as to obtain a set of poses of the camera and the lidar for each device pose. That is, by changing the position and / or attitude of the autonomous mobile device in the calibration scene multiple times, multiple poses of the camera and multiple poses of the lidar can be obtained based on the pre-constructed point cloud profile of the calibration scene. Furthermore, the calibration extrinsic parameters with higher accuracy can be obtained by respectively performing optimization calculation on the multiple poses of the camera and the multiple poses of the lidar.

[0097] Exemplarily, in this implementation, the at least one calibration image can include a plurality of calibration image sets, and the at least one calibration point cloud frame can include a plurality of calibration point cloud frame sets. The plurality of calibration image sets and the plurality of calibration point cloud frame sets are respectively collected by the camera and the lidar when the autonomous mobile device is in the plurality of device poses in the calibration scene.

[0098] Here, the aforementioned step a can specifically include: determining, according to the plurality of sets of calibration images and the point cloud contour, a plurality of first camera poses of the camera corresponding to the plurality of device poses; and performing optimization calculation on the plurality of first camera poses to obtain the pose of the camera. The aforementioned step b can specifically include: determining, according to the plurality of sets of calibration point cloud frames and the point cloud contour, a plurality of first radar poses of the laser radar corresponding to the plurality of device poses; and performing optimization calculation on the plurality of first radar poses to obtain the pose of the laser radar.

[0099] Further, in an optional implementation, when collecting at each device pose of the autonomous mobile device, a plurality of calibration images can be collected by the camera, and / or a plurality of calibration point cloud frames can be collected by the laser radar. That is, each set of calibration images can include a plurality of calibration images, and / or each set of calibration point cloud frames can include a plurality of calibration point cloud frames.

[0100] Here, the specific process of determining the first camera pose can include: determining, according to the plurality of calibration images and the point cloud contour, a plurality of second camera poses of the camera corresponding to each device pose; and performing optimization calculation on the plurality of second camera poses to obtain each first camera pose. In addition, the specific process of determining the first radar pose can include: determining, according to the plurality of calibration point cloud frames and the point cloud contour, a plurality of second radar poses of the laser radar corresponding to each device pose; and performing optimization calculation on the plurality of second radar poses to obtain each first radar pose.

[0101] Based on the plurality of calibration images at one position, a plurality of poses of the camera at the device pose can be obtained, and then a more accurate pose of the camera at the device pose can be obtained through optimization calculation; similarly, based on the plurality of calibration point cloud frames at one device pose, a plurality of poses of the laser radar at the device pose can be obtained, and then a more accurate pose of the laser radar at the device pose can be obtained through optimization calculation. In this way, based on the high-precision pose, the accuracy of the extrinsic calibration result will be significantly improved.

[0102] It should be understood that the method of optimizing the pose can be selected by those skilled in the art according to actual needs, and the embodiments of the present application do not limit this.

[0103] Preferably, in the embodiments of the present application, the calibration images and calibration point cloud frames obtained in the above step S210 can be obtained by the camera and the laser radar collecting the calibration scene when the autonomous mobile device is in a stationary state. That is, the autonomous mobile device can be made to be stationary at a first position in the calibration scene at a first attitude (i.e., the autonomous mobile device is made to be stationary at a first device pose), and then the camera and the laser radar are started to collect at least one calibration image and at least one calibration point cloud frame corresponding to the first device pose.

[0104] In other words, in the extrinsic calibration method provided by the embodiments of the present application, the position and pose of the autonomous mobile device relative to the calibration scene remain unchanged when the camera and the lidar perform information collection. Naturally, the poses of the camera and the lidar on the autonomous mobile device relative to the calibration scene also remain unchanged. In this state, the camera and the lidar can perform information collection respectively, so as to obtain at least one calibration image and at least one calibration point cloud frame which can accurately reflect the relative poses of the camera and the lidar.

[0105] Optionally, if it is necessary to further collect information at other positions, the autonomous mobile device can be allowed to displace or rotate again after the number or quality of the calibration images and the calibration point cloud frames collected at the determined device pose reaches the expectation.

[0106] It can be understood that if information collection is performed in the process of movement of the autonomous mobile device, the time points at which the camera and the lidar perform information collection can not be kept synchronous, and the autonomous mobile device can move during the time difference between the two time points, so that the position at which the camera is located when the calibration image is captured and the position at which the lidar is located when the calibration point cloud frame is captured are not accurately corresponding. Since it is difficult to ensure accurate synchronization of the time points at which the two sensors perform information collection, the calibration image and the calibration point cloud frame collected in the process of movement of the autonomous mobile device often cannot directly reflect the relative poses of the camera and the lidar, but need to be subjected to additional steps such as interpolation calculation in subsequent calculation process to realize extrinsic calibration, which results in large calculation amount and limited accuracy. In addition, the movement of the autonomous mobile device can also cause motion blur of the image and motion distortion of the point cloud frame, and can also cause the accuracy of extrinsic calibration to decrease.

[0107] Therefore, in the extrinsic calibration method provided by the embodiments of the present application, the collection of the calibration image and the calibration point cloud frame is performed on the premise that the autonomous mobile device is in a stationary state, which can effectively avoid the above problems existing in the dynamic collection method, so that the calibration image and the calibration point cloud frame are accurately aligned on the time axis and do not contain defects such as blur and distortion, thereby being capable of greatly improving the accuracy of extrinsic calibration while reducing the calculation amount.

[0108] In an optional implementation, the point cloud profile of the calibration scene can be a point cloud profile of a local range in the calibration scene. For example, the point cloud profile of the calibration scene can include a point cloud profile of a first scene range in the calibration scene and a point cloud profile of a second scene range in the calibration scene.

[0109] Specifically, in the stage of point cloud contour reconstruction of the calibration scene, the mounting pose of the camera and the lidar on the autonomous mobile device to be calibrated can be determined in advance, so as to determine the estimated acquisition range of the camera and the lidar, and determine the first scene range and the second scene range in the calibration scene according to the estimated acquisition range, so that the acquisition range of the camera and the acquisition range of the lidar fall within the first scene range and the second scene range respectively when calibration is performed. That is, when the autonomous mobile device is located at the first position in the calibration scene, the acquisition range of the camera can be located within the first scene range, and the acquisition range of the lidar can be located within the second scene range.

[0110] When the acquisition ranges of the camera and the lidar to be calibrated do not overlap with each other, that is, the two do not have a common view area, the two ranges that do not overlap with each other can be selected as the first scene range and the second scene range in the calibration scene according to the relative positional relationship between the estimated acquisition ranges of the camera and the lidar, so that the acquisition range of the camera falls within the first scene range and the acquisition range of the lidar falls within the second scene range.

[0111] In this way, the scene range for which the point cloud contour needs to be constructed can be selected in the calibration scene, so as to improve the construction efficiency of the point cloud contour, reduce the time spent on preparation work, and enable the subsequent matching process to be more efficient.

[0112] Further, in another optional implementation, the point cloud contour of the calibration scene can be the point cloud contour of a specific object in the calibration scene. For example, the first calibration object can be provided in the first scene range, and the point cloud contour of the first scene range includes the point cloud contour of the first calibration object; similarly, the second calibration object can be provided in the second scene range, and the point cloud contour of the second scene range includes the point cloud contour of the second calibration object.

[0113] As an example, the specific object in the calibration scene can be a calibration board. When the object to be calibrated is two sensors (a camera and a lidar) on the autonomous mobile device, two calibration boards of different shapes can be provided in the calibration scene in the stage of point cloud contour reconstruction of the calibration scene. The two calibration boards can be provided at positions that can be acquired by the camera and the lidar respectively. That is, one calibration board can be provided as the first calibration object in the estimated acquisition range of the camera, and another calibration board can be provided as the second calibration object in the estimated acquisition range of the lidar.

[0114] When the acquisition ranges of the camera and the lidar to be calibrated do not overlap with each other, that is, the two do not have a common view area, the first calibration object and the second calibration object can be provided at positions corresponding to the two acquisition ranges.

[0115] In the above example, assuming that the camera is arranged at the front of the autonomous mobile device and has a line of sight towards the front, and the lidar is arranged at the tail of the autonomous mobile device and has a line of sight towards the back, the two calibration boards can be arranged on two opposite walls in the calibration scene respectively. That is, one wall can be used as a first scene range, and the calibration board as the first calibration object is arranged on the wall; the other wall can be used as a second scene range, and the other calibration board as the second calibration object is arranged on the wall.

[0116] Through such an arrangement, the point cloud profile of the first scene range can include the point cloud profile of the first calibration object, and the point cloud profile of the second scene range can include the point cloud profile of the second calibration object. When the autonomous mobile device is located between the two walls and the center line is substantially perpendicular to the two walls during information collection of the calibration scene by the camera and the lidar, one calibration board is located in the collection range of the camera and the other calibration board is located in the collection range of the lidar, so that the image collected by the camera contains image information of one calibration board and the point cloud frame collected by the lidar contains point cloud information of the other calibration board.

[0117] By arranging the calibration objects with obvious structural features in the calibration scene, the effective spatial information contained in the point cloud profile can be increased, so that the accuracy of matching the calibration image and the calibration point cloud frame based on the point cloud profile is improved.

[0118] As another example, the first calibration object and the second calibration object can have a preset relative position relationship. The first calibration object and the second calibration object can have different shapes or the same shape.

[0119] Specifically, the first calibration object and the second calibration object can be arranged according to the preset relative pose, or the relative pose of the first calibration object and the second calibration object can be determined through a measurement operation after the first calibration object and the second calibration object are arranged in the estimated collection range. That is, before the pose of the camera and the lidar is determined using the point cloud profile of the first calibration object and the point cloud profile of the second calibration object, the relative pose between the two calibration objects can be determined in advance, so that the relative pose between the camera and the lidar can be determined based on the relative pose between the two calibration objects in the subsequent calculation process.

[0120] Here, the point cloud profile used in the foregoing step S220 can only include the point cloud information of the first and second calibration objects. It can be understood that as long as the camera captures a calibration image containing the first calibration object and the laser radar captures a calibration point cloud frame containing the second calibration object, the pose of the camera relative to the first calibration object and the pose of the laser radar relative to the second calibration object can be determined. On this basis, in the present example, the poses of the camera and the laser radar in different coordinate systems can be further converted into the same coordinate system by using the relative pose between the first calibration object and the second calibration object, and then the extrinsic parameter calibration for both is realized. In this way, the computational complexity can be greatly reduced, and the efficiency of extrinsic parameter calibration can be effectively improved without affecting the calibration accuracy.

[0121] After the autonomous mobile device is shipped, it may be subjected to strong jolts during transportation, or may be modified by the user or encounter accidents during use. In these cases, the poses of various sensors installed on the autonomous mobile device can change, and once the poses of the sensors change, the pre-determined extrinsic parameters between the sensors are likely to be no longer accurate, resulting in reduced running accuracy of the autonomous mobile device, and thus the sensors on the autonomous mobile device need to be re-calibrated.

[0122] In related technologies and some implementation manners of embodiments of the present application, a specially-made calibration object such as a calibration board can be used for extrinsic parameter calibration. However, for ordinary users or general businesses providing after-sales service, the cost, acquisition difficulty and operation difficulty of such a calibration object are all high. Therefore, the process of extrinsic parameter calibration using a specially-made calibration object such as a calibration board is usually implemented by professionals in experimental sites or the like. That is, when the autonomous mobile device needs to be calibrated again after being shipped, the user needs to send the autonomous mobile device to a professional calibration site to be operated by professionals, which costs a lot of time and transportation cost.

[0123] In addition, when a specially-made calibration object is used for extrinsic parameter calibration, there are high requirements for the selection and setting of the calibration scene, and the manufacturing accuracy of the calibration object can greatly affect the accuracy of the extrinsic parameter calibration.

[0124] To solve the above problems, in some preferred implementation manners, the extrinsic parameter calibration method provided by the embodiments of the present application can not use a specially-made calibration object, but can use a scene that is easy for users to find and can be used by the autonomous mobile device for daily operation as a calibration scene, and implement extrinsic parameter calibration for multiple sensors on the autonomous mobile device based on the point cloud profile information of the calibration scene.

[0125] Specifically, the point cloud profile of the calibration scene can contain global information of the calibration scene, that is, the point cloud profile of the calibration scene used in the foregoing step S220 can be a global point cloud profile of the calibration scene.

[0126] It can be understood that the global point cloud profile of the calibration scene contains spatial features of the calibration scene as a whole, and the global point cloud profile not only reflects a plurality of spatial points in the calibration scene, but also contains relative positional relationships between any two spatial points in the plurality of spatial points. Therefore, in the extrinsic parameter calibration method provided by the embodiments of the present application, when calibrating the camera and the lidar, it is only required to make the camera and the lidar in the common calibration scene, which is equivalent to making the camera and the lidar have an indirect common view relationship.

[0127] Based on this, the embodiments of the present application take the point cloud profile of the calibration scene as a matching reference, can independently determine the pose of the camera in the calibration scene through the calibration image, and can independently determine the pose of the lidar in the same calibration scene through the calibration point cloud frame, thereby achieving the determination of the poses of the camera and the lidar in the coordinate system in which the calibration scene is located, and further achieving the extrinsic parameter calibration between the camera and the lidar.

[0128] Optionally, after the global point cloud profile of the calibration scene is determined in advance, the point cloud profile of the calibration scene used in the foregoing step S220 can also be a part of the global point cloud profile, thereby reducing the amount of calculation, saving the computing resources, and improving the computing efficiency.

[0129] Specifically, similar to the foregoing embodiments, after the approximate installation poses of the autonomous mobile device to be calibrated and the plurality of sensors to be calibrated on the autonomous mobile device are determined, the estimated acquisition ranges of the plurality of sensors can be determined. According to the estimated acquisition ranges, a plurality of scene ranges in the calibration scene can be determined. For example, when the sensors to be calibrated are a camera and a lidar, a first scene range covering the estimated acquisition range of the camera and a second scene range covering the estimated acquisition range of the lidar can be selected in the calibration scene.

[0130] Further, according to the plurality of scene ranges, a plurality of local point cloud profiles in the global point cloud profile of the calibration scene can be determined, for example, the point cloud profile of the calibration scene in the foregoing step S220 can include a point cloud profile of a first scene range in the calibration scene and a point cloud profile of a second scene range in the calibration scene. It should be understood that according to the relative positional relationships of the acquisition ranges of the plurality of sensors, the plurality of local ranges can have overlapping ranges, or can not overlap with each other.

[0131] Figure 5 The flowchart of the extrinsic parameter calibration method provided by another embodiment of the present application is shown. As shown in FIG. 3, the extrinsic parameter calibration method provided by the embodiment of the present application includes the following steps. Figure 5As shown, before S220, the extrinsic calibration method provided by the embodiments of the present application can further include:

[0132] S510: Collect information of the calibration scene to construct a point cloud profile.

[0133] Step S510 can be performed by, for example, the scene reconstruction system 110 as shown. Here, step S510 can be performed before step S210 or after step S210, as long as the object collected in the two steps is the same calibration scene. Figure 1

[0134] Here, in order to improve the accuracy of extrinsic calibration, a scene with more obvious boundaries and more structured features (for example, objects with obvious shape features such as points, lines, and surfaces in the scene) can be selected as the calibration scene. At the same time, in order to avoid errors in the subsequent matching process and thus increase the computational load, preferably, the structure in the calibration scene should not include substantially repeated elements (for example, multiple windows with identical shapes and spacing, large-area repeated patterns on the wall, etc.).

[0135] Specifically, the calibration scene can be information-collected by a device capable of directly collecting spatial information such as a laser radar or a depth camera, and a point cloud profile of the calibration scene is generated. Alternatively, the calibration scene can also be photographed by a camera to obtain multiple scene images, and then the spatial information of the calibration scene is calculated based on the image information contained in the scene images, and then the point cloud profile of the calibration scene is restored.

[0136] Preferably, in an embodiment, step S510 can specifically include: photographing the calibration scene at two different positions in the calibration scene to obtain two sets of scene images, wherein each set of scene images includes multiple scene images (for example, which can be performed by the first sensor 111 in the scene reconstruction system 110); and calculating the point cloud profile according to the two sets of scene images (for example, which can be performed by the first computing device 112 in the scene reconstruction system 110).

[0137] ​Here, a binocular camera can be selected as the first sensor 111, and two sets of scene images can be obtained by taking multiple pictures of the calibration scene in the calibration scene by the binocular camera. For example, in order to construct a global point cloud profile of the calibration scene, the binocular camera can be placed in the calibration scene, and the shooting angle is adjusted after each shooting, and finally at least 360° of shooting is completed to obtain two sets of scene images containing multiple scene images. That is, all scene images in one set of scene images can cover the whole calibration scene, and the scene images in the two sets correspond to each other. Based on this, the embodiments of the present application can calculate the spatial information of the calibration scene based on the corresponding multiple pairs of scene images in the two sets of scene images, and further recover the spatial information of the whole calibration scene.

[0138] In another implementation, a monocular camera can also be selected as the first sensor 111. In order to achieve similar effects as the binocular camera, multiple shooting position groups can be selected in the calibration scene, each shooting position group including two shooting points for the monocular camera to shoot the calibration scene at each shooting point. That is, the distance between the two shooting points is equivalent to the baseline of the binocular camera. Through this implementation, the device cost can be reduced, and the universality of the external parameter calibration method provided by the embodiments of the present application can be improved.

[0139] The external parameter calibration method provided by the embodiments of the present application can use images as the reconstruction basis in the reconstruction phase of the calibration scene, so that the high-quality point cloud profile can be recovered while the image information of the calibration scene and the correlation between the image information and the point cloud profile are preserved. Therefore, the registration accuracy and efficiency of the calibration image can be improved in the subsequent camera calibration process, and the accuracy of the calibration result can be greatly improved while the efficiency of the overall process of the external parameter calibration is improved.

[0140] Figure 6 An exemplary process diagram of an external parameter calibration method provided by the embodiments of the present application is shown. The method can be executed by, for example, Figure 1 The external parameter calibration system 100 shown. As Figure 6 The method can include the following contents.

[0141] S601: Use the first sensor to shoot the calibration scene to obtain scene images.

[0142] In the external parameter calibration method provided by the embodiments of the present application, the image information and the point cloud information of the calibration scene can be preserved when the calibration scene is reconstructed, so that the camera is registered by image and the laser radar is registered by point cloud in the process of performing calibration, so as to obtain the respective poses of the camera and the laser radar.

[0143] Specifically, in some embodiments, the calibration scene can be reconstructed in the following ways: 1) sparse point cloud contour recovery of the calibration scene is completed by monocular SLAM (simultaneous localization and mapping), and then dense point cloud contour is further recovered by depth map and SLAM result. 2) the calibration scene is generated by using a combination of a camera and a laser radar, or a combination of a camera, a laser radar and an IMU (inertial measurement unit), and spatial points, descriptors and the correspondence between the spatial points and the descriptors of the visual features are reserved. 3) sparse point cloud contour is constructed by using an RGBD camera, and dense reconstruction is completed in real time by using depth information.

[0144] Preferably, in order to improve the reconstruction accuracy, a monocular camera or a binocular camera is used as the first sensor to capture the calibration scene in the embodiment, and then the sparse point cloud contour of the calibration scene is reconstructed based on the motion recovery structure by using the captured scene image, and the corresponding dense point cloud contour is generated by dense reconstruction.

[0145] In order to obtain a high-quality point cloud contour, in the embodiment of the present application, the first sensor with an undistorted lens or a low-distortion lens can be used to capture the image of the calibration scene. It can be understood that the higher the resolution of the scene image, the better the recovery effect of the point cloud contour.

[0146] After each capture is completed, the position of the first sensor can be changed and the next capture can be performed. When the position is changed, in order to better cover the whole calibration scene, a small rotation angle (for example, 1°), a small translation distance and a depth of field (for example, 0.2 meters) are preferably used as the amplitude of each change. After the whole calibration scene is captured to obtain a plurality of scene images, the scene images can be sorted in the order of the time stamp of each scene image to obtain a scene image set I. The scene image set I includes two scene image sets I L and I R .

[0147] The prior set S of scales can include a metric unit distance (for example, when the first sensor is a binocular camera, the distance is the baseline of the binocular camera, or when the first sensor is a monocular camera, the distance is the distance between two different positions of the monocular camera when capturing a pair of scene images) between the capture points of any one pair of scene images.

[0148] {I L , I R , d RL}∈S (1)

[0149] wherein d RL is IR With I L The metric unit can be meter.

[0150] S602: Recovering a sparse point cloud profile of the calibration scene from the scene images.

[0151] Specifically, a Structure From Motion (SFM) method can be used to recover a sparse point cloud profile of the calibration scene from the collected scene images, and meanwhile, the scene images, the correlation between the scene images, and the attributes and correlation between the scene images and the sparse point cloud profile in the recovery process can be preserved. Here, the SFM method can adopt a global reconstruction method or an incremental reconstruction method. Preferably, in order to improve the recovery accuracy, the embodiment of the present application can adopt an incremental reconstruction method (incremental SFM), and a timestamp can be preserved while shooting, so as to greatly shorten the matching time in the subsequent calibration process.

[0152] It should be understood that if the intrinsic parameters of the first sensor are known, the structure recovery can be performed by using the Euclidean method; if the distortion coefficient of the collected image is small, or the intrinsic parameters of the first sensor are unknown, the structure recovery can be performed by using the affine method. Exemplarily, the intrinsic parameters of the first sensor can be calibrated in advance in the embodiment of the present application, so that the metric structure of the calibration scene can be recovered by using the Euclidean method according to the intrinsic parameters of the first sensor. The important steps of the SFM method mainly include: feature extraction; matching and error elimination; geometric verification; initialization; image registration and resection; bundle adjustment and outlier filtering; forward intersection. It should be understood that within the logical framework provided by the embodiment of the present application, the person skilled in the art can design the implementation mode of each step in the SFM method according to the actual needs, and the embodiment of the present application does not limit this.

[0153] Further, by using the known scale prior set S, the metric structure recovered from the image can be further recovered to the real scale by constructing a similarity transformation, or by directly performing a linear transformation after estimating the optimal scale. For example, the process of recovering the real scale can be as shown in formulas (2) and (3).

[0154] M s ←s·M s (2)

[0155] t c ←s·t c , t c ∈ T c (3)

[0156] Simultaneously, relevant information about the motion reconstruction structure can be preserved, such as the word vector set V of the scene image, the descriptor subset D of the feature points, the co-view relationship set C between images, and the sparse point cloud contour M of the calibrated scene. s The set H of correspondences between each spatial point in the point cloud contour and feature points in each image, and the recovered camera motion set T. c The process of reconstructing the calibrated scene based on the scene image set I using the SFM method can be described as shown in formula (4).

[0157] V, D, C, H, M s T c ←sfm(I) (4)

[0158] S603: Restore the dense point cloud outline of the calibration scene.

[0159] Optionally, to improve the efficiency and accuracy of point cloud registration for LiDAR, thereby enhancing its calibration precision, embodiments of this application can further recover the dense point cloud contour of the calibration scene based on the recovered sparse point cloud contour. Specifically, the dense point cloud contour can be obtained through point cloud enhancement or dense reconstruction methods.

[0160] In a preferred implementation, embodiments of this application may employ a dense reconstruction method to obtain the dense point cloud contour of the calibrated scene. The process may include the following steps: selecting an image sequence; selecting key points; optimizing the depth and normal vector of the key points; restoring the depth and normal vector of the remaining pixels other than the key points; and restoring the depth of all valid pixels.

[0161] Specifically, when selecting an image sequence, a frame of scene image (including two scene images with corresponding timestamps) can be selected as a reference image from the scene image set I according to a preset metric. The preset metric can be the field of view of the two scene images (for example, the field of view can be set to 5°). Those skilled in the art will understand that a field of view that is too small will result in a short baseline, leading to significant errors during reconstruction, while a field of view that is too large will result in excessive differences between the two scene images, also making it difficult to achieve good restoration results.

[0162] Furthermore, from the sparse point cloud contour M of the calibrated scene s Spatial points p that can be simultaneously projected onto the reference image and at least one other scene image are selected as key points. Based on spatial point p, the initial values ​​of the depth and normal vector of the pixels corresponding to spatial point p (including feature points and pixels around feature points) in each scene image can be determined. Subsequently, optimization calculations can be performed based on the initial values ​​to obtain the depth and normal vector of these pixels.

[0163] Optionally, since only one pixel point can not accurately represent the similarity between images, a pixel template can be introduced in the embodiments of the present application to represent the features of the central pixel point area of the image. The feature difference between the pixel template of the reference image and the pixel template of other scene images is the key to optimizing the depth and normal vector. That is, the similarity between the pixel points on different images can be determined by measuring the similarity degree of the pixel templates of different images. Therefore, in order to determine the similarity between the pixel points corresponding to the key points on the reference image and the pixel points corresponding to the key points on other scene images, a pixel template can be established in advance, and photometric consistency can be used as an index to measure the similarity degree between the pixel templates.

[0164] For example, in the embodiments of the present application, a 7x7 square pixel template can be established with the pixel point corresponding to the key point as the center, so as to reduce the amount of calculation as much as possible while improving the accuracy of the results. After projecting all the pixel points in the pixel template of a scene image to the global coordinate system, the spatial points corresponding to the pixel points in the pixel template are projected onto other scene images to determine the corresponding pixel points in other scene images, and then the depth and normal vector of the pixel points are optimized. Here, the difference (which can include geometric model, photometric model, color scale, etc. parameters) between the pixel template of the reference image and the pixel template on other scene images can be used as the optimization target.

[0165] According to the order of the confidence degree of the optimization results from high to low, a queue is established for the multiple seed points (optimal candidates) in the sparse point cloud contour M s , and the surrounding pixel points of the seed point with the highest confidence degree and not processed in the queue are sequentially added to the queue. At the same time, the depth and normal vector of the seed point are used as the initial values of the depth and normal vector of the four pixel points adjacent to the seed point for optimization calculation, and then the optimized depth and normal vector of each pixel point are obtained.

[0166] After determining the depth information of the effective pixel points in each scene image, the effective depth map of each scene image can be obtained. According to the pixel depth in the effective depth map, the pixel points can be projected to the three-dimensional space by using the inverse projection operator π -1 (·), so as to obtain the dense point cloud contour M d of the calibration scene. At this point, the reconstruction process of the dense point cloud contour can be described as formula (5).

[0167] M d ←dense(V,D,C,H,M s ,T c ) (5)

[0168] After reconstructing the calibration scene and acquiring relevant information, an autonomous mobile device can be placed in the calibration scene to collect data through multiple sensors. The poses of these sensors are then registered based on the collected data. These multiple sensors may include at least one camera and at least one LiDAR. For example, in this embodiment, the multiple sensors may include multiple cameras and multiple LiDARs.

[0169] S604: Collects information about the calibration scene using multiple cameras and multiple lidars on an autonomous mobile device, obtaining at least one calibration image corresponding to each camera and at least one calibration point cloud frame corresponding to each lidar.

[0170] In the embodiments of this application, information acquisition can be performed using a combination of static and dynamic methods. That is, the autonomous mobile device to be calibrated can change multiple poses within the calibration scene (for ease of explanation, the pose of the autonomous mobile device will be referred to as device pose below), and a set of data can be collected through multiple sensors when the autonomous mobile device is stationary in each device pose. By acquiring information while the autonomous mobile device is stationary, motion blur in camera images and motion distortion in LiDAR scanning can be effectively avoided. Furthermore, interpolation and alignment errors caused by motion-based acquisition methods can be effectively eliminated, and the consistency issues of trigger time and frequency among various sensors can be avoided. In addition, by setting multiple device poses in the calibration scene to change multiple acquisition positions, the entire calibration scene can be effectively utilized to obtain more reliable data, thereby improving the accuracy of the calibration results.

[0171] As a specific example, embodiments of this application may employ the following methods: Figures 7a-7c The structured scene shown is used as the calibration scene, in which, Figures 7a-7c Three device poses of the autonomous mobile device to be calibrated are shown in the calibration scenario. For example, Figure 7a The image shows the pose of the first device. Figure 7b The image shows the pose of the second device. Figure 7c The diagram shows the pose of the third device. It should be understood that the device poses and their number can be set according to actual needs, and the embodiments of this application do not limit this.

[0172] After docking the autonomous mobile device to be calibrated in the first device pose, the multiple sensors to be calibrated are activated to collect information from the calibration scene. For example, the multiple sensors to be calibrated on the autonomous mobile device may include a first camera C0, a second camera C1, a first lidar L1, and a second lidar L2. The acquisition time of each sensor can be set according to actual needs, such as 3 seconds. Each sensor must acquire at least one image / one point cloud frame (hereinafter, the images and point cloud frames acquired by the sensors to be calibrated during the calibration process are referred to as calibration images and calibration point cloud frames). Similarly, after completing the acquisition in the first device pose, the autonomous mobile device to be calibrated can be docked in the second device pose and then the third device pose, and the aforementioned sensors can be activated to perform acquisition operations respectively.

[0173] After completing the acquisition operations on all device poses, we can obtain data sets (i.e., calibration images or calibration point cloud frames) corresponding to the poses of the first camera C0, the second camera C1, the first lidar L0, and the second lidar L1, respectively. Where k = [1, 3] represents different device poses.

[0174] S605: Based on at least one calibration image corresponding to each camera, perform pose registration for each camera to obtain the optimal pose for each of the multiple cameras.

[0175] Camera pose registration can utilize the sparse point cloud contours of the calibrated scene and relevant information retained during the point cloud contour reconstruction process (the set of word vectors V of the scene image generated in the reconstruction step, the descriptive subset D of the spatial points, and the sparse point cloud contours M of the calibrated scene). s The set H of correspondences between each sparse point in the point cloud contour and feature points in each scene image, and the reconstructed set of camera motion T. c (etc.). Here, the camera pose registration can be performed using incremental reconstruction, visual relocalization, or deep learning-based image registration, etc. Preferably, embodiments of this application can use visual relocalization to register the camera pose.

[0176] Specifically, the following steps can be performed for each calibration image captured by any camera:

[0177] 1) Extract calibration image The feature points in the data are used to obtain the feature point set. Simultaneously determine the descriptor of feature points And generate bag-of-words vectors Where m = [0,1] represents different cameras;

[0178] 2) Utilize Match with the word vector set V of the scene image, select the most similar multiple scene images from the multiple scene images as candidate scene images, for example, 5 candidate scene images can be selected;

[0179] 3) Select the descriptor corresponding to the candidate scene image in the descriptor set D of the scene image, and match the calibration image with the candidate scene image according to the descriptor of the calibration image

[0180] 4) Select the candidate scene image with the largest number of matches from at least one candidate scene image with successful matching as the standard scene image, and find the spatial point set p s corresponding to the standard scene image in the dense point cloud contour M i of the calibration scene according to the correspondence set H of each sparse point in the point cloud contour and the feature points in each scene image;

[0181] 5) According to the spatial point set p i , the initial pose of the camera is calculated by the PnP (Perspective-n-Point) algorithm;

[0182] 6) BA (Bundle Adjustment) optimization is performed on the initial pose of the camera, and the abnormal points are iteratively removed. After the optimization is completed (for example, the optimization can be determined to be completed when the number of inliers is greater than 50), the current pose can be determined as the optimal pose of the camera.

[0183] After performing steps 1-6 on all calibration images captured by all cameras, the optimal pose of any camera when capturing any calibration image can be obtained, and the corresponding optimal pose set

[0184]

[0185] Formula (6) is a detailed representation of the camera pose set , where each item is a set composed of the optimal pose of the corresponding camera when capturing the corresponding device pose.

[0186] S606: According to at least one frame of point cloud frames corresponding to each laser radar, the pose registration of each laser radar is performed to obtain the optimal pose of each laser radar.

[0187] The pose registration of the laser radar can adopt the dense point cloud contour M​​d Specifically, in the embodiments of the present application, the optimal pose of any laser radar L m Each frame of the collected calibration point cloud frame The following steps are performed. Wherein, m = [0, 1] represents different laser radars.

[0188] 1) According to the approximate pose of the corresponding laser radar when collecting the calibration point cloud frame, a region capable of covering the collection range of the laser radar is determined in the dense point cloud profile of the calibration scene, and the Monte Carlo localization method is used (for example, the particle displacement interval can be set to 0.2 m, and the attitude interval is 5°) to estimate x first candidate poses of the laser radar corresponding to the region, and then x first candidate point cloud frames corresponding to the x first candidate poses are determined;

[0189] 2) The normal distribution transformation (NDT) registration is performed between each first candidate point cloud frame and the calibration point cloud frame (for example, the voxel can be set to 5 cm, the grid resolution is 2 cm, and the minimum transformation difference is 0.1), and y (for example, y = 5) first candidate point cloud frames with the smallest matching error are determined from the multiple first candidate point cloud frames as second candidate point cloud frames, so that the y first candidate poses corresponding to the y second candidate point cloud frames are determined as second candidate poses;

[0190] 3) For the 5 second candidate poses, the iterative closest point (ICP) algorithm is used for fine processing (for example, the error measurement can be performed by combining point to line and point to plane measurement methods), so that the second candidate pose with the smallest matching error is obtained as the optimal pose of the laser radar.

[0191] After performing the above steps 1) to 3) on all calibration point cloud frames collected by all laser radars, the optimal pose of any laser radar when collecting any calibration point cloud frame is obtained, and then the corresponding optimal pose set Wherein, k = [1, 3] represents different device poses.

[0192]

[0193] Formula (7) is a detailed description of the laser radar pose set , wherein each term is a set composed of the optimal pose of the corresponding laser radar when scanning at the corresponding device pose.

[0194] S607: Calculate the calibration extrinsic parameters of each camera and each lidar according to the optimal pose of each camera and each lidar, taking the coordinate system of the first camera in the plurality of cameras as the reference.

[0195] Optionally, before calculating the calibration extrinsic parameters of each sensor, the optimal pose of each camera and each lidar can also be further optimized to obtain the final pose of each camera and each lidar. For example, the final pose of each sensor corresponding to the device pose can be calculated by optimization as shown in formula (8) And Wherein, the final pose is the pose in the global coordinate system.

[0196]

[0197] Exemplarily, in the embodiment of the application, the first camera C0 is taken as the master sensor, and therefore the camera coordinate system of the first camera C0 can be taken as the reference. As shown in formula (9), the extrinsic parameters of the second camera C1, the first lidar L0 and the second lidar L1 under each device pose are calculated

[0198]

[0199] Optionally, as shown in formula (10), each group of extrinsic parameters can be calculated again to obtain the calibration extrinsic parameters The calibration is completed.

[0200]

[0201] The extrinsic parameter calibration method provided by the embodiment of the application can improve the accuracy of the point cloud profile and retain more abundant information of the calibration scene for use in the calibration process by collecting binocular images of the calibration scene and constructing a sparse point cloud profile of the calibration scene according to the information extracted from the images and then recovering a dense point cloud profile. In the calibration process, the pose of the camera can be determined based on the pre-retained image information, which effectively improves the matching accuracy of the pose of the camera and further improves the accuracy of the final calibration result. Meanwhile, for the pose of the lidar, a plurality of candidate lidar poses are assumed according to the dense point cloud profile, which can effectively reduce the calculation burden, improve the matching efficiency, obtain a more accurate pose of the lidar, and further make the final calibration result more accurate.

[0202] In summary, the embodiment of the present application provides an innovative external parameter calibration method, which can calibrate the external parameters of multiple sensors without common viewing areas, effectively improves the design flexibility and operation accuracy of the autonomous mobile device; at the same time, since the specially-made calibration object can be avoided, the external parameter calibration method provided by the embodiment of the present application has high universality, which can bring great convenience to users who use the autonomous mobile device in daily life.

[0203] Exemplary apparatus

[0204] Figure 8 Fig. 8 shows a schematic diagram of an external parameter calibration device 800 provided by an embodiment of the present application. As shown in Fig. 8, the external parameter calibration device 800 provided by the embodiment of the present application can include an acquisition module 810, a determination module 820 and a calibration module 830. Figure 8

[0205] The acquisition module 810 is configured to acquire at least one calibration image collected by a camera in a calibration scene and at least one frame of calibration point cloud frame collected by a laser radar in the calibration scene, wherein the camera and the laser radar are fixedly arranged on an autonomous mobile device; the determination module 820 is configured to determine the pose of the camera according to the calibration image and the point cloud contour of the calibration scene, and determine the pose of the laser radar according to the calibration point cloud frame and the point cloud contour; and the calibration module 830 is configured to determine the calibration external parameters of the camera and the laser radar according to the pose of the camera and the pose of the laser radar.

[0206] The determination module 820 can include a camera determination unit and a radar determination unit, wherein the camera determination unit can be configured to determine the pose of the camera according to the calibration image and the point cloud contour of the calibration scene, and the radar determination unit can be configured to determine the pose of the laser radar according to the calibration point cloud frame and the point cloud contour.

[0207] In a preferred implementation manner, the camera determination unit can be configured to determine a plurality of spatial points in the point cloud contour corresponding to a plurality of feature points of the calibration image, and acquire position information of the plurality of spatial points; and calculate the pose of the camera according to the position information of the plurality of spatial points.

[0208] ​Specifically, in determining the plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image, the camera determining unit can perform the following steps: determining, among a plurality of scene images obtained by photographing different ranges in the calibration scene in advance, a plurality of candidate scene images satisfying a preset condition in similarity with the calibration image; determining a candidate scene image containing the plurality of feature points in the plurality of candidate scene images as a standard scene image; and determining, according to a predetermined correspondence between feature points of the plurality of scene images and spatial points in the point cloud profile, a plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the standard scene image as the plurality of spatial points in the point cloud profile corresponding to the plurality of feature points of the calibration image.

[0209] Correspondingly, in a preferred implementation, in determining the pose of the lidar, the radar determining unit can perform the following steps: estimating m first candidate poses of the lidar according to the point cloud profile, and determining m first candidate point cloud frames corresponding to the m first candidate poses; registering the m first candidate point cloud frames with the calibration point cloud frame, and determining n first candidate poses corresponding to n first candidate point cloud frames closest to the calibration point cloud frame as second candidate poses; and determining a second candidate pose with the minimum matching error among the n second candidate poses as the pose of the lidar.

[0210] The external parameter calibration device provided by the embodiments of the present application can directly determine the pose of each sensor by using the point cloud profile of the calibration scene constructed in advance and combining the optical information collected by each sensor on the autonomous mobile device in the calibration scene, and further calibrate the external parameters of the plurality of sensors. In this way, the external parameters of the plurality of sensors without a common viewing area can be calibrated, and high-precision fusion of multi-modal data can be achieved, thereby significantly improving the operation precision of the autonomous mobile device.

[0211] Optionally, in an implementation, the at least one calibration image can include a plurality of calibration image sets, and the at least one calibration point cloud frame can include a plurality of calibration point cloud frame sets, wherein the plurality of calibration image sets and the plurality of calibration point cloud frame sets are respectively collected by the camera and the lidar when the autonomous mobile device is in a plurality of device poses in the calibration scene.

[0212] Herein, when determining the pose of the camera according to the point cloud profile of the calibration scene and the calibration image, the camera determination unit can perform the following steps: determining, according to the point cloud profile and a plurality of calibration image sets, a plurality of first camera poses of the camera corresponding to a plurality of device poses; and performing optimization calculation on the plurality of first camera poses to obtain the pose of the camera. Also, when determining the pose of the laser radar according to the point cloud profile and the calibration point cloud frame, the radar determination unit can perform the following steps: determining, according to the point cloud profile and a plurality of calibration point cloud frame sets, a plurality of first radar poses of the laser radar corresponding to a plurality of device poses; and performing optimization calculation on the plurality of first radar poses to obtain the pose of the laser radar.

[0213] Further, each calibration image set can include a plurality of calibration images, and each calibration point cloud frame set can include a plurality of calibration point cloud frames.

[0214] Herein, when determining the pose of the camera according to the point cloud profile and the calibration image, the camera determination unit can perform the following steps: determining, according to the point cloud profile and a plurality of calibration image sets, a plurality of first camera poses of the camera corresponding to a plurality of device poses; and performing optimization calculation on the plurality of first camera poses to obtain the pose of the camera. Also, when determining the pose of the laser radar according to the point cloud profile and the calibration point cloud frame, the radar determination unit can perform the following steps: determining, according to the point cloud profile and a plurality of calibration point cloud frame sets, a plurality of first radar poses of the laser radar corresponding to a plurality of device poses; and performing optimization calculation on the plurality of first radar poses to obtain the pose of the laser radar.

[0215] The external parameter calibration device provided by the embodiments of the present application can obtain a plurality of poses of the camera in the device pose based on a plurality of calibration images in one position, and then can obtain a more accurate pose of the camera in the device pose through optimization calculation. Similarly, a plurality of poses of the laser radar in the device pose can be obtained based on a plurality of calibration point cloud frames in one device pose, and then a more accurate pose of the laser radar in the device pose can be obtained through optimization calculation. In this way, based on the high-precision pose, the accuracy of the external parameter calibration result is significantly improved.

[0216] It should be understood that the principles, functions, features of the data used, processing methods for the data, and technical effects of all optional implementation manners of the modules in the external parameter calibration device 800 provided by the embodiments can refer to the corresponding contents in the exemplary method, and will not be repeated here.

[0217] Figure 9 Fig. 9 shows a schematic diagram of an external parameter calibration device 900 provided by another embodiment of the present application. As shown in Fig. 9, the external parameter calibration device 900 includes a camera determination unit 910 and a radar determination unit 920. Figure 9As shown, the external parameter calibration device 900 provided in this application embodiment may further include a construction module 940.

[0218] Specifically, the construction module 940 can be used to collect information about the calibration scene to construct a point cloud profile. It should be understood that the construction module 940 can obtain the point cloud profile of the calibration scene in advance before the determination module 820 determines the pose of the camera and the LiDAR, so that the determination module 820 can use it when performing the determination step.

[0219] In a preferred implementation, the construction module 940 can be specifically used to: capture images of the calibration scene at two different locations in the calibration scene to obtain two scene image sets, wherein each scene image set includes multiple scene images; and calculate the point cloud contour based on the two scene image sets.

[0220] The extrinsic parameter calibration device provided in this application uses images as the basis for reconstruction during the reconstruction stage of the calibration scene. This allows for the restoration of high-quality point cloud contours while preserving the image information of the calibration scene and the correlation between the image information and the point cloud contours. This improves the registration accuracy and efficiency of the calibration images during subsequent camera calibration, thereby significantly increasing the accuracy of the calibration results while improving the overall efficiency of the extrinsic parameter calibration process.

[0221] It should be understood that the principles, functions, characteristics of the data used, processing methods for the data, and technical effects of all optional implementation methods of each module in the external parameter calibration device 900 provided in this embodiment can be referred to the corresponding content in the exemplary method, and will not be repeated here.

[0222] Exemplary device

[0223] Figure 10 The diagram shown is of an autonomous mobile device 1000 provided in an embodiment of this application. The autonomous mobile device 1000 can be a vehicle with autonomous driving capabilities, or an electronic device such as a robot with autonomous mobility capabilities.

[0224] like Figure 10 As shown, the autonomous mobile device may include at least one camera 1010, at least one lidar 1020, and a processor 1030. The at least one camera 1010 can be used to capture images, for example, by acquiring calibration images in a calibration scene; the at least one lidar 1020 can be used to acquire point cloud frames, for example, by scanning the calibration scene to obtain calibration point cloud frames. Furthermore, the processor 1030 can be used to execute the extrinsic parameter calibration method provided in any of the above embodiments of this application to perform extrinsic parameter calibration on at least one camera 1010 and at least one lidar 1020.

[0225] Here, the at least one camera 1010 can include multiple cameras, and the multiple cameras can have different collection ranges; similarly, the at least one laser radar 1020 can include multiple laser radars, and the multiple laser radars can also have different collection ranges. That is, based on the external parameter calibration method provided in the embodiments of the present application, multiple different types of sensors on the autonomous mobile device can be simultaneously calibrated in a multi-modal joint manner.

[0226] Figure 11 An exemplary electronic device 1100 provided by the embodiments of the present application is shown in the schematic diagram. As shown, the electronic device can include a processor 1110 and a memory 1120. The memory 1120 stores computer instructions, and the processor 1110 is configured to execute the computer instructions to implement the external parameter calibration method provided in any of the above embodiments. Figure 11

[0227] Exemplary computer-readable storage medium

[0228] Other embodiments of the present application further provide a computer readable storage medium, which includes computer instructions stored thereon, and the computer instructions, when executed by a processor, cause the processor to perform the external parameter calibration method provided in any of the above embodiments.

[0229] Optionally, the computer storage medium can be any tangible medium, such as a floppy disk, a CD-ROM, a DVD, a hard disk drive, or a network medium, etc.

[0230] Exemplary computer program product

[0231] Other embodiments of the present application further provide a computer program product, which includes instructions, and the instructions, when executed by a processor of a computer device, can cause the computer device to perform the external parameter calibration method provided in any of the above embodiments.

[0232] The block diagrams of the apparatuses, devices, and systems involved in the present application are only exemplary examples and are not intended to require or imply that the connection, arrangement, and configuration must be as shown in the block diagrams. Those skilled in the art can understand that the apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprise", "include", "have", etc. are open-ended words, and "include but are not limited to", and can be used interchangeably, unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0233] ​It should also be noted that in the apparatus, device and method of the present application, each module or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalents of the present application.

[0234] The above description of disclosed aspects is intended to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0235] The above description is intended to enable any person skilled in the art to make or use the application. The description is not intended to limit the embodiments of the application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, other variations, modifications, changes, additions or sub-combinations are possible based on the above disclosure and can be made by a person skilled in the art.

[0236] The above descriptions are only preferred embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement and the like made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for calibrating an extrinsic parameter, characterized in that, The method comprises: acquiring at least one calibration image captured by a camera in a calibration scene and at least one calibration point cloud frame captured by a laser radar in the calibration scene, wherein the camera and the laser radar are fixedly arranged on an autonomous mobile device; determining a pose of the camera according to the calibration image and a point cloud outline of the calibration scene, and determining a pose of the laser radar according to the calibration point cloud frame and the point cloud outline; determining calibration extrinsic parameters of the camera and the laser radar according to the pose of the camera and the pose of the laser radar, wherein the determination of the pose of the camera according to the calibration image and the point cloud outline of the calibration scene comprises: determining a plurality of spatial points in the point cloud outline corresponding to a plurality of feature points of the calibration image, and acquiring position information of the plurality of spatial points; calculating the pose of the camera according to the position information of the plurality of spatial points, and the determination of the pose of the laser radar according to the calibration point cloud frame and the point cloud outline comprises: estimating m first candidate poses of the laser radar according to the point cloud outline, and determining m first candidate point cloud frames corresponding to the m first candidate poses; registering the m first candidate point cloud frames with the calibration point cloud frame to determine n first candidate poses corresponding to n first candidate point cloud frames closest to the calibration point cloud frame as second candidate poses; determining a second candidate pose with the minimum matching error in the n second candidate poses as the pose of the laser radar.

2. The extrinsic calibration method of claim 1, wherein, The determination of the plurality of spatial points in the point cloud outline corresponding to the plurality of feature points of the calibration image comprises: determining a plurality of candidate scene images with a similarity between the calibration image and the plurality of candidate scene images satisfying a preset condition in a plurality of scene images taken in different ranges of the calibration scene in advance; determining a candidate scene image containing the plurality of feature points in the plurality of candidate scene images as a standard scene image; determining a plurality of spatial points in the point cloud outline corresponding to a plurality of feature points of the standard scene image as the plurality of spatial points in the point cloud outline corresponding to the plurality of feature points of the calibration image according to a predetermined correspondence between feature points of the plurality of scene images and spatial points in the point cloud outline.

3. The extrinsic calibration method of claim 1 or 2, wherein, The point cloud outline comprises a global point cloud outline of the calibration scene.

4. The extrinsic parameter calibration method according to claim 1 or 2, wherein the point cloud outline comprises a point cloud outline of a first scene range in the calibration scene and a point cloud outline of a second scene range in the calibration scene, wherein when the autonomous mobile device is in a first device pose in the calibration scene, a capture range of the camera is located within the first scene range, and a capture range of the laser radar is located within the second scene range.

5. The extrinsic parameter calibration method according to claim 4, wherein the point cloud outline of the first scene range comprises a point cloud outline of a first calibration object, and the point cloud outline of the second scene range comprises a point cloud outline of a second calibration object.

6. The extrinsic parameter calibration method according to claim 4, wherein The first scene range and the second scene range do not overlap each other.

7. The method of claim 1 or 2, wherein, Before the determining the pose of the camera according to the calibration image and the point cloud profile of the calibration scene and the determining the pose of the laser radar according to the calibration point cloud frame and the point cloud profile, the method further comprises: acquiring information of the calibration scene to construct the point cloud profile.

8. The method of calibrating extrinsic parameters according to claim 7, wherein, The acquiring information of the calibration scene to construct the point cloud profile comprises: capturing the calibration scene at two different positions in the calibration scene to obtain two scene image sets, wherein each of the scene image sets comprises a plurality of scene images; calculating the point cloud profile according to the two scene image sets.

9. The method of calibrating extrinsic parameters according to claim 1 or 2, wherein, The at least one calibration image comprises a plurality of calibration image sets, and the at least one calibration point cloud frame comprises a plurality of calibration point cloud frame sets, wherein the plurality of calibration image sets and the plurality of calibration point cloud frame sets are respectively acquired by the camera and the laser radar when the autonomous mobile device is at a plurality of device poses in the calibration scene, wherein the determining the pose of the camera according to the calibration image and the point cloud profile of the calibration scene comprises: determining a plurality of first camera poses corresponding to the plurality of device poses of the camera according to the plurality of calibration image sets and the point cloud profile; optimizing the plurality of first camera poses to obtain the pose of the camera, wherein the determining the pose of the laser radar according to the calibration point cloud frame and the point cloud profile comprises: determining a plurality of first radar poses corresponding to the plurality of device poses of the laser radar according to the plurality of calibration point cloud frame sets and the point cloud profile; optimizing the plurality of first radar poses to obtain the pose of the laser radar.

10. The method of calibrating extrinsic parameters according to claim 9, wherein, Each of the calibration image sets comprises a plurality of calibration images, and each of the calibration point cloud frame sets comprises a plurality of calibration point cloud frames, wherein the determining a plurality of first camera poses corresponding to the plurality of device poses of the camera according to the plurality of calibration image sets and the point cloud profile comprises: determining a plurality of second camera poses corresponding to each of the device poses of the camera according to the plurality of calibration images and the point cloud profile; optimizing the plurality of second camera poses to obtain each of the first camera poses, wherein the determining a plurality of first radar poses corresponding to the plurality of device poses of the laser radar according to the plurality of calibration point cloud frame sets and the point cloud profile comprises: determining a plurality of second radar poses corresponding to each of the device poses of the laser radar according to the plurality of calibration point cloud frames and the point cloud profile; optimizing the plurality of second radar poses to obtain each of the first radar poses.

11. An external parameter calibration device, characterized in that comprises: an acquisition module, configured to acquire at least one calibration image collected by a camera in a calibration scene and at least one calibration point cloud frame collected by a laser radar in the calibration scene, wherein the camera and the laser radar are fixedly arranged on an autonomous mobile device; determine a pose of the camera according to the calibration image and a point cloud outline of the calibration scene, and determine a pose of the lidar according to the calibration point cloud frame and the point cloud outline; a calibration module configured to determine a calibration extrinsic parameter of the camera and the lidar according to the pose of the camera and the pose of the lidar, wherein the determination module comprises a camera determination unit and a radar determination unit, wherein the camera determination unit is configured to determine a plurality of spatial points in the point cloud outline corresponding to a plurality of feature points of the calibration image, and obtain position information of the plurality of spatial points; and calculate the pose of the camera according to the position information of the plurality of spatial points, and the radar determination unit is configured to estimate m first candidate poses of the lidar according to the point cloud outline, and determine m first candidate point cloud frames corresponding to the m first candidate poses; register the m first candidate point cloud frames with the calibration point cloud frame, determine n first candidate poses corresponding to n first candidate point cloud frames closest to the calibration point cloud frame as second candidate poses; and determine a second candidate pose with a minimum matching error in the n second candidate poses as the pose of the lidar.

12. The external parameter calibration apparatus according to claim 11, wherein, Further comprising: a construction module configured to collect information of the calibration scene to construct the point cloud outline.

13. An autonomous mobile device, comprising: comprising: at least one camera configured to collect images; at least one lidar configured to collect point cloud frames; a processor configured to execute the extrinsic calibration method of any one of claims 1-10 to calibrate the extrinsic parameter of the at least one camera and the at least one lidar.

14. An electronic device, comprising: comprising: a memory configured to store computer instructions; a processor configured to execute the computer instructions to implement the extrinsic calibration method of any one of claims 1-10.

15. A computer-readable storage medium, characterized in that, a computer instruction stored in the memory, when executed by the processor, to implement the extrinsic calibration method of any one of claims 1-10.

Images