Indoor scene-oriented heterogeneous sensor array online automatic calibration method

Abstract

The invention provides a heterogeneous sensor array online automatic calibration method for an indoor scene, and the method comprises the steps: obtaining point cloud data of a heterogeneous sensor array laser radar and RGB images of a multi-camera array, carrying out the preprocessing, and carrying out the cross-modal matching algorithm, the pose solving algorithm and the alignment function according to the target prior; determining external parameter information of the laser radar-camera; determining global pose information of the multi-camera array based on a pixel feature region calibration algorithm and a scale prior global pose optimization algorithm of image semantic segmentation; and according to the point cloud data information and the RGB image information of the continuous frames, laser radar-camera continuous external parameter information and all camera continuous pose information are obtained, and external parameter online automatic calibration of the heterogeneous sensor array is determined. According to the method, camera RGB image information of an indoor scene and point cloud data acquired by the laser radar are fully utilized, laser radar-camera and multi-camera online calibration is realized, high automation and real-time online performance are realized, and the calibration precision and stability are improved.

Description

Technical Field

[0001] The present application relates to the technical field of automatic sensor calibration, and in particular, to an online automatic calibration method and system for heterogeneous sensor arrays for indoor scenes. Background Art

[0002] With the steady and rapid development of the economy and the continuous improvement of science and technology, the number of private cars is increasing, and low traffic efficiency has gradually become an important problem restricting people's lives and development. Among the many driving problems faced by drivers, "difficult parking" is the most prominent. Driven by policies and markets, intelligent parking services for indoor scenes have been developed to solve this problem. Automated Valet Parking (AVP) has become a current research hotspot. Compared with the single-vehicle intelligent parking solution, the parking solution for the field has the advantages of rich global information, interactive communication in the parking lot, and optimal allocation. Based on the design ideas of LiDAR modeling and camera perception and positioning, the heterogeneous sensor array that integrates the three-dimensional information of LiDAR point cloud and the environmental texture information of RGB camera images has promoted the implementation of the field-side intelligent AVP solution.

[0003] Heterogeneous sensor array, fuses the depth information of LiDAR point cloud into a multi-camera array, captures color images of indoor parking lots from different positions and angles, and reconstructs the color and geometry information of the scene. Online automatic accurate and fast calibration of heterogeneous sensor array extrinsics (translation and rotation) is a basic step in building a heterogeneous sensor array network that can capture large scenes and merge multi-view views into the same coordinate.

[0004] Heterogeneous sensor array extrinsic calibration involves lidar-camera extrinsic calibration and multi-camera array extrinsic calibration.

[0005] LiDAR-camera extrinsic calibration methods can be divided into methods using specific targets and methods without specific targets. The methods using specific targets require the production of specific targets and the design of stable and accurate detection algorithms. At the same time, due to the limitations of the calibration objects, they are not real-time and are not conducive to maintenance. The methods without specific targets are highly dependent on the scene and require an accurate understanding of the scene features and the design of better optimization algorithms to improve the calibration accuracy. Online automatic calibration of LiDAR-camera extrinsic parameters must be a method without specific targets. Although there are some LiDAR-camera extrinsic methods without specific targets, most of them are research methods based on the vehicle side and need to be combined with the vehicle side applications or scenarios. They have poor robustness, low accuracy and stability, and are not suitable for indoor high-precision positioning scenarios.

[0006] The calibration of multi-camera array extrinsic parameters can be divided into methods that use calibration objects and methods that do not use calibration objects. Methods that use calibration objects, such as calibration plates and calibration balls, require the design of stable and accurate detection algorithms and better error functions. They also require manual empirical operations to find the placement of calibration objects, which is not real-time, and when the number of cameras is large, the workload is huge; methods that do not use calibration objects are highly dependent on the scene, and the understanding of unknown scene features is inaccurate, resulting in low calibration accuracy. Online automatic calibration of multi-camera array extrinsic parameters must be a method that does not use calibration objects, but most camera calibration methods that do not use calibration objects have poor robustness, rely on the accuracy of understanding scene features, and the calibration accuracy is not guaranteed. When the number of cameras is large, the calibration error will increase significantly. Summary of the invention

[0007] In view of the defects in the prior art, the purpose of this application is to provide an online automatic calibration method for heterogeneous sensor arrays for indoor scenes, which realizes lidar-camera and multi-camera online calibration respectively, with high automation and real-time online, and improves the calibration accuracy and stability.

[0008] In one aspect of the present application, a method for online automatic calibration of a heterogeneous sensor array for indoor scenes is provided, the method comprising:

[0009] Obtain the point cloud data of the heterogeneous sensor array lidar and the RGB image of the multi-camera array, and perform preprocessing, and determine the external parameter information of the lidar-camera according to the target prior cross-modal matching algorithm, pose solution algorithm and alignment function;

[0010] Determine the global pose information of the multi-camera array based on the pixel feature region calibration algorithm and scale prior global pose optimization algorithm based on image semantic segmentation;

[0011] According to the point cloud data information and RGB image information of continuous frames, the continuous external parameter information of the lidar-camera and the continuous pose information of all cameras are obtained, and the online automatic calibration of the external parameters of the heterogeneous sensor array is determined.

[0012] Furthermore, the obtaining of the heterogeneous sensor array lidar point cloud data and the RGB image of the multi-camera array and preprocessing includes:

[0013] Real-time acquisition of heterogeneous sensor array LiDAR point cloud data and multi-camera array RGB image information;

[0014] Performing distortion correction processing on the RGB image information according to the calibrated internal parameters of the camera to determine the RGB image after the distortion correction processing;

[0015] The point cloud data is first preprocessed according to a point cloud preprocessing algorithm to determine the laser radar point cloud data after the first preprocessing.

[0016] Furthermore, the cross-modal matching algorithm, posture solving algorithm and alignment function based on the target prior; determining the external parameter information of the laser radar-camera includes:

[0017] According to the target prior detection algorithm, the RGB image after distortion correction is screened to determine whether the current LiDAR-Camera field of view scene is conducive to LiDAR-Camera extrinsic calibration;

[0018] According to the cross-modal target matching algorithm, cross-modal target matching is performed on favorable scenes to determine the corresponding mapping between the camera's RGB image target and the target center and vertex of the lidar point cloud data;

[0019] According to the pose initialization algorithm and alignment function, the lidar point cloud data is mapped to the RGB image pixels one by one, and then refined and optimized to determine the external parameter information of the lidar-camera.

[0020] Furthermore, the cross-modal target matching is performed on the favorable scene according to the cross-modal target matching algorithm, and the corresponding mapping between the RGB image target of the camera and the target center and vertex of the laser radar point cloud data is determined specifically including:

[0021] Build a deep neural network for cross-modal matching;

[0022] Training deep neural networks;

[0023] A trained deep neural network is used to establish a one-to-one mapping between the target in the lidar point cloud data and the target in the RGB image based on the RGB image after distortion correction and the lidar point cloud data after the first preprocessing. The corresponding data pairs of 2d-3d target centers are collected to determine the data pairs of the lidar target center to the RGB image target center.

[0024] Furthermore, the one-to-one mapping of the laser radar point cloud to the camera RGB image pixels is performed according to the pose initialization algorithm and the alignment function, and the extrinsic parameter information of the laser radar-camera is determined, which specifically includes:

[0025] According to the particle fitness alignment function, the data pair from the lidar target center to the RGB image target center is gradually refined, and based on the refined data pair from the lidar target center to the RGB image target center, the current initialization pose of the lidar-camera is determined by solving PnP;

[0026] According to the current initialization pose of the LiDAR-camera, the particle fitness alignment function is used again to directly optimize the alignment of the RGB image target pixels and the LiDAR target point cloud data to determine the external parameter information of the LiDAR-camera.

[0027] Furthermore, the pixel feature region calibration algorithm based on image semantic segmentation and the scale prior global pose optimization algorithm determine the global pose information of the multi-camera array specifically including:

[0028] According to the image semantic segmentation algorithm, the feature area of ​​the current two-dimensional image is segmented to determine the pixel features of different areas;

[0029] Determine the current scale-free global pose information of the multi-camera array based on the pixel region feature calibration algorithm;

[0030] According to the scale prior global pose optimization algorithm, the prior LiDAR point cloud scale information is integrated to perform alignment optimization and determine the current global pose information of the multi-camera array.

[0031] Furthermore, the scale prior global pose optimization algorithm includes: a lidar point cloud scale fusion algorithm, which determines the RGB image scale prior information based on the lidar-camera external parameter information and the prior lidar point cloud scale information.

[0032] Furthermore, the method of obtaining the continuous external parameter information of the laser radar-camera and the continuous pose information of all cameras based on the continuous point cloud data information and the RGB image information, and determining the online automatic calibration of the external parameters of the heterogeneous sensor array, specifically includes:

[0033] The lidar-camera extrinsic parameter information updated in the previous frame is associated with the current lidar-camera extrinsic parameter information to determine the lidar-camera continuous extrinsic parameter information;

[0034] Associating the global pose information of the multi-camera array updated in the previous frame with the global pose information of the current multi-camera array to determine the global continuous external parameter information of the multi-camera array;

[0035] The lidar-camera continuous extrinsic parameter information and the multi-camera array global continuous extrinsic parameter information are unified in a unified spatiotemporal coordinate system to determine the global continuous pose information of the heterogeneous sensor array.

[0036] The second aspect of the present application provides an online automatic calibration system for heterogeneous sensor arrays for indoor scenes, comprising:

[0037] A laser radar module, wherein the laser radar module includes at least two laser radars, and the at least two laser radars are arranged on both sides of the road near the entrance and exit of the indoor scene and face the ground of the indoor scene, and the laser radar module is used to collect dynamic and static point cloud information of the indoor scene;

[0038] A multi-camera array module, the multi-camera array module comprising a plurality of cameras, the plurality of cameras being arranged at a ceiling position of the indoor scene road and / or parking space area and facing the ground of the indoor scene, the multi-camera array module being used to collect dynamic and static RGB image information of the indoor scene;

[0039] A communication network module, the communication network module includes a circuit power supply part and an information transmission network part, and is used for powering the circuits of the heterogeneous sensor array;

[0040] A main control computer, wherein the main control computer is connected to the laser radar module and the camera array module respectively through the communication network module, and is used for performing data analysis and data processing on the laser radar point cloud information and RGB image information transmitted back by the camera, including but not limited to the laser radar-camera online automatic calibration, the multi-camera online automatic calibration, and the heterogeneous sensor array global online automatic calibration.

[0041] Furthermore, it also includes a mounting bracket and a power supply module;

[0042] The mounting bracket is used to mount the laser radar module, the multi-camera array module, and the power supply module on the ceiling of the indoor scene;

[0043] The power supply module is used to provide electrical energy to the laser radar module, the multi-camera array module, and the main control computer.

[0044] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0045] (1) The online automatic calibration method of heterogeneous sensor arrays for indoor scenes of the present invention realizes online calibration of lidar-camera and multiple cameras respectively, which is highly automated and real-time online, thereby improving the calibration accuracy and stability.

[0046] (2) The present invention can realize online automatic calibration of heterogeneous sensor arrays for indoor scenes, with low cost, strong robustness, simple operation, reduced manual participation, and easy real-time maintenance.

[0047] (3) The online automatic calibration method of heterogeneous sensor arrays for indoor scenes of the present invention broadens the solution ideas for problems in similar fields, promotes the implementation of field-side intelligent AVP solutions, and has high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0049] Figure 1This is a flow chart of an online automatic calibration method for a heterogeneous sensor array for indoor scenes according to an embodiment of the present application.

[0050] Figure 2 This is a specific flow chart of step S1 of an embodiment of the present application.

[0051] Figure 3 This is a structural diagram of an online automatic calibration system for heterogeneous sensor arrays for indoor scenes according to an embodiment of the present application.

[0052] Figure 4 This is a functional schematic diagram of an online automatic calibration system for heterogeneous sensor arrays for indoor scenes according to an embodiment of the present application.

[0053] Figure 5 This is a structural diagram of an online automatic calibration method for a heterogeneous sensor array for indoor scenes according to an embodiment of the present application.

[0054] Figure 6 It is a specific flow chart of the online automatic calibration method of heterogeneous sensor arrays for indoor scenes in this application.

[0055] In the figure: 1. Mounting bracket; 2. Heterogeneous sensor array; 3. Camera scanning coverage area; 4. LiDAR shooting coverage area. DETAILED DESCRIPTION

[0056] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0057] The present application provides an online automatic calibration method and system for heterogeneous sensor arrays for indoor scenes, which fully utilizes the camera RGB image information of indoor scenes and the strong robustness of point cloud information collected by lidar. The method is suitable for indoor environments with stable ambient light, has a high degree of automation and strong robustness, and can reduce operating costs and difficulty while ensuring positioning accuracy and maintain good stability.

[0058] Reference Figure 1 As shown, an online automatic calibration method for a heterogeneous sensor array for indoor scenes according to an embodiment of the present application is provided, and the method includes:

[0059] S1. Obtain the point cloud data of the heterogeneous sensor array lidar and the RGB image of the multi-camera array, and perform preprocessing. According to the target prior cross-modal matching algorithm, pose solution algorithm and alignment function; determine the external parameter information of the lidar-camera.

[0060] S2. A pixel feature region calibration algorithm based on image semantic segmentation and a scale prior global pose optimization algorithm are used to determine the global pose information of the multi-camera array.

[0061] S3. According to the point cloud data information and RGB image information of continuous frames, the continuous external parameter information of the lidar-camera and the continuous pose information of all cameras are obtained to determine the online automatic calibration of the external parameters of the heterogeneous sensor array.

[0062] This application utilizes a cross-modal matching algorithm and a pose solving algorithm, combined with the point cloud data of the lidar and the RGB image of the camera, to more accurately determine the external parameter information between the lidar and the camera, thereby improving the overall calibration accuracy. It can complete the calibration in real time based on the observation information generated by the sensor during movement, has stronger adaptability and flexibility, reduces costs, and through continuous frame data processing, can dynamically correct the calibration parameters, reduce the problem of inaccurate calibration parameters caused by factors such as carrier vibration, long-term operation and temperature changes, and improve the stability and reliability of the system.

[0063] In the above steps, the extrinsic parameter information and the posture information have the same meaning but different representation methods. The extrinsic parameter information describes the relative position relationship between the two sensors and is generally represented in matrix form, including the rotation matrix R and the translation vector T. The posture information describes the position and posture of a single sensor in the global (another sensor) coordinate system and is generally represented in the form of six degrees of freedom, including translation (position, usually x, y, z coordinates in three-dimensional space) and rotation posture (usually angles around the x, y, z axes). The two can be directly converted to each other.

[0064] Reference Figure 5 As shown, specifically, by acquiring the point cloud data of the laser radar in the heterogeneous sensor array and the RGB image of the multi-camera array, the point cloud data and the RGB image are preprocessed to improve accuracy and efficiency; then, the target prior cross-modal matching algorithm is used to match the point cloud data of the laser radar with the target in the RGB image of the multi-camera array, and the relative pose between the laser radar and the camera, that is, the extrinsic parameter information, is calculated according to the matching result through the pose solution algorithm; then, based on the pixel feature area calibration algorithm of the image semantic segmentation, the RGB image is semantically segmented to extract the key feature area, and the scale prior global pose optimization algorithm is used to determine the global pose information of the multi-camera array in combination with the semantic segmentation result; finally, according to the point cloud data information and RGB image information of continuous frames, the above-mentioned algorithm is used to continuously calculate the continuous extrinsic parameter information between the laser radar and the camera and the continuous pose information of all cameras, and the calibration parameters are updated and adjusted in real time through the online calibration algorithm to adapt to the changes of the sensor in the actual motion process.

[0065] Reference Figure 2As shown, in some possible embodiments, S1, obtaining heterogeneous sensor array lidar point cloud data and RGB images of multi-camera arrays, and preprocessing includes: S11, obtaining heterogeneous sensor array lidar point cloud data and RGB image information of multi-camera arrays in real time.

[0066] S12: performing distortion correction processing on the RGB image information according to the calibrated internal parameters of the camera to determine the RGB image after the distortion correction processing.

[0067] S13. Perform a first preprocessing on the point cloud data according to a point cloud preprocessing algorithm to determine the laser radar point cloud data after the first preprocessing.

[0068] In the above embodiment, an image pre-processing module and a point cloud data processing module are included.

[0069] Specifically, the laser radar point cloud data of the heterogeneous sensor array and the RGB image of the multi-camera array are acquired in real time, and then the image preprocessing module performs distortion correction processing on the RGB image information of the camera according to the calibration internal parameters of the camera to determine the RGB image after the distortion correction processing, and then the point cloud preprocessing module uses the point cloud preprocessing algorithm to perform the first preprocessing on the scene point cloud data information to determine the laser radar point cloud data after the first preprocessing

[0070] Among them, the camera's intrinsic parameters and distortion coefficients can be determined using Zhang Zhengyou's calibration algorithm.

[0071] Among them, the first preprocessing includes external point removal processing, which uses a statistical filtering algorithm to remove flying points, and eliminates points with positions or features that are significantly different from surrounding points to improve the quality of point cloud data.

[0072] Through the above embodiments, the point cloud preprocessing module uses a statistical filtering algorithm to process flying points in the current environment, eliminates points with positions or features that are significantly different from the surrounding points, improves the quality of point cloud data, helps to improve the robustness of lidar-camera extrinsic calibration, and improves the accuracy of lidar point cloud prior information fusion in multi-camera array calibration.

[0073] In some possible embodiments, based on a target prior cross-modal matching algorithm, a pose solving algorithm, and an alignment function; determining the external parameter information of the lidar-camera includes: S14, based on a target prior detection algorithm, screening the RGB image after distortion correction processing to determine whether the current field of view scene of the lidar-camera is conducive to the lidar-camera external parameter calibration.

[0074] S15. According to the cross-modal target matching algorithm, cross-modal target matching is performed on the favorable scene to determine the corresponding mapping between the camera's RGB image target and the laser radar point cloud data target center and vertex.

[0075] S16. According to the pose initialization algorithm and alignment function, the lidar point cloud data is mapped to the RGB image pixels one by one, and refined and optimized to determine the external parameter information of the lidar-camera.

[0076] In the above embodiment, it includes a target priori detection unit, a cross-modal target matching unit and a posture solving unit.

[0077] Specifically, first, the target prior detection unit adopts a lightweight target detection algorithm to determine the scene that is conducive to the laser radar-camera extrinsic calibration. Among them, the lightweight target detection algorithm includes the YOLO-Fastest target detection algorithm and a lightweight replaceable algorithm that is better than YOLO-Fastest. The scenes that are conducive to the laser radar-camera extrinsic calibration include the overlapping area of ​​the laser radar scanning coverage area and the camera scanning coverage area, where there are vehicle targets, people and other targets, and the overlapping area of ​​the laser radar scanning coverage area and the camera scanning coverage area. The central sub-image is a sub-image that takes the center of the original image as the origin and proportionally crops the surrounding area of ​​the original image.

[0078] Next, the cross-modal target matching unit uses a cross-modal matching algorithm to perform cross-modal target matching on the favorable scene, and determines the corresponding mapping between the vehicle target in the camera RGB image and the vehicle target center and special vertices in the lidar point cloud. Among them, the cross-modal matching algorithm includes a target-level cross-modal registration network, which uses a target-level cross-modal registration network to collect data pairs and corresponding mappings between the camera RGB image target and the lidar point cloud target center and special vertices.

[0079] Finally, the pose solving unit uses the pose solving algorithm and alignment function to refine and optimize the one-to-one mapping of the lidar point cloud to the camera RGB image pixels, and determine the external parameter information of the lidar-camera.

[0080] In the above embodiment, the scanning coverage area of ​​the laser radar is larger than the camera scanning coverage area and may include the camera scanning coverage area, which is beneficial for the scene of the laser radar-camera external parameter calibration, such as the scene where there are vehicle targets, people and other targets in the camera scanning coverage area, and the scene where there are vehicle targets, people and other targets in the central sub-image of the camera scanning coverage area.

[0081] In some possible embodiments, cross-modal target matching is performed on favorable scenes according to a cross-modal target matching algorithm, and determining the corresponding mapping between the camera's RGB image target and the lidar point cloud data target center and vertices specifically includes: constructing a deep neural network for cross-modal matching; and training the deep neural network.

[0082] According to the trained deep neural network, a one-to-one mapping of the target in the lidar point cloud data to the target in the RGB image is established according to the RGB image after distortion correction and the lidar point cloud data after the first preprocessing, and the corresponding data pairs of 2d-3d target centers are collected to determine the data pairs of the lidar target center to the RGB image target center.

[0083] Specifically, first, according to the requirements of cross-modal target matching, the architecture of the deep neural network is designed. The network should contain multiple convolutional layers (for processing image data) and fully connected layers (for processing point cloud data and fusing information from different modalities). For example, an encoder-decoder structure can be used, in which the encoder part is used to extract the features of RGB images and lidar point clouds respectively, and the decoder part is used to fuse these features and achieve cross-modal matching; prepare a dataset containing RGB images and corresponding lidar point cloud data. The dataset contains rich scenes and diverse targets to ensure that the trained model has good generalization ability. Finally, the RGB image is subjected to distortion correction processing to eliminate image distortion caused by camera motion or installation errors. At the same time, the lidar point cloud data is preprocessed, such as denoising, filtering, and segmentation, to extract the target point cloud of interest.

[0084] Next, the trained deep neural network is used to process the input data to achieve cross-modal matching of the target in the lidar point cloud data to the target in the RGB image. Through the matching results output by the network, the corresponding data pairs of 2D-3D target centers are collected, and the data pairs from the lidar target center to the RGB image target center are determined.

[0085] By constructing and training a deep neural network, this application can automatically learn the complex mapping relationship between RGB images and lidar point clouds, improve the accuracy of cross-modal target matching, and the optimized deep neural network can complete the cross-modal target matching task in a shorter time, meeting application scenarios with high real-time requirements.

[0086] In the above embodiment, according to the pose initialization algorithm and the alignment function, the laser radar point cloud is mapped one by one to the camera RGB image pixel points, and is refined and optimized to determine the external parameter information of the laser radar-camera, specifically including: according to the particle fitness alignment function, the laser radar target center to RGB image target center data pair is gradually refined, and based on the refined laser radar target center to RGB image target center data pair, the current initialization pose of the laser radar-camera is determined by solving PnP.

[0087] According to the current initialization pose of the LiDAR-camera, the particle fitness alignment function is used again to directly optimize the alignment of the RGB image target pixels and the LiDAR target point cloud data to determine the external parameter information of the LiDAR-camera.

[0088] Specifically, the distortion-corrected camera RGB image and the preprocessed lidar point cloud data are obtained; a particle fitness alignment function based on particle swarm optimization (PSO) or other optimization algorithms is used to preliminarily align the target center in the lidar point cloud with the target center in the camera RGB image, and the position and velocity of the particles (representing possible lidar-camera external parameter information) are iteratively adjusted to minimize the deviation between the target centers. Then, based on the refined lidar target center to RGB image target center data pair, the perspective-n-point problem (PnP) algorithm is used to solve the initial pose between the lidar and the camera. The PnP algorithm uses known spatial points (points in the lidar) and their projections on the image plane (points in the RGB image) to estimate the camera pose. Then, after obtaining the initial pose, a more refined particle fitness alignment function is used to directly optimize the alignment of the target pixels in the RGB image with the target point cloud data in the lidar; the pose parameters of the lidar-camera are further adjusted to minimize the error between the pixel points and the corresponding point cloud points.

[0089] Among them, the pose solving algorithm includes a multi-point perspective solving algorithm PnP (Perspective-n-Points), and the alignment functions include image pixel alignment and image target alignment functions. The pose solving unit uses the image pixel alignment function to refine the camera RGB image vehicle target and the lidar point cloud vehicle target center and special vertex data pairs, and uses the multi-point perspective solving algorithm PnP (Perspective-n-Points) to solve the initial pose, and then uses the image target alignment function to directly optimize the pose.

[0090] Through the above technical solution, the lidar-camera calibration adopts a lightweight target detection algorithm to determine the scene that is conducive to the lidar-camera extrinsic calibration without increasing the computing power cost, and then uses a cross-modal registration network to match the target in the scene, collect 2D-3D data point pairs of the target center and vertex, and then use the pose solution algorithm to refine and optimize the one-to-one mapping of the lidar point cloud to the camera RGB image pixel points to determine the lidar-camera extrinsic parameters in the current scene.

[0091] This application can automatically calibrate the lidar-camera extrinsic parameters online with a high degree of automation, without the need for initial pose estimation and special calibration targets. In addition, the calibration scenes are screened by a lightweight target detection algorithm, which saves computing power while ensuring the stability of the lidar-camera extrinsic calibration results and improving the calibration accuracy.

[0092] In some possible embodiments, S2, based on the pixel feature area calibration algorithm of image semantic segmentation and the scale prior global pose optimization algorithm, determines the global pose information of the multi-camera array, specifically including: S21, according to the image semantic segmentation algorithm, performs feature area segmentation on the current two-dimensional image, and determines the pixel features of different areas.

[0093] S22. Determine the current scale-free global pose information of the multi-camera array according to a pixel region feature calibration algorithm.

[0094] S23. According to the scale prior global pose optimization algorithm, the prior laser radar point cloud scale information is integrated to perform alignment optimization and determine the current global pose information of the multi-camera array.

[0095] Among them, the multi-camera array calibration includes an image semantic analysis unit, a scale-free global pose solution unit and a scale-prior global pose optimization unit.

[0096] In the above embodiment, specifically, the image semantic segmentation unit uses an image semantic segmentation algorithm to segment the camera RGB image into pixel feature regions and determine the pixel features of different regions. The image semantic segmentation algorithm includes a lightweight semantic segmentation algorithm CSFCN, and the image semantic segmentation unit uses a lightweight semantic segmentation algorithm CSFCN to quickly segment the camera RGB image into pixel feature regions and determine the pixel features of different regions.

[0097] Next, the scale-free global pose solving unit uses the pixel region feature calibration algorithm to determine the scale-free global pose information of the multi-camera array. The pixel region feature calibration algorithm includes the pixel region feature alignment algorithm and the multi-point perspective solving algorithm PnP (Perspective-n-Points). The pixel region feature calibration uses the minimum mean square value of the pixel region feature alignment algorithm to refine multiple feature points, and then uses the multi-point perspective solving algorithm PnP (Perspective-n-Points) to determine the scale-free global pose information of the multi-camera array.

[0098] Finally, the scale prior global pose optimization unit performs alignment optimization based on the prior LiDAR point cloud information to determine the global pose information of the multi-camera array. The prior LiDAR point cloud information is obtained through LiDAR-camera extrinsics. The prior LiDAR point cloud information can be replaced by other prior scale information that can be integrated into the multi-camera array.

[0099] Through the above technical solution, the multi-camera array calibration uses an image semantic segmentation network to segment pixel feature regions, determine the pixel features of different regions, and then use the pixel region feature calibration algorithm to obtain the scale-free global pose information of the multi-camera array, and then use the prior laser radar point cloud information for alignment to determine the global pose information of the multi-camera array. Through the present disclosure, the multi-camera array external parameters can be automatically calibrated online, with a high degree of automation, reducing labor costs, and the integrated prior laser radar point cloud information has obvious three-dimensional features and a large number of point sets, which better guarantees the accuracy and robustness of the multi-camera array external parameters.

[0100] Specifically, the scale prior global pose optimization algorithm includes: a lidar point cloud scale fusion algorithm, which determines the RGB image scale prior information based on the lidar-camera external parameter information and the prior lidar point cloud scale information.

[0101] In some possible embodiments, S3, based on continuous point cloud data information and RGB image information, obtain the lidar-camera continuous external parameter information and all camera continuous pose information, and determine the online automatic calibration of the heterogeneous sensor array external parameters, specifically including: S31, associate the lidar-camera external parameter information updated in the previous frame with the current lidar-camera external parameter information to determine the lidar-camera continuous external parameter information.

[0102] In the above embodiment, the position and posture are more conducive to intuitive display, and the external parameter information is more conducive to the mathematical operation of coordinate system conversion.

[0103] S32, associating the multi-camera array global pose information updated in the previous frame with the current multi-camera array global pose information to determine the multi-camera array global continuous extrinsic parameter information.

[0104] S33, unify the laser radar-camera continuous extrinsic parameter information and the multi-camera array global continuous extrinsic parameter information in a unified spatiotemporal coordinate system to determine the global continuous pose information of the heterogeneous sensor array.

[0105] Among them, the continuous external parameter information of the lidar-camera and the continuous pose information of all cameras are obtained according to the continuous frame point cloud information and RGB image information. The data association algorithm is used to associate the external parameter information updated in the previous frame with the current external parameter information to determine the continuous external parameter information of the lidar-camera and the continuous pose information of all cameras.

[0106] This application can significantly improve the accuracy of heterogeneous sensor data fusion by continuously tracking and updating the external parameter information between the lidar and the camera, as well as the global pose information of the multi-camera array, and achieve more accurate environmental perception and target tracking in the fields of autonomous driving and robot navigation; online automatic calibration technology can adapt to environmental changes in real time and maintain the stability and reliability of the system; by associating the sensor information of the previous frame and the current frame, repeated calibration process is avoided and the efficiency of data processing is improved.

[0107] Specifically, continuous point cloud data and RGB image data are received, and for each frame of data, preliminary external parameter calibration of the lidar and camera is performed using a feature point matching, PnP algorithm or deep learning method; the lidar-camera external parameter information updated in the previous frame is associated with the preliminary calibration result of the current frame to determine the lidar-camera continuous external parameter information; for each camera in the multi-camera array, its continuous global pose information is estimated using a visual odometer or SLAM algorithm; the global pose information of the multi-camera array updated in the previous frame is associated with the estimated result of the current frame to ensure the consistency and accuracy of the global pose information; the lidar-camera continuous external parameter information and the global continuous external parameter information of the multi-camera array are unified into a unified space-time coordinate system; the finally determined global continuous pose information of the heterogeneous sensor array is output to the subsequent data processing or decision-making system.

[0108] Before executing the online automatic calibration method for heterogeneous sensor arrays for indoor scenes, the following preparations are required:

[0109] Determine the internal parameters and distortion coefficients of all cameras; ensure that the network power and network of the heterogeneous sensor array for indoor scenes are normal and start up normally; ensure that all algorithms stored in the main control computer for data analysis and data processing of the transmitted LiDAR point cloud information and the RGB image information transmitted by the camera are operable.

[0110] Reference Figure 6 As shown, the specific implementation process is:

[0111] Acquire heterogeneous sensor array LiDAR point cloud data and camera array RGB image information in real time; obtain distortion-corrected RGB images through image preprocessing module; obtain LiDAR point cloud data after first preprocessing through point cloud preprocessing module; screen the distortion-corrected RGB images of the camera according to target prior detection algorithm to determine whether the current LiDAR-camera field of view scene is conducive to the LiDAR-camera external parameter calibration; perform cross-modal target matching on favorable scenes according to cross-modal target matching algorithm to determine the corresponding mapping between camera RGB image targets and LiDAR point cloud data target centers and vertices; perform data pair refinement on the corresponding mapping between target centers and vertices according to pose initialization algorithm, and use PnP algorithm to determine the current initialization pose of LiDAR-camera; directly optimize and determine the LiDAR-camera according to pose optimization algorithm. The method comprises the following steps: obtaining the current extrinsic parameter information of the multi-camera array; segmenting the feature regions of the current two-dimensional image according to the image semantic segmentation algorithm to determine the pixel features of different regions; determining the current scale-free global pose information of the multi-camera array according to the pixel region feature calibration algorithm; fusing the prior scale information of the lidar point cloud according to the scale prior global pose optimization algorithm to determine the current global pose information of the multi-camera array; correlating the lidar-camera extrinsic parameter information updated in the previous frame with the current lidar-camera extrinsic parameter information to determine the lidar-camera continuous extrinsic parameter information; correlating the multi-camera array global pose information updated in the previous frame with the current multi-camera array global pose information to determine the multi-camera array global continuous extrinsic parameter information; unifying the lidar-camera continuous extrinsic parameter information and the multi-camera array global continuous extrinsic parameter information in a unified spatiotemporal coordinate system to determine the global continuous pose information of the heterogeneous sensor array.

[0112] Among them, the core algorithms of lidar-camera calibration, multi-camera array calibration, and global online automatic calibration of heterogeneous sensor arrays all adopt lightweight architecture. The algorithms run very fast and consume very little time. The global continuous pose information of heterogeneous sensor arrays is very timely, so as to realize the online automatic calibration of heterogeneous sensor arrays.

[0113] Reference Figure 3 to Figure 4 As shown, the second aspect of the present application provides an online automatic calibration system for heterogeneous sensor arrays for indoor scenes, including a lidar module, a camera array module, a communication network module, and a main control computer.

[0114] The laser radar module includes at least two laser radars, which are arranged on both sides of the road near the entrance and exit of the indoor scene and facing the ground of the indoor scene. The laser radar module is used to collect dynamic and static point cloud information of the indoor scene; including but not limited to vehicle targets, people, etc., and introduce three-dimensional scale information into the heterogeneous sensor array 2.

[0115] Among them, the angle between the direction of the laser radar and the ground can be adjusted according to the actual height of the ceiling of the indoor scene.

[0116] In a specific implementation, a high-precision ML30S laser radar can be used to collect point cloud information when the laser radar scans the coverage area, including scenes with vehicle targets, people and other targets passing by and scenes without any targets, such as Figure 3 As shown, the scanning coverage area of ​​a single laser radar is trapezoidal, and the laser radar shooting coverage areas 4 installed on both sides of the indoor scene road have partial overlapping areas, and the adjacent laser radar shooting coverage areas 4 and camera scanning coverage areas 3 installed on the same side of the indoor road have a large overlapping area.

[0117] In the present application, the number of laser radars is at least 2, and the number of laser radars can be adjusted according to the perception and positioning requirements of the heterogeneous sensor array 2 network for indoor scenes, mainly considering the positioning target modeling requirements.

[0118] The multi-camera array module includes multiple cameras, which are arranged on the ceiling of the indoor scene road and / or parking area and face the ground of the indoor scene. The multi-camera array module is used to collect dynamic and static RGB image information of the indoor scene, covering the indoor scene application area.

[0119] Among them, the angle between the camera lens direction and the ground can be adjusted according to the actual height of the indoor ceiling.

[0120] The camera's intrinsic parameters and distortion coefficients can be determined using Zhang Zhengyou's calibration algorithm.

[0121] In a specific embodiment, a high-precision, low-cost RER-USBGS1200P01 camera from RER Vision Technology can be used. Figure 3 As shown, the shooting coverage area of ​​a single camera is 3 trapezoids.

[0122] In the present application, the number of cameras is 18, and the number of cameras can be adjusted according to the size of the indoor scene. If the area of ​​the indoor scene is larger, a larger number of cameras are selected. If the area of ​​the indoor scene is smaller, a smaller number of cameras are selected.

[0123] The communication network module includes a circuit power supply part and an information transmission network part, which are used for the circuit power supply of the heterogeneous sensor array 2 and the transmission of the node data of the heterogeneous sensor array 2 to the main control computer. The information transmission network part includes a wired network composed of switches and six types of network cables, and a wireless network composed of wireless base stations or WiFi.

[0124] The main control computer is connected to the laser radar module and the camera array module respectively through the communication network module, which is used to perform data analysis and data processing on the laser radar point cloud information and RGB image information transmitted back by the camera, including but not limited to laser radar-camera online automatic calibration, multi-camera online automatic calibration, and heterogeneous sensor array 2 global online automatic calibration.

[0125] Through the above technical solution, laser radars are set on both sides of the road near the entrance and exit of the indoor scene and facing the ground of the indoor scene to achieve target modeling and three-dimensional scale information integration, and a multi-camera array module with a field of view covering the application area of ​​the indoor scene is used to collect dynamic and static RGB image information of the indoor scene to achieve global perception of the indoor scene. In indoor scenes without GNSS, the light environment is relatively stable, and the camera can maintain good stability. The number of laser radars installed can be reduced, thereby reducing the deployment cost of the heterogeneous sensor network. The heterogeneous sensor array 2 network for indoor scenes of this application uses a small amount of laser radar to fuse point cloud three-dimensional information and RGB camera image environment texture information. It also has the advantages of obtaining a data basis with higher positioning accuracy, lower cost, simple operation, and easy installation.

[0126] In the above embodiment, a mounting bracket 1 and a power supply module are also included. The mounting bracket 1 is used to install the laser radar module, the multi-camera array module, and the power supply module on the ceiling of the indoor scene; the power supply module is used to provide power to the laser radar module, the multi-camera array module, and the main control computer.

[0127] Among them, the mounting bracket 1 is set on the ceiling of the indoor scene, the mounting bracket 1 is a detachable mechanism, and the deployment position of the mounting bracket 1 can be adjusted according to the area of ​​the application area of ​​the indoor scene.

[0128] An online calibration system for a heterogeneous sensor array 2 for indoor scenes provided in this embodiment realizes online automatic calibration of external parameters through an online automatic calibration method for a heterogeneous sensor array 2 for indoor scenes.

[0129] That is to say, the steps in the method provided by the present invention can realize online automatic calibration in a normally operating online automatic calibration system of a heterogeneous sensor array 2. The embodiments in the method can be referred to as preferred examples for realizing the method, which will not be described in detail here.

[0130] The online automatic calibration method and system of the heterogeneous sensor array 2 for indoor scenes provided in the present application fully utilize the camera RGB image information of the indoor scene and the strong robustness of the point cloud information collected by the lidar. It is suitable for indoor environments with stable ambient light, has a high degree of automation and strong robustness, and can reduce operating costs and difficulty while ensuring positioning accuracy and maintain good stability.

[0131] The above describes the specific embodiments of the present application. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the substantive content of the present application. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. An online automatic calibration method for heterogeneous sensor arrays for indoor scenes, characterized in that: The method comprises: Obtain the point cloud data of the heterogeneous sensor array lidar and the RGB image of the multi-camera array, and perform preprocessing, and determine the external parameter information of the lidar-camera according to the target prior cross-modal matching algorithm, pose solution algorithm and alignment function; Determine the global pose information of the multi-camera array based on the pixel feature region calibration algorithm and scale prior global pose optimization algorithm based on image semantic segmentation; According to the point cloud data information and RGB image information of continuous frames, the continuous external parameter information of the lidar-camera and the continuous pose information of all cameras are obtained, and the online automatic calibration of the external parameters of the heterogeneous sensor array is determined.

2. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 1 is characterized in that: The obtaining of the heterogeneous sensor array lidar point cloud data and the RGB image of the multi-camera array and preprocessing includes: Real-time acquisition of heterogeneous sensor array LiDAR point cloud data and multi-camera array RGB image information; Performing distortion correction processing on the RGB image information according to the calibrated internal parameters of the camera to determine the RGB image after the distortion correction processing; The point cloud data is first preprocessed according to a point cloud preprocessing algorithm to determine the laser radar point cloud data after the first preprocessing.

3. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 2 is characterized in that: The cross-modal matching algorithm, posture solving algorithm and alignment function based on target prior; Determine the external parameter information of the laser radar-camera including: According to the target prior detection algorithm, the RGB image after distortion correction is screened to determine whether the current LiDAR-Camera field of view scene is conducive to LiDAR-Camera extrinsic calibration; According to the cross-modal target matching algorithm, cross-modal target matching is performed on favorable scenes to determine the corresponding mapping between the camera's RGB image target and the target center and vertex of the lidar point cloud data; According to the pose initialization algorithm and alignment function, the lidar point cloud data is mapped to the RGB image pixels one by one, and then refined and optimized to determine the external parameter information of the lidar-camera.

4. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 3 is characterized in that: The cross-modal target matching algorithm is used to perform cross-modal target matching on the favorable scene, and the corresponding mapping between the RGB image target of the camera and the target center and vertex of the laser radar point cloud data is determined, which specifically includes: Build a deep neural network for cross-modal matching; Training deep neural networks; A trained deep neural network is used to establish a one-to-one mapping between the target in the lidar point cloud data and the target in the RGB image based on the RGB image after distortion correction and the lidar point cloud data after the first preprocessing. The corresponding data pairs of 2d-3d target centers are collected to determine the data pairs of the lidar target center to the RGB image target center.

5. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 4 is characterized in that: According to the pose initialization algorithm and alignment function, the laser radar point cloud is mapped to the camera RGB image pixel points one by one, and refined and optimized to determine the external parameter information of the laser radar-camera, specifically including: According to the particle fitness alignment function, the data pair from the lidar target center to the RGB image target center is gradually refined, and based on the refined data pair from the lidar target center to the RGB image target center, the current initialization pose of the lidar-camera is determined by solving PnP; According to the current initialization pose of the LiDAR-camera, the particle fitness alignment function is used again to directly optimize the alignment of the RGB image target pixels and the LiDAR target point cloud data to determine the external parameter information of the LiDAR-camera.

6. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 1 is characterized in that: The pixel feature region calibration algorithm based on image semantic segmentation and the scale prior global pose optimization algorithm determine the global pose information of the multi-camera array specifically including: According to the image semantic segmentation algorithm, the feature area of ​​the current two-dimensional image is segmented to determine the pixel features of different areas; Determine the current scale-free global pose information of the multi-camera array based on the pixel region feature calibration algorithm; According to the scale prior global pose optimization algorithm, the prior LiDAR point cloud scale information is integrated to perform alignment optimization and determine the current global pose information of the multi-camera array.

7. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 6 is characterized in that: The scale prior global pose optimization algorithm includes: a laser radar point cloud scale fusion algorithm, which determines the RGB image scale prior information based on the laser radar-camera external parameter information and the prior laser radar point cloud scale information.

8. The method for online automatic calibration of heterogeneous sensor arrays for indoor scenes according to claim 1 is characterized in that: The method of obtaining continuous external parameter information of the laser radar camera and continuous pose information of all cameras based on continuous point cloud data information and RGB image information, and determining the online automatic calibration of external parameters of the heterogeneous sensor array, specifically includes: The lidar-camera extrinsic parameter information updated in the previous frame is associated with the current lidar-camera extrinsic parameter information to determine the lidar-camera continuous extrinsic parameter information; Associating the global pose information of the multi-camera array updated in the previous frame with the global pose information of the current multi-camera array to determine the global continuous external parameter information of the multi-camera array; The lidar-camera continuous extrinsic parameter information and the multi-camera array global continuous extrinsic parameter information are unified in a unified spatiotemporal coordinate system to determine the global continuous pose information of the heterogeneous sensor array.

9. An online automatic calibration system for heterogeneous sensor arrays for indoor scenes, characterized in that: include: A laser radar module, wherein the laser radar module includes at least two laser radars, and the at least two laser radars are arranged on both sides of the road near the entrance and exit of the indoor scene and face the ground of the indoor scene, and the laser radar module is used to collect dynamic and static point cloud information of the indoor scene; A multi-camera array module, the multi-camera array module comprising a plurality of cameras, the plurality of cameras being arranged at a ceiling position of the indoor scene road and / or parking space area and facing the ground of the indoor scene, the multi-camera array module being used to collect dynamic and static RGB image information of the indoor scene; A communication network module, the communication network module includes a circuit power supply part and an information transmission network part, and is used for powering the circuits of the heterogeneous sensor array; A main control computer, wherein the main control computer is connected to the laser radar module and the camera array module respectively through the communication network module, and is used for performing data analysis and data processing on the laser radar point cloud information and RGB image information transmitted back by the camera, including but not limited to the laser radar-camera online automatic calibration, the multi-camera online automatic calibration, and the heterogeneous sensor array global online automatic calibration.

10. The online automatic calibration system for heterogeneous sensor arrays for indoor scenes according to claim 9, characterized in that: It also includes a mounting bracket and a power supply module; The mounting bracket is used to mount the laser radar module, the multi-camera array module, and the power supply module on the ceiling of the indoor scene; The power supply module is used to provide electrical energy to the laser radar module, the multi-camera array module, and the main control computer.

Images