Internal and external parameter calibration method, device and electronic equipment

Abstract

The embodiment of the present application provides a method, device and electronic device for internal and external parameter calibration, which relates to the technical field of sensor calibration. The method includes: obtaining environmental point cloud and environmental image corresponding to the same environment; according to the initial external parameters, performing angular mapping, edge feature extraction and inverse mapping on the environmental point cloud to obtain the target edge point cloud in the environmental point cloud, and the mapped space is the target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as the orthogonal axes; according to the initial internal parameters, performing angular mapping, edge feature extraction and inverse mapping on the environmental image to obtain the target edge pixel points in the environmental image, and the mapped space of the environmental image is the target space; according to the target edge pixel points and the target edge point cloud, iteratively update the initial internal parameters and the initial external parameters to obtain the target internal parameters and the target external parameters. In this way, the internal and external parameter joint calibration of the camera and the lidar is automatically completed without markers.

Description

Technical Field

[0001] This application relates to the technical field of sensor calibration, and more particularly, to an internal and external parameter calibration method, apparatus, and electronic device. Background Art

[0002] Currently, cameras and lidar are often used in combination. In this method, two types of parameters need to be calibrated. The first is the intrinsic parameters of the camera, which are used to map pixel coordinates to spatial coordinates. The second is the extrinsic parameters between the camera and the lidar, that is, the rigid body transformation parameters between the two sensor coordinate systems. For the combined use of multiple sensors, sensor calibration is the most fundamental and important task.

[0003] Most existing camera intrinsic parameter calibration methods are based on fixed markers. The intrinsic parameter calibration method based on fixed markers is limited by the calibration scene and requires manual operation of the markers, which is very cumbersome. At the same time, this calibration method requires good design of the marker positions to obtain more accurate calibration results, which requires high manual operation requirements and further limits the calibration working scene.

[0004] Moreover, all existing extrinsic parameter calibration methods between cameras and lidar are carried out in two steps, that is, first calibrate the camera intrinsic parameters, and then calibrate the extrinsic parameters based on the intrinsic parameters. In this traditional two-step calibration method for internal and external parameters, the error of the intrinsic parameter calibration result will be further amplified in the extrinsic parameter calibration. Summary of the Invention

[0005] Embodiments of this application provide an internal and external parameter calibration method, apparatus, electronic device, and readable storage medium, which can automatically complete the joint calibration of the internal and external parameters of the camera and lidar without markers, solve the problems existing in the existing camera intrinsic parameter calibration method and the extrinsic parameter calibration method between the camera and the lidar, and avoid the error amplification caused by the two-step calibration process.

[0006] Embodiments of this application can be implemented as follows:

[0007] In a first aspect, embodiments of this application provide an internal and external parameter calibration method, the method including:

[0008] Obtain environmental point cloud and environmental image corresponding to the same environment, wherein the environmental point cloud is obtained by a lidar, and the environmental image is obtained by a camera;

[0009] According to the initial extrinsic parameters, perform angular mapping, edge feature extraction, and inverse mapping on the environmental point cloud to obtain the target edge point cloud in the environmental point cloud, where the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes;

[0010] According to the initial intrinsic parameters, perform angular mapping, edge feature extraction, and inverse mapping on the environmental image to obtain the target edge pixel points in the environmental image, where the space to which the environmental image is mapped is the target space;

[0011] According to the target edge pixel points and the target edge point cloud, iteratively update the initial intrinsic parameters and the initial extrinsic parameters to obtain the target intrinsic parameters of the camera and the target extrinsic parameters between the lidar and the camera.

[0012] In a second aspect, an embodiment of the present application provides an intrinsic and extrinsic parameter calibration device, and the device includes:

[0013] An acquisition module, configured to acquire an environmental point cloud and an environmental image corresponding to the same environment, where the environmental point cloud is acquired by a lidar, and the environmental image is acquired by a camera;

[0014] A first edge determination module, configured to perform angular mapping, edge feature extraction, and inverse mapping on the environmental point cloud according to the initial extrinsic parameters to obtain the target edge point cloud in the environmental point cloud, where the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes;

[0015] A second edge determination module, configured to perform angular mapping, edge feature extraction, and inverse mapping on the environmental image according to the initial intrinsic parameters to obtain the target edge pixel points in the environmental image, where the space to which the environmental image is mapped is the target space;

[0016] An optimization module, configured to iteratively update the initial intrinsic parameters and the initial extrinsic parameters according to the target edge pixel points and the target edge point cloud to obtain the target intrinsic parameters of the camera and the target extrinsic parameters between the lidar and the camera.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the intrinsic and extrinsic parameter calibration method described in the foregoing embodiments.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the intrinsic and extrinsic parameter calibration method described in the foregoing embodiments.

[0019] The internal and external parameter calibration method, device, electronic device and readable storage medium provided by the embodiments of the present application. First, an environmental image obtained by a camera is acquired, and an environmental point cloud obtained by collecting information on the same environment using a lidar is acquired. Then, based on the initial external parameters, angular mapping, edge feature extraction, and inverse mapping are performed on the environmental point cloud to obtain the target edge point cloud in the environmental point cloud, where the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes. And according to the initial internal parameters, angular mapping, edge feature extraction, and inverse mapping are performed on the environmental image to obtain the target edge pixel points in the environmental image, where the space to which the environmental image is mapped is the target space. Finally, based on the target edge pixel points and the target edge point cloud, iterative updating is performed on the initial internal parameters and the initial external parameters to obtain the target internal parameters of the camera and the target external parameters between the lidar and the camera. In this way, the internal and external parameter joint calibration of the camera and the lidar can be automatically completed in an environment without markers, solving the problems existing in the existing camera internal parameter calibration method and the external parameter calibration method between the camera and the lidar, and avoiding the error magnification caused by the two-step calibration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a block diagram of the electronic device provided by the embodiments of the present application;

[0022] Figure 2 It is a flowchart of the internal and external parameter calibration method provided by the embodiments of the present application;

[0023] Figure 3 It is a schematic diagram of the lidar and the camera provided by the embodiments of the present application;

[0024] Figure 4 For Figure 2 One of the flowcharts of the sub-steps included in step S110 in

[0025] Figure 5 For Figure 2 Another flowchart of the sub-steps included in step S110 in

[0026] Figure 6 For Figure 2 The flowchart of the sub-steps included in step S120 in

[0027] Figure 7 For Figure 6 Schematic diagram of the sub - steps included in sub - step S123;

[0028] Figure 8 For Figure 2 Schematic diagram of the sub - steps included in step S130;

[0029] Figure 9 For Figure 8 Schematic diagram of the sub - steps included in sub - step S132;

[0030] Figure 10 For Figure 2 Schematic diagram of the sub - steps included in step S140;

[0031] Figure 11 Schematic diagram of an application of the internal and external parameter calibration method provided by the embodiment of the present application;

[0032] Figure 12 Block diagram of the internal and external parameter calibration device provided by the embodiment of the present application.

[0033] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - internal and external parameter calibration device; 210 - acquisition module; 220 - first edge determination module; 230 - second edge determination module; 240 - optimization module. Detailed implementation manners

[0034] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0036] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0037] Taking the combined use of a fish-eye camera and a lidar as an example, the following introduces how to calibrate the internal and external parameters currently.

[0038] A fish-eye camera is an ultra-wide-angle lens, which generally uses multiple groups of convex lenses, or a combination of convex lenses and plane mirrors to change the incident light path, so that the incident light in a large field of view can be focused on a small imaging plane. Although the fish-eye camera has a large field of view, it has not been widely used in the field of robotics. One of the reasons is that the fish-eye camera has serious distortion and is difficult to be accurately calibrated.

[0039] When combining a fish-eye camera with a lidar, two parameters need to be calibrated. The first is the internal parameters of the fish-eye camera, which are used to map pixel coordinates to spatial coordinates. The second is the external parameters between the camera and the lidar, that is, the rigid body transformation parameters between the two sensor coordinate systems. For the use of sensors, or the fusion use of multiple sensors, sensor calibration is the most important and fundamental work.

[0040] Currently, when a fish-eye camera and a lidar are used in combination, the general steps are to first calibrate the internal parameters of the camera, and then, based on the calibrated internal parameters of the camera, calibrate the external parameters between the camera and the lidar. The following explains the calibration methods of the two parameters.

[0041] The earliest internal parameter models of fish-eye cameras were all constructed by different types of trigonometric functions, such as the orthographic projection model, the stereographic projection model, the equal-area projection model, etc. Later, using Taylor expansion, all these trigonometric function models can be uniformly described by polynomials. So far, the vast majority of fish-eye camera internal parameter models use polynomials to construct.

[0042] All existing methods for calibrating the internal parameters of fish-eye cameras are based on fixed markers. For example, the widely used OcamCalib MATLAB Toolbox requires manually placing a chessboard at different spatial positions and taking at least 10 photos. This method realizes automatic corner extraction. Based on the estimated poses of different chessboards, it calculates the reprojection error of the corners under the internal parameter model and uses this as a cost function for non-linear optimization.

[0043] Another example is to swap the independent and dependent variables of the polynomial fish-eye camera model, which is a more direct way to construct the model. At the same time, a fixed proportional coefficient is added to limit the value range of the coefficients of different orders of the polynomial. This method uses a stripe pattern as a fixed marker, constructs a cost function based on the prior geometric relationship given by the pattern, and obtains the internal parameter estimation of the fish-eye camera by non-linear optimization.

[0044] The current method for calibrating the external parameters of a fish-eye camera and a lidar uses the point cloud of the lidar to generate a bearing angle image and enhances the edge features. Manually select corresponding edge feature points on the bearing angle image of the lidar and the image of the fish-eye camera, and construct a PnP problem to optimize and solve the external parameters of the two sensors.

[0045] For existing internal parameter calibration methods, the internal parameter calibration method based on fixed markers is restricted by the calibration scene and requires manual operation of the markers, which is very cumbersome. At the same time, when the marker is close to the camera, the corner extraction is accurate, but the corner density is small, and the internal parameter polynomial curve calibrated is not accurate in the area with few corner distributions. When the marker is far away, although the corner density can be increased by using multiple chessboards, at this time, the pixel area occupied by the chessboard in the image is small, and the error of corner extraction will increase. Also, because this method needs to estimate the pose of the chessboard relative to the camera, the farther the distance, the less sensitive the estimation of the translation vector is, and errors are also likely to occur. And there is only one existing method for calibrating the external parameters of a fish-eye camera and a lidar, and it completely relies on manual selection of feature pixel points, with a very large error.

[0046] It should also be noted that almost all current camera calibration methods are in two steps. First, the internal parameters of the camera are calibrated, and then the external parameters of the camera are calibrated based on the internal parameters. For the fish-eye camera model, the instability of the high-order polynomial causes even a small error in the optimization of each polynomial coefficient to result in a large error in the polynomial curve. If an unstable internal parameter calibration result is used as a prior and then the external parameters are calibrated, the deviation of the parameters will be further amplified.

[0047] It can be understood that the above analysis is based on the fish-eye camera as an example. When other types of cameras and lidars are used in combination, similar problems still exist in the calibration methods used.

[0048] Based on the above research, the embodiments of the present application provide an internal and external parameter calibration method, device, electronic device and readable storage medium. First, obtain the environmental image obtained by the camera and the environmental point cloud obtained by the lidar, and then obtain the target edge pixel points and the target edge point cloud through mapping, edge extraction and inverse mapping. Furthermore, obtain the target internal parameters and the target external parameters based on the target edge pixel points and the target edge point cloud. In this way, the situation that the calibration process is limited by the calibration environment or fixed markers can be solved, and the dependence on manual work in the calibration process can be eliminated, enabling it to be automatically completed in any environment. Moreover, it solves the situation that the insufficient density of feature points of the fixed marker leads to inaccurate calibration, or the large distance between the fixed marker and the sensor greatly affects the calibration result. It also solves the problem of calibration error caused by manually selecting feature points in the existing external parameter calibration methods for fisheye cameras and lidars. And because it jointly calibrates the internal and external parameters of the fisheye camera and the lidar and completes the estimation of all parameters in the same optimization process, it can solve the problem that the error caused by the unstable internal parameter calibration result in the traditional two-step internal and external parameter calibration method is further amplified in the external parameter calibration.

[0049] It should be noted that all the defects existing in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by the embodiments of the present application below for the above problems should be the contributions made by the inventors to the present application during the process of the present application.

[0050] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0051] Please refer to Figure 1 , Figure 1 which is a block diagram of the electronic device 100 provided by the embodiments of the present application. The electronic device 100 can be, but is not limited to, a computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120 and a communication unit 130. The memory 110, the processor 120 and the communication unit 130 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0052] Among them, the memory 110 is used to store programs or data. The memory 110 can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0053] The processor 120 is used to read / write the data or programs stored in the memory 110 and perform corresponding functions. For example, the internal and external parameter calibration device 200 is stored in the memory 110. The internal and external parameter calibration device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running the software programs and modules stored in the memory 110, such as the internal and external parameter calibration device 200 in the embodiments of the present application, thereby implementing the internal and external parameter calibration method in the embodiments of the present application.

[0054] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network and is used to transmit and receive data through the network.

[0055] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may further include more or fewer components than those shown Figure 1 in the figure (for example, the electronic device 100 may further include a camera and a radar), or have a different configuration from that shown Figure 1 in the figure. Figure 1 Each component shown in the figure can be implemented by hardware, software, or a combination thereof.

[0056] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of the internal and external parameter calibration method provided by the embodiments of the present application. The method can be applied to the electronic device 100. The specific process of the internal and external parameter calibration method will be elaborated in detail below. In this embodiment, the method may include steps S110 to S140.

[0057] Step S110: Obtain the environmental point cloud and environmental image corresponding to the same environment.

[0058] In this embodiment, a camera can be used to capture any environment to obtain an environmental image. The environmental image includes color information in the environment. A lidar can also be used to collect information on the same environment to obtain an environmental point cloud corresponding to the same environment. The environmental point cloud can include the three-dimensional coordinates of each scanned point in the lidar coordinate system and the reflectivity of the point. The range of the reflectivity is 0 to 255, and the reflectivity can be directly used as the gray value of the scanned point on the image. Among them, the camera can be a fish-eye camera or other cameras, which can be specifically determined according to the actual situation.

[0059] The electronic device, the camera, and the lidar can be independent devices. In this case, the camera can send the captured image to the electronic device so that the electronic device can obtain the environmental image; the lidar can send the point cloud collected for the same environment to the electronic device so that the electronic device can obtain the environmental point cloud. The electronic device, the camera, and the lidar can also be integrated devices, and the electronic device can obtain the environmental image and the environmental point cloud by controlling the camera and the lidar. It can be immediately seen that the above methods for obtaining the environmental image and the environmental point cloud are only for illustrative purposes, and the environmental image and the environmental point cloud can also be obtained by other methods. For example, another device can send the point cloud and the image of the same environment to the electronic device.

[0060] Step S120: According to the initial extrinsic parameters, perform angular mapping, edge feature extraction, and inverse mapping on the environmental point cloud to obtain the target edge point cloud in the environmental point cloud.

[0061] In this embodiment, the initial extrinsic parameters between the lidar and the camera can be preset, that is, the initial values of the extrinsic parameters are preset. In the case of obtaining the environmental point cloud, according to the initial extrinsic parameters, angular mapping is performed on the environmental point cloud to map the environmental point cloud into the target space A. Among them, the target space A is a space with the pitch angle and azimuth angle in the camera coordinate system (three-dimensional coordinate system) of the camera as the orthogonal axes. After completing the angular mapping, edge feature extraction can be performed to extract edge features from the image corresponding to the environmental point cloud in the target space A, and then the edge features are inversely mapped back to the lidar coordinate system to obtain the target edge point cloud in the environmental point cloud. The points in the target edge point cloud are the edge points determined from the environmental point cloud.

[0062] Step S130: According to the initial intrinsic parameters, perform angular mapping, edge feature extraction, and inverse mapping on the environmental image to obtain the target edge pixel points in the environmental image.

[0063] In this embodiment, the initial internal parameters of the camera can be preset, that is, the initial values of the internal parameters are preset. When the environmental image is obtained, according to the initial internal parameters, the environmental image is subjected to angular mapping to map the environmental image into the target space A. That is, the space to which the environmental image and the environmental point cloud are mapped is the same, and both are mapped into the target space A. Then, similar to the processing process of the environmental point cloud, edge extraction is performed to extract edge features from the image corresponding to the environmental image in the target space A, and then the edge features are inversely mapped back to the camera image coordinate system (two-dimensional coordinate system, that is, the coordinate system where the environmental image is located) of the camera to obtain the target edge pixel points in the environmental image. The target edge pixel points are the edge pixel points determined in the environmental image.

[0064] Step S140, according to the target edge pixel points and the target edge point cloud, iteratively update the initial internal parameters and the initial external parameters to obtain the target internal parameters of the camera and the target external parameters between the lidar and the camera.

[0065] The initial internal parameters and the initial external parameters can be optimized and updated according to the target edge pixel points and the target edge point cloud until the result after processing the updated internal and external parameters based on the target edge point cloud and the target edge pixel points meet the preset conditions, then it can be determined that the optimization is completed. When the optimization is completed, the internal parameters at this time can be used as the target internal parameters of the camera, and the external parameters at this time can be used as the target external parameters between the lidar and the camera. The target internal parameters are the calibrated internal parameters of the camera, and the target external parameters are the calibrated external parameters of the lidar relative to the camera.

[0066] The embodiment of the present application proposes a method for jointly calibrating internal and external parameters for the case of the combined use of a camera and a lidar. In an environment without markers, the internal and external parameters of the camera and the lidar are jointly calibrated automatically in the same optimization process, solving the problems existing in the internal parameter calibration method of the fisheye camera and the external parameter calibration method between the camera and the lidar, and avoiding the error magnification caused by the two-step calibration process.

[0067] Optionally, the lidar can be a repetitive scanning radar or a non-repetitive scanning radar, and can be specifically determined in combination with actual requirements. As a possible implementation, such as Figure 3As shown, the lidar is a Livox Mid-360 lidar, which is a 4-line lidar and the first lidar with both non-repetitive scanning characteristics and a 360° horizontal field of view. Due to its non-repetitive scanning characteristics, as time accumulates, the field of view coverage rate will approach 100%. The camera is a fisheye camera, which has a 360° horizontal field of view and a 70° vertical field of view. The volume of the entire sensor combination (including the Livox Mid-360 lidar and the fisheye camera) is very small. The volumes of the lidar and the fisheye camera are only 6.5x6.5x6.5 cm and 5x5x10 cm respectively, so it can be integrated on any mobile platform, such as a mobile chassis, a robotic dog, or a drone, etc.

[0068] Since the field of view of the camera in the vertical direction is different from that of the lidar in the vertical direction, when the field of view of the camera in the vertical direction is greater than that of the lidar in the vertical direction, if the internal and external parameters are directly calibrated based on the point cloud collected by the lidar and the image collected by the camera, the calibration result may be poor due to the lack of point cloud information of the lidar. To avoid this situation, the environmental point cloud can be obtained by Figure 4 the method shown.

[0069] Please refer to Figure 4 , Figure 4 which is Figure 2 one of the schematic flowcharts of the sub-steps included in step S110 in. In this embodiment, the process of obtaining the environmental point cloud in step S110 may include sub-steps S111 to sub-step S112.

[0070] Sub-step S111, obtaining the initial point cloud at different angles by the lidar.

[0071] Sub-step S112, stitching the initial point clouds at different angles to obtain the environmental point cloud.

[0072] In this embodiment, when the field of view of the camera in the vertical direction is greater than the field of view of the lidar in the vertical direction, the attitude of the lidar can be changed so that the lidar scans in different attitudes, thereby obtaining initial point clouds corresponding to different angles. In this way, point clouds can be accumulated from multiple perspectives. Among them, the specific way of attitude transformation can be determined according to actual needs, as long as the field of view corresponding to the subsequent obtained environmental point cloud is not less than the field of view corresponding to the environmental image, that is, the field of view corresponding to the environmental point cloud needs to include the field of view corresponding to the environmental image. Then, the initial point clouds at different angles can be stitched together to obtain the environmental point cloud. Optionally, the ICP (Iterative Closest Point) algorithm can be used for stitching. For example, an initial point cloud can be obtained using a non-repetitive scanning lidar, and through time accumulation, a sub-pixel level (characteristic of non-repetitive scanning lidar) point cloud close to covering the entire field of view of the lidar can be obtained.

[0073] As a possible implementation, in order to enhance the robustness in different lighting environments, it can be obtained by Figure 5 the method shown. Please refer to Figure 5 , Figure 5 is Figure 2 a second schematic flow diagram of the sub-steps included in step S110 in. In this embodiment, the process of obtaining the environmental image in step S110 may include sub-steps S115 to sub-steps S116.

[0074] Sub-step S115, obtaining initial images obtained by the camera in the environment with different exposure durations.

[0075] Sub-step S116, performing exposure fusion on the obtained multiple initial images to obtain the environmental image.

[0076] In this embodiment, the camera can be controlled to perform image acquisition in the same environment with different exposure durations, so as to obtain multiple initial images corresponding to different exposure durations. Then, the multiple initial images can be subjected to exposure fusion to obtain a high dynamic range image (HDR) as the environmental image. In this way, the robustness in different lighting environments can be enhanced.

[0077] In the case of obtaining the environmental point cloud, it can be obtained by Figure 6 the method shown. Please refer to Figure 6 , Figure 6 is Figure 2 a schematic flow diagram of the sub-steps included in step S120 in. In this embodiment, step S120 may include sub-steps S121 to sub-steps S125.

[0078] Sub-step S121: According to the initial extrinsic parameters, transform the environmental point cloud into the camera coordinate system to obtain the transformed environmental point cloud.

[0079] Sub-step S122: For each point in the transformed environmental point cloud, calculate the corresponding first pitch angle and first azimuth angle according to the coordinates of the point in the camera coordinate system.

[0080] In this embodiment, the first pitch angle and first azimuth angle corresponding to each point in the environmental point cloud transformed into the camera coordinate system can be calculated according to the following formulas (1) to (3). Extrinsic parameters According to the initial extrinsic parameters, the environmental point cloud can be transformed from the lidar coordinate system to the camera coordinate system through the following formula (1) to obtain the transformed environmental point cloud. Among them, formula (1) is:

[0081]

[0082] Among them, C P represents a three-dimensional point in the camera coordinate system {C}, L P represents a three-dimensional point in the lidar coordinate system; given the extrinsic parameters Δ, represents the extrinsic transformation of the three-dimensional point in the lidar coordinate system from the lidar coordinate system {L} to the camera coordinate system {C} (through rotation and translation ).

[0083] After the transformation is completed, according to the three-dimensional coordinates of each point in the transformed environmental point cloud, the pitch angle θ can be calculated based on formula (2), and the azimuth angle φ can be calculated based on formula (3). Among them, formula (2) is: Formula (3)

[0084] Sub-step S123: Generate a first projection image in the target space according to the first pitch angle, first azimuth angle and reflectivity corresponding to each point.

[0085] A first projection image can be generated in the target space A according to the first pitch angle, first azimuth angle and reflectivity corresponding to each point in the environmental point cloud in the camera coordinate system, with the reflectivity as the gray value. Optionally, the determination method of the gray value of each pixel grid in the first projection image can be determined according to actual needs.

[0086] Optionally, the size of the target space A can be set in advance. For example, Among them, the range of the azimuth angle is [0, 2π], and the range of the elevation angle is [0, π]. In the target space A, the horizontal axis is the azimuth angle, the vertical axis is the elevation angle, and 8000 is the set size of the target space. In this way, the corresponding elevation angle and azimuth angle ranges of each pixel grid in the target space A can be determined.

[0087] After determining each pixel grid in the target space A and the first elevation angle, the first azimuth angle, and the reflectivity corresponding to each point in the environmental point cloud in the camera coordinate system, the gray value of each pixel point in the target space A can be determined in a corresponding manner according to actual needs, so as to obtain the first projection image.

[0088] As a possible implementation, it can be obtained through Figure 7 the method shown to obtain the first projection image. Please refer to Figure 7 , Figure 7 is Figure 6 a schematic flowchart of the sub-steps included in the sub-step S123. In this embodiment, step S123 may include sub-steps S1231 to S1232.

[0089] Sub-step S1231: According to the first elevation angle and the first azimuth angle corresponding to each point, for each pixel grid in the target space, search is performed with the center point of the pixel grid as the center and the first preset distance as the radius.

[0090] Sub-step S1232: Determine the gray value of the pixel grid according to the reflectivity of the searched point to obtain the first projection image.

[0091] In this embodiment, according to the first elevation angle and the first azimuth angle corresponding to each point in the environmental point cloud in the camera coordinate system, for each pixel grid in the target space A, with the center point of the pixel grid as the search center, search is performed in the point cloud mapped into the target space A with the first preset distance as the radius; and then, according to the reflectivity of all points searched based on the pixel grid, the gray value of the pixel grid is determined. For example, the average value of the reflectivities of all points searched based on the pixel grid is used as the gray value of the pixel grid. Among them, the first preset distance can be specifically determined in combination with actual needs. In this way, when there are multiple projection points in a pixel grid, after performing the above processing on each pixel grid, the gray value of each pixel can be determined, so as to obtain the first projection image.

[0092] Sub-step S124: Perform edge extraction on the first projection image to obtain the first target edge feature.

[0093] Optionally, an edge feature extraction algorithm, such as the Canny algorithm, can be used to perform edge feature extraction on the first projection image to obtain the first target edge feature.

[0094] As a possible implementation, an edge feature extraction algorithm can be used to extract the first edge feature from the first projection image, and then filter according to the edge pixel length, and retain the first edge feature with a longer length as the first target edge feature. Optionally, the pixel length of each first edge feature can be compared with a first preset pixel length, and the first edge feature with a pixel length greater than the first preset pixel length is used as the first target edge feature.

[0095] Sub-step S125: According to the mapping method used when mapping to the target space, perform inverse mapping on the first target edge feature to obtain the target edge point cloud.

[0096] In the case of obtaining the first target edge feature, according to the mapping relationship between the pixel grid and the corresponding point cloud when generating the first projection image previously, the first target edge feature in the target space A can be inversely mapped back to the corresponding original space (i.e., the radar coordinate system), so as to obtain the target edge point cloud. In this way, the target edge point cloud can be automatically determined from the environmental point cloud.

[0097] Similarly, the environmental image is processed in a similar manner to obtain target edge pixel points.

[0098] Please refer to Figure 8 , Figure 8 For Figure 2 the flowchart of the sub-steps included in step S130 in. In this embodiment, step S130 may include sub-steps S131 to S134.

[0099] Sub-step S131: According to the initial internal parameters and the environmental image, calculate the second pitch angle and the second azimuth angle corresponding to each pixel point in the environmental image.

[0100] Taking the camera as a fisheye camera as an example below, introduce how to obtain the second pitch angle and the second azimuth angle. In this embodiment, the second pitch angle and the second azimuth angle corresponding to each pixel point in the environmental image can be calculated according to the following formulas (4) to (7). Internal parameters The environmental image can be corrected first through formula (4). Among them, formula (4) is:

[0101]

[0102] Among them, [u', v'] represents the pixel coordinates of the two-dimensional points of the processed environmental image, represents the distortion correction matrix, [u, v] represents the pixel coordinates of the two-dimensional points of the environmental image, and [u0, v0] represents the pixel coordinates of the center point of the environmental image.

[0103] Then, the second pitch angle and the second azimuth angle corresponding to each pixel point in the environmental image can be calculated based on formulas (5) to (7):

[0104]

[0105]

[0106] θ = F -1 (r; a0,..., a n ) (7)

[0107] Among them, formula (7) is the inverse process of formula (8) r = F(θ; a0,..., a n ) = a0 + a1θ +... + a n θ n and is obtained by spline curve fitting. F(θ; a0,..., a n ) is a polynomial of the internal parameter model of the fisheye camera, which calculates the pitch angle as the pixel radius r centered on [u0, v0].

[0108] In this way, starting from the camera pixel coordinates, the spatial pitch angle and azimuth angle can be calculated, so as to project them into the target space.

[0109] Sub-step S132: Generate a second projection image in the target space according to the second pitch angle, the second azimuth angle and the gray value corresponding to each pixel point in the environmental image.

[0110] The gray value corresponding to each pixel grid in the target space A can be determined according to the second pitch angle, the second azimuth angle and the gray value corresponding to each pixel point in the environmental image, so as to generate a second projection image in the target space A. Optionally, the determination method of the gray value of each pixel grid in the second projection image can be determined in combination with actual requirements.

[0111] As a possible implementation manner, the second projection image can be obtained through the Figure 9 shown manner. Please refer to Figure 9 , Figure 9 is Figure 8 the schematic flow diagram of the sub-steps included in sub-step S132 in

[0112] Sub-step S1321: According to the second pitch angle and the second azimuth angle corresponding to each pixel point in the environmental image, search for each pixel grid in the target space with the center point of the pixel grid as the center and the second preset distance as the radius.

[0113] Sub-step S1322: Determine the gray value of the pixel grid according to the gray value of the searched point to obtain the second projection image.

[0114] In this embodiment, according to the second pitch angle and the second azimuth angle corresponding to each point in the environmental image, for each pixel grid in the target space A, with the center point of the pixel grid as the search center, search within the pixel points mapped into the target space A with a second preset distance as the radius; then determine the gray value of the pixel grid according to the gray values of all pixel points searched based on this pixel grid. For example, take the average value of the gray values of all pixel points searched based on this pixel grid as the gray value of this pixel grid. Among them, the second preset distance can be specifically determined in combination with actual requirements. In this way, when the resolution of the environmental image is smaller than the size of the target space A, the gray values of each pixel can still be determined, so as to obtain the second projection image.

[0115] Sub-step S133: Perform edge extraction on the second projection image to obtain the second target edge feature.

[0116] Similar to obtaining the first target edge feature from the first projection image, the second edge feature can be extracted from the second projection image using an edge feature extraction algorithm, and then filtered according to the edge pixel length, and the second edge feature with a longer length is retained as the second target edge feature. Optionally, the pixel length of each second edge feature can be compared with a second preset pixel length, and the second edge feature with a pixel length greater than the second preset pixel length is used as the second target edge feature.

[0117] Sub-step S134: According to the mapping method used when mapping to the target space, perform inverse mapping on the second target edge feature to obtain the target edge pixel points.

[0118] In the case of obtaining the second target edge feature, according to the mapping relationship between the pixel grid and the corresponding pixel points in the corresponding environmental image when the second projection image was previously generated, the second target edge feature in the target space A can be inversely mapped back to the corresponding original space (i.e., the camera image coordinate system), so as to obtain the target edge pixel points. In this way, the target edge pixel points can be automatically determined from the environmental image.

[0119] In the case of obtaining the target edge point cloud and the target edge pixel points, the target edge point cloud can be projected onto the camera image plane corresponding to the camera, and the projected pixel points can be used as the projection result. Taking the distribution of the projection result to be consistent with the distribution of the target edge pixel points as the goal, the initial internal parameters and the initial external parameters can be iteratively updated to obtain the target internal parameters and the target external parameters.

[0120] Optionally, the ICP can be used to calculate the closest point distance for the projection result and the target edge pixel points. At this time, the closest point distance is the function value of the cost function. The initial internal parameters and initial external parameters can be optimized with the maximum of the cost function corresponding to the closest point distance as the goal. Other methods can also be used to determine the cost function and then optimize it.

[0121] As a possible implementation, it can be completed by using Figure 10 the method shown in the figure to obtain the target internal parameters and target external parameters. Please refer to Figure 10 , Figure 10 which is Figure 2 a schematic flowchart of the sub-steps included in step S140 in [reference]. In this embodiment, step S140 may include sub-steps S141 to S144.

[0122] Sub-step S141: Calculate the target kernel density estimation function according to the target edge pixel points.

[0123] In this embodiment, in order to make the cost function gradient continuous and easy to optimize, the kernel density estimation (KDE) is used to estimate the edge distribution of the target edge pixel points based on the preset window width with the target edge pixel points as samples, so as to obtain the target kernel density estimation function.

[0124] Sub-step S142: Project the target edge point cloud onto the camera image plane according to the current internal parameters and external parameters, and determine the positions of the projection points.

[0125] In the case of obtaining the target edge point cloud, the target edge point cloud can be projected onto the camera plane to obtain the two-dimensional coordinates of the projection points. Taking a fisheye camera as an example, the projection process is as follows:

[0126] Internal parameters External parameters

[0127]

[0128]

[0129] Among them,

[0130] r = F(θ; a0,..., a n ) = a0 + a1θ +... + a n θ n ,

[0131] Among them, L P represents a three-dimensional point in the lidar coordinate system {L}, Represents the extrinsic parameter transformation from the {L} coordinate system to the fisheye camera coordinate system {C} (a three-dimensional coordinate system) (through rotation and translation ); θ represents the pitch angle calculated in the {L} coordinate system, and φ represents the azimuth angle calculated in the {L} coordinate system; F(θ; a0,..., a n ) represents the polynomial of the fisheye camera intrinsic parameter model, which calculates the pitch angle as the pixel radius r centered on [u0, v0]; C p represents the pixel point coordinates (two-dimensional coordinates in the camera image coordinate system), and Π( C P; Θ) represents the intrinsic parameter transformation that transforms a three-dimensional space point from the coordinate system {C} to the two-dimensional fisheye camera image coordinate system, representing the distortion correction matrix.

[0132] Sub-step S143: Calculate the function values corresponding to each projection point according to the positions of the projection points and the target kernel density estimation function.

[0133] For each projection point, the position coordinates of the projection point can be substituted into the target kernel density estimation function to calculate the function value corresponding to the projection point, that is, calculate the value of the projection point on the KDE distribution probability density function.

[0134] Sub-step S144: With the goal of maximizing the function value, adjust the currently used intrinsic and extrinsic parameters, and jump to the function value calculation step according to the adjusted intrinsic and extrinsic parameters, so as to obtain the target intrinsic and extrinsic parameters when the iteration is completed.

[0135] The sum of the squares of the function values of all projection points can be used as the cost function, or the result of dividing the square of the function value of all projection points by the number of projection points can be used as the cost function, etc. Through a non-linear optimization method (such as the L-M method), optimize with the goal of maximizing the cost function to obtain the optimized values of the internal and external parameters.

[0136] In this optimization process, the function value corresponding to the radar projection point can be calculated according to the current intrinsic and extrinsic parameters and the current target kernel density estimation function. Then, adjust the current intrinsic and extrinsic parameters according to the function value, and use the adjusted intrinsic and extrinsic parameters as the updated current intrinsic and extrinsic parameters, and then jump to sub-step S142 to recalculate the function value corresponding to the radar projection point. When the above process is repeated a preset number of times or the calculated function value meets the requirements, it can be determined that the iteration is completed and the optimal internal and external parameters are obtained.

[0137] Optionally, in order to make the gradient of the cost function as effective as possible for optimization, multiple rounds of optimization can be performed to gradually approximate the parameter values to the correct values. In order to be able to effectively optimize in different regions, the window width of the KDE can be readjusted after each round of optimization. That is, during the parameter optimization process, the window width corresponding to the target kernel density estimation function can also be adjusted, and the updated target kernel density estimation function can be calculated based on the adjusted window width. Among them, the window width before adjustment is greater than the window width after adjustment, and the target kernel density estimation function corresponding to one window width is used to calculate the function values corresponding to multiple sets of internal and external parameters. After obtaining the updated target kernel density estimation function, use this function to calculate the function values again and optimize the internal and external parameters.

[0138] In KDE, the window width determines the range within which the kernel function is affected by the sampling points, that is, it affects the smoothness of the estimated probability density function. At the beginning of the optimization, the window width can be set relatively large. The smooth function can make the optimization quickly approach the optimal solution region. When entering the optimal solution region, gradually reduce the window width to increase the gradient in the local region and let the optimization continue to approach the optimal solution.

[0139] Among them, during one round of optimization, it can be determined that this round of optimization is completed when the preset number of iterations is reached, or when the value of the cost function calculated is greater than the preset value.

[0140] In this method, first, the current target kernel density estimation function can be calculated under one window width. Then, through sub-step S142 to sub-step S143, the function value of the radar projection point obtained based on the current internal and external parameters under the current window width is calculated; the current internal and external parameters are adjusted based on the function value, and then jump to sub-step S142. After repeating a certain number of times, the optimal internal and external parameters corresponding to the current window are obtained. Then, the window width can be adjusted, and the target kernel density estimation function is re-estimated according to the new window width. After that, the internal and external parameters are adjusted again based on the new target kernel density estimation function. When the adjustment of the window width and the internal and external parameters stops, the target internal parameters and target external parameters are obtained.

[0141] The following calibration Figure 3 Taking the internal and external parameters of the fish-eye camera and lidar shown as an example, combined with Figure 11 the above internal and external parameter calibration method is described. Among them, the vertical field of view of the fish-eye camera is greater than the vertical field of view of the lidar, and the lidar is a non-repeating scanning lidar.

[0142] First, perform pre - processing. Make the lidar scan from different views, and then stitch the scanned point clouds to obtain the environmental point cloud. Make the fisheye camera capture images with different exposure durations to obtain fisheye images with different exposure times. Perform exposure fusion on multiple fisheye images to obtain the environmental image. For the environmental point cloud and the environmental image, perform projection according to the azimuth angle and the pitch angle respectively to obtain the first projection image corresponding to the environmental point cloud and the second projection image corresponding to the environmental image.

[0143] Next, perform edge extraction. For the first projection image and the second projection image respectively, use the Canny algorithm to perform edge extraction. Then, the inverse mapping can be performed on the extracted edge features to obtain the original point clouds and the original fisheye pixels corresponding to the edge features.

[0144] Then, perform iterative optimization. For the original fisheye pixels corresponding to the determined edge features, use KDE to perform edge distribution estimation to obtain an edge probability prediction function. For the original point clouds corresponding to the determined edge features, perform point cloud projection, and calculate the edge probability thresholds corresponding to the projected points in combination with the edge probability prediction function. With the goal of maximizing the cost function corresponding to the optimized internal parameters and external parameters, that is Use the L - M method for optimization. After one parameter adjustment, the parameters can be updated, and the current cost value can be recalculated based on the updated parameters.

[0145] It is also possible to reset the bandwidth when the optimal internal parameters and optimal external parameters corresponding to a certain bandwidth are determined, and calculate a new edge probability prediction function based on the reset window, and then perform parameter optimization under the newly set bandwidth.

[0146] The embodiment of the present application proposes a method for jointly calibrating internal and external parameters for the fusion use of a fisheye camera and a lidar, and completes the estimation of all internal and external parameters in the same optimization process. Utilizing the non-repetitive characteristics of the Livox lidar, accumulate sufficiently dense (sub-pixel level) point clouds, extract edge features in its reflectivity projection map and the projection map of the fisheye camera, and give the edge distribution probability density function of the fisheye camera through kernel density estimation (KDE). Project the edge feature points of the lidar onto the fisheye camera plane through internal and external parameter transformation, and maximize the mean square value of the probability density function of each projection point on the fisheye camera distribution to obtain the optimal internal and external parameters.

[0147] To execute the corresponding steps in the above embodiments and various possible ways, the following gives an implementation manner of an internal and external parameter calibration device 200. Optionally, the internal and external parameter calibration device 200 may adopt the Figure 1 device structure of the electronic device 100 shown above. Further, please refer to Figure 12 , Figure 12 which is a schematic block diagram of the internal and external parameter calibration device 200 provided by the embodiment of the present application. It should be noted that for the internal and external parameter calibration device 200 provided in this embodiment, its basic principle and the technical effects produced are the same as those in the above embodiments. For the sake of brief description, for the parts not mentioned in this embodiment, reference may be made to the corresponding content in the above embodiments. In this embodiment, the internal and external parameter calibration device 200 may include: an acquisition module 210, a first edge determination module 220, a second edge determination module 230, and an optimization module 240.

[0148] The acquisition module 210 is configured to acquire environmental point clouds and environmental images corresponding to the same environment. Among them, the environmental point clouds are acquired by a lidar, and the environmental images are acquired by a camera.

[0149] The first edge determination module 220 is configured to perform angle mapping, edge feature extraction, and inverse mapping on the environmental point clouds according to the initial external parameters, and obtain the target edge point clouds in the environmental point clouds, where the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes;

[0150] The second edge determination module 230 is configured to perform angle mapping, edge feature extraction, and inverse mapping on the environmental images according to the initial internal parameters, and obtain the target edge pixel points in the environmental images. Among them, the space to which the environmental images are mapped is the target space.

[0151] The optimization module 240 is used to iteratively update the initial intrinsic parameters and the initial extrinsic parameters according to the target edge pixel points and the target edge point cloud to obtain the target intrinsic parameters of the camera and the target extrinsic parameters between the laser radar and the camera.

[0152] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0153] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the internal and external parameter calibration method is implemented.

[0154] In summary, the embodiments of the present application provide an internal and external parameter calibration method, device, electronic device and readable storage medium. First, an environment image obtained by using a camera is obtained, and an environment point cloud obtained by collecting information on the same environment by using a laser radar is obtained; then, based on the initial external parameters, the environment point cloud is angle mapped, edge feature extracted and reverse mapped to obtain a target edge point cloud in the environment point cloud, wherein the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes; and according to the initial internal parameters, the environment image is angle mapped, edge feature extracted and reverse mapped to obtain target edge pixel points in the environment image, wherein the space to which the environment image is mapped is the target space; finally, according to the target edge pixel points and the target edge point cloud, the initial internal parameters and the initial external parameters are iteratively updated to obtain the target internal parameters of the camera and the target external parameters between the laser radar and the camera. In this way, the internal and external parameters of the camera and lidar can be automatically calibrated in a marker-free environment, which solves the problems existing in the existing camera internal parameter calibration method and the external parameter calibration method between the camera and lidar, and avoids the error amplification caused by the two-step calibration process.

[0155] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] In addition, each functional module in various embodiments of this application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0157] If the above functions are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0158] The above are only optional embodiments of this application and are not used to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims

1. An internal and external parameter calibration method, characterized in that, The method includes: Obtaining environmental point cloud and environmental image corresponding to the same environment, wherein the environmental point cloud is obtained by a lidar, and the environmental image is obtained by a camera; According to the initial extrinsic parameters, performing angular mapping, edge feature extraction and inverse mapping on the environmental point cloud to obtain target edge point cloud in the environmental point cloud, wherein the space to which it is mapped is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes; According to the initial intrinsic parameters, performing angular mapping, edge feature extraction and inverse mapping on the environmental image to obtain target edge pixel points in the environmental image, wherein the space to which the environmental image is mapped is the target space; According to the target edge pixel points and the target edge point cloud, iteratively updating the initial intrinsic parameters and the initial extrinsic parameters to obtain the target intrinsic parameters of the camera and the target extrinsic parameters between the lidar and the camera; Wherein, the performing angular mapping, edge feature extraction and inverse mapping on the environmental point cloud according to the initial extrinsic parameters to obtain the target edge point cloud in the environmental point cloud includes: according to the initial extrinsic parameters, transforming the environmental point cloud to the camera coordinate system to obtain the transformed environmental point cloud; for each point in the transformed environmental point cloud, calculating the corresponding first pitch angle and first azimuth angle according to the coordinates of the point in the camera coordinate system; generating a first projection image in the target space according to the first pitch angle, first azimuth angle and reflectivity corresponding to each point; performing edge extraction on the first projection image to obtain a first target edge feature; and performing inverse mapping on the first target edge feature according to the mapping method used when mapping to the target space to obtain the target edge point cloud; Wherein, the generating a first projection image in the target space according to the first pitch angle, first azimuth angle and reflectivity corresponding to each point includes: according to the first pitch angle and first azimuth angle corresponding to each point, searching for each pixel grid in the target space with the center point of the pixel grid as the center and a first preset distance as the radius; determining the gray value of the pixel grid according to the reflectivity of the searched point to obtain the first projection image; Wherein, the performing angular mapping, edge feature extraction and inverse mapping on the environmental image according to the initial intrinsic parameters to obtain the target edge pixel points in the environmental image includes: calculating the corresponding second pitch angle and second azimuth angle of each pixel point in the environmental image according to the initial intrinsic parameters and the environmental image; generating a second projection image in the target space according to the second pitch angle, second azimuth angle and gray value corresponding to each pixel point in the environmental image; performing edge extraction on the second projection image to obtain a second target edge feature; and performing inverse mapping on the second target edge feature according to the mapping method used when mapping to the target space to obtain the target edge pixel points.

2. The method according to claim 1, wherein When the field of view of the camera in the vertical direction is greater than the field of view of the lidar in the vertical direction, the obtaining of the environmental point cloud corresponding to the same environment includes: Initial point clouds obtained by the lidar at different angles; Stitching the initial point clouds at different angles to obtain the environmental point cloud, where the field of view corresponding to the environmental point cloud is not less than the field of view corresponding to the environmental image; And / or The obtaining of the environmental image corresponding to the same environment includes: Obtaining initial images obtained by the camera in the environment with different exposure durations; Performing exposure fusion on the obtained multiple initial images to obtain the environmental image.

3. The method according to any one of claims 1-2, characterized in that, The iteratively updating the initial internal parameters and initial external parameters according to the target edge pixel points and target edge point cloud to obtain the target internal parameters of the camera and the target external parameters between the lidar and the camera includes: Taking the distribution of the projection result to be consistent with the distribution of the target edge pixel points as the target, and iteratively updating the initial internal parameters and initial external parameters to obtain the target internal parameters and target external parameters, where the projection result is the pixel points obtained when the target edge point cloud is projected onto the camera image plane corresponding to the camera.

4. The method according to claim 3, wherein The taking the distribution of the projection result to be consistent with the distribution of the target edge pixel points as the target, and iteratively updating the initial internal parameters and initial external parameters to obtain the target internal parameters and target external parameters includes: Calculating a target kernel density estimation function according to the target edge pixel points; Projecting the target edge point cloud onto the camera image plane according to the current internal parameters and external parameters to determine the positions of the projection points; Calculating the function values corresponding to the projection points according to the positions of the projection points and the target kernel density estimation function; Taking the maximum function value as the target, adjusting the currently used internal parameters and external parameters, and jumping to the step according to the adjusted internal parameters and external parameters: Projecting the target edge point cloud onto the camera image plane according to the current internal parameters and external parameters to determine the positions of the projection points, until the iterative update is completed to obtain the target internal parameters and target external parameters.

5. The method according to claim 4, characterized in that, The taking the distribution of the projection result to be consistent with the distribution of the target edge pixel points as the target, and iteratively updating the initial internal parameters and initial external parameters to obtain the target internal parameters and target external parameters further includes: During the iterative update process, adjusting the window width corresponding to the target kernel density estimation function, and calculating an updated target kernel density estimation function according to the adjusted window width, where the window width before adjustment is greater than the window width after adjustment, and the target kernel density estimation function corresponding to one window width is used to calculate the function values corresponding to multiple sets of internal and external parameters.

6. An internal and external parameter calibration device, characterized in that, The device includes: An obtaining module, configured to obtain an environmental point cloud and an environmental image corresponding to the same environment, where the environmental point cloud is obtained by a lidar, and the environmental image is obtained by a camera; The first edge determination module is configured to perform angular mapping, edge feature extraction, and inverse mapping on the environmental point cloud according to the initial extrinsic parameters to obtain the target edge point cloud in the environmental point cloud, where the mapped space is a target space with the pitch angle and azimuth angle in the camera coordinate system of the camera as orthogonal axes; The second edge determination module is configured to perform angular mapping, edge feature extraction, and inverse mapping on the environmental image according to the initial intrinsic parameters to obtain the target edge pixel points in the environmental image, where the space to which the environmental image is mapped is the target space; The optimization module is configured to iteratively update the initial intrinsic parameters and the initial extrinsic parameters according to the target edge pixel points and the target edge point cloud to obtain the target intrinsic parameters of the camera and the target extrinsic parameters between the lidar and the camera; Among them, the first edge determination module is specifically configured to: convert the environmental point cloud to the camera coordinate system according to the initial extrinsic parameters to obtain the converted environmental point cloud; for each point in the converted environmental point cloud, calculate the corresponding first pitch angle and first azimuth angle of the point according to the coordinates of the point in the camera coordinate system; generate a first projection image in the target space according to the first pitch angle, first azimuth angle, and reflectivity corresponding to each point; perform edge extraction on the first projection image to obtain the first target edge feature; perform inverse mapping on the first target edge feature according to the mapping method used when mapping to the target space to obtain the target edge point cloud; Among them, the first edge determination module generates the first projection image in the following manner: according to the first pitch angle and first azimuth angle corresponding to each point, search for each pixel grid in the target space with the center point of the pixel grid as the center and a first preset distance as the radius; determine the gray value of the pixel grid according to the reflectivity of the searched point to obtain the first projection image; Among them, the second edge determination module is specifically configured to: calculate the corresponding second pitch angle and second azimuth angle of each pixel point in the environmental image according to the initial intrinsic parameters and the environmental image; generate a second projection image in the target space according to the second pitch angle, second azimuth angle, and gray value corresponding to each pixel point in the environmental image; perform edge extraction on the second projection image to obtain the second target edge feature; perform inverse mapping on the second target edge feature according to the mapping method used when mapping to the target space to obtain the target edge pixel points.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the internal and external parameter calibration method according to any one of claims 1-5.

Images