An online camera and vehicle extrinsic parameter calibration optimization method and related device

Abstract

The application discloses a kind of camera and vehicle extrinsic parameter online calibration optimization method and related equipment.Method includes: obtaining target image, extracting vanishing point in the target image, the target image is the image that omnidirectional camera is photographed when target vehicle walks in straight line;Obtain the first vanishing point and the second vanishing point satisfying the first condition, respectively join to first constraint set and second constraint set;Re-execute the step of obtaining target image until the number of vanishing points in the first constraint set and the second constraint set reaches the preset threshold, obtain target first constraint set and target second constraint set;According to the target first constraint set and the target second constraint set, the rotation extrinsic parameter of the target vehicle is optimized.The camera and vehicle extrinsic parameter online calibration optimization method provided by the application can make the sensor data obtained by the user when using the target vehicle more accurate, and drive more safely.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle external parameters, and in particular to a camera and vehicle external parameter online calibration optimization method and related equipment. BACKGROUND

[0002] In the field of automatic driving, whether it is mapping and positioning or perception detection, the raw data of multiple sensors needs to be used, including but not limited to multiple cameras, LiDAR, RTK GPS, IMU, etc. Each sensor has its own advantages and disadvantages, so the fusion of multiple sensors is particularly important, which can make the best of each sensor and achieve the purpose of automatic driving module algorithm. The fundamental of fusion is to know the external parameter relationship between each sensor, otherwise it is impossible to achieve the purpose of fusion.

[0003] Among them, the external parameters between the vehicle surround-view multi-camera and the vehicle are one of the most important parameters. There are many ways to calibrate the external parameters between surround-view cameras, such as: 1) factory calibration, which means that there is a specific calibration workshop, and the vehicle can be calibrated by arranging corresponding checkerboards or other types of calibration boards around or on the ground in the calibration workshop; 2) online calibration, which means that the sensor collects external environment data in real time during the operation of the vehicle, and performs real-time calibration.

[0004] Factory calibration generally has high precision, but factory calibration is one-time, and when the user drives the vehicle on the road, the vehicle will inevitably be jolted multiple times, which will cause the sensor on the vehicle to deviate from the position, which will cause the external parameters between the sensors to be inaccurate, thereby greatly affecting the output precision of the automatic driving algorithm. In this case, online calibration is particularly important, firstly, it can monitor whether the external parameters are accurate or have large changes in real time; secondly, it can update the external parameters when the external parameters are found to be broken.

[0005] However, online calibration relies on real-time collection of external environment data, and if the constraints provided by the external environment data are not enough, it will cause the calibration result to deviate greatly in precision. Moreover, the current online calibration technology often cannot extract enough vanishing points, which are not enough constraints, and are prone to online calibration failure. Therefore, there is an urgent need for a vehicle external parameter online calibration optimization method to ensure the smooth progress of online calibration.

[0006] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0007] In view of the above defects of the prior art, an optimization method for online calibration of camera and vehicle external parameters and a related device are provided to solve the problem of imperfect online calibration method of vehicle external parameters and large deviation of calibration result precision in the prior art.

[0008] In a first aspect, the present application provides an optimization method for online calibration of camera and vehicle external parameters, comprising:

[0009] Obtaining a target image and extracting vanishing points in the target image, the target image being an image captured by a panoramic camera when a target vehicle is walking in a straight line, the vanishing points in the target image including a first vanishing point, a second vanishing point and a third vanishing point, and the panoramic camera being placed in a middle position of a roof of the target vehicle;

[0010] Obtaining the first vanishing point and the second vanishing point satisfying a first condition and adding them to a first constraint set and a second constraint set, respectively;

[0011] Re-executing the step of obtaining a target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, to obtain a target first constraint set and a target second constraint set;

[0012] Optimizing the rotation external parameters of the target vehicle according to the target first constraint set and the target second constraint set.

[0013] The optimization method for online calibration of camera and vehicle external parameters, wherein the step of obtaining a target image comprises:

[0014] Obtaining an initial image, the initial image being an image corresponding to each frame of the panoramic camera of the target vehicle;

[0015] Obtaining a first image and performing de-distortion processing on the first image to obtain the target image, the first image being an image satisfying a second condition in the initial image.

[0016] The optimization method for online calibration of camera and vehicle external parameters, wherein the second condition is:

[0017] When the pose of the target vehicle corresponding to the shooting time of the initial image is straight driving, the initial image is output as the first image.

[0018] The optimization method for online calibration of camera and vehicle external parameters, wherein the step of extracting vanishing points in the target image comprises:

[0019] Extracting straight line segments in the target image, clustering the straight line segments in the target image and calculating intersection points to obtain the vanishing points in the target image.

[0020] The optimization method for online calibration of a camera and vehicle external parameters, wherein the first vanishing point and the second vanishing point meeting the first condition are obtained and added to a first constraint set and a second constraint set, respectively, comprises the following steps:

[0021] A target cost function is obtained, and the vanishing points are divided into the first vanishing point, the second vanishing point and the third vanishing point according to the target cost function, the first vanishing point is a vanishing point corresponding to a lane line parallel to a driving direction of the target vehicle on the ground, the second vanishing point is a vanishing point corresponding to a lane line perpendicular to the driving direction of the target vehicle on the ground, and the third vanishing point is a vanishing point corresponding to a straight line segment perpendicular to the ground;

[0022] The first vanishing point meeting the first condition is extracted and added to the first constraint set;

[0023] The second vanishing point meeting the first condition is extracted and added to the second constraint set.

[0024] The optimization method for online calibration of a camera and vehicle external parameters, wherein the target cost function is as follows:

[0025]

[0026] wherein e is a residual error, K1 is an internal parameter matrix of the panoramic camera, vp i is a position of a vanishing point in the target image, p vehicle is a homogeneous coordinate of the vanishing point in a world coordinate system, is the rotation external parameter.

[0027] The optimization method for online calibration of a camera and vehicle external parameters, wherein the first condition is that the residual error is less than or equal to 0.02.

[0028] The second aspect of the present application provides an optimization device for online calibration of a camera and vehicle external parameters, comprising:

[0029] An image acquisition module is configured to acquire a target image and extract vanishing points in the target image, the target image is an image captured by a panoramic camera when a target vehicle travels in a straight line, the vanishing points in the target image include a first vanishing point, a second vanishing point and a third vanishing point, and the panoramic camera is arranged at a middle position of a roof of the target vehicle;

[0030] A collection module is configured to obtain the first vanishing point and the second vanishing point meeting the first condition and add them to a first constraint set and a second constraint set, respectively;

[0031] a detection module configured to re-perform the step of acquiring a target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, to obtain a target first constraint set and a target second constraint set;

[0032] an optimization module configured to optimize the rotation extrinsic parameter of the target vehicle according to the target first constraint set and the target second constraint set.

[0033] In a third aspect, the present application provides a terminal, comprising: a processor, a storage medium connected to the processor in communication, the storage medium being adapted to store a plurality of instructions, and the processor being adapted to invoke the instructions in the storage medium to perform the steps of the optimization method for online calibration of camera and vehicle extrinsic parameter according to any one of the above aspects.

[0034] In a fourth aspect, the present application provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the steps of the optimization method for online calibration of camera and vehicle extrinsic parameter according to any one of the above aspects.

[0035] Advantages: Compared with the prior art, the present application provides an optimization method for online calibration of camera and vehicle extrinsic parameter and related equipment. In the optimization method for online calibration of camera and vehicle extrinsic parameter, a target image is acquired, and vanishing points in the target image are extracted. The target image is an image captured by a panoramic camera when a target vehicle is walking in a straight line. The vanishing points in the target image include first vanishing points, second vanishing points, and third vanishing points. The panoramic camera is placed in the middle position of the roof of the target vehicle. The first vanishing points and the second vanishing points that meet a first condition are acquired and added to a first constraint set and a second constraint set, respectively. Then, the step of acquiring a target image is re-performed until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, to obtain a target first constraint set and a target second constraint set. Finally, the rotation extrinsic parameter of the target vehicle is optimized according to the target first constraint set and the target second constraint set. Through the optimization method for online calibration of camera and vehicle extrinsic parameter, enough vanishing points can be extracted as constraints, and the rotation extrinsic parameter of the target vehicle is optimized through online calibration. The problem of the prior art that the online calibration method for vehicle extrinsic parameter is not perfect and the accuracy of the calibration result is large is solved. The user can obtain more accurate sensing data when using the target vehicle, and driving is safer. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 a flowchart of an embodiment of the optimization method for online calibration of camera and vehicle extrinsic parameter provided by the present application;

[0037] Figure 2A camera installation schematic diagram of an embodiment of the optimization method for online calibration of a camera and vehicle external parameters provided by the present application;

[0038] Figure 3 An external parameter online calibration flowchart in an embodiment of the optimization method for online calibration of a camera and vehicle external parameters provided by the present application;

[0039] Figure 4 A coordinate system schematic diagram in an embodiment of the optimization method for online calibration of a camera and vehicle external parameters provided by the present application;

[0040] Figure 5 A lane line schematic diagram in an embodiment of the optimization method for online calibration of a camera and vehicle external parameters provided by the present application;

[0041] Figure 6 A structure schematic diagram of an embodiment of the optimization device for online calibration of a camera and vehicle external parameters provided by the present application;

[0042] Figure 7 A structure schematic diagram of an embodiment of the terminal provided by the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0044] It can be understood by those skilled in the art that, unless specifically stated, the singular forms "a", "an" and "the" used herein include plural forms. It should be further understood that the use of the phrase "comprising" in the specification of the present application means that the features, integers, steps, operations, elements and / or components are present, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0045] Those skilled in the art of the technology can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such herein.

[0046] Embodiment one

[0047] The camera and vehicle extrinsic parameter online calibration optimization method provided in the embodiment can be executed by a terminal, which can be, but is not limited to, a smart car, a computer, etc. The following will be described taking a smart car as an example, as shown in Figure 1 The camera and vehicle extrinsic parameter online calibration optimization method provided in the embodiment includes the following steps:

[0048] In the embodiment, a camera and vehicle extrinsic parameter online calibration optimization method is provided. As shown in Figure 1 The camera and vehicle extrinsic parameter online calibration optimization method provided in the embodiment includes the following steps:

[0049] S100, acquiring a target image and extracting vanishing points in the target image, the target image being an image captured by a panoramic camera when a target vehicle is walking in a straight line, the vanishing points in the target image including a first vanishing point, a second vanishing point and a third vanishing point, and the panoramic camera being placed in a middle position of a roof of the target vehicle.

[0050] In the field of automatic driving, etc., whether it is mapping and positioning, or perception detection, the raw data of multiple sensors need to be used, including but not limited to multiple cameras, LiDAR, RTK GPS, IMU, etc. Each sensor has its own advantages and disadvantages, so the fusion of multiple sensors is particularly important, which can make the best of the advantages of multiple sensors and achieve the purpose of automatic driving module algorithm. The fundamental of fusion is to know the extrinsic relationship between each sensor, otherwise it is impossible to achieve the purpose of fusion.

[0051] Among them, the extrinsic parameters between the vehicle surround-view multi-camera and the multi-camera and the vehicle are one of the most important parameters. The extrinsic parameters refer to the relative relationship between two sensor coordinates, generally six degrees of freedom, that is, three degrees of freedom of relative translation and three degrees of freedom of relative rotation. There are many ways to calibrate the extrinsic parameters between surround-view cameras, such as factory calibration and online calibration. Factory calibration refers to a specific calibration workshop, where the vehicle is surrounded by or on the ground, and corresponding checkerboard or other types of calibration board is arranged to complete calibration; online calibration refers to real-time acquisition of external environment data by the sensor during vehicle operation, and real-time calibration.

[0052] Factory calibration generally has high precision because in the calibration workshop, there are high-precision known calibration boards or other size information. However, factory calibration is one-time, and when the user drives the vehicle automatically on the road, the vehicle will inevitably be jolted multiple times, causing the sensor on the vehicle to deviate in pose, which will cause the extrinsic parameters between the sensors to be inaccurate, thereby greatly affecting the output precision of the automatic driving algorithm. In this case, online calibration is particularly important, firstly, it can monitor whether the extrinsic parameters are accurate or have large changes in real time; secondly, it can update the extrinsic parameters when it is found that the extrinsic parameters have been damaged. However, since online calibration relies on real-time acquisition of external environment data, its precision is generally not as high as factory calibration, especially when the constraints provided by the external environment data are not sufficient, which will cause the calibration result to deviate greatly in precision.

[0053] The current automatic calibration scheme for multi-camera extrinsic parameters and camera-vehicle extrinsic parameters mainly uses the vanishing point scheme. However, the cameras installed around the vehicle in the prior art often cannot see enough lane lines, resulting in the problem that the extrinsic parameters between the surrounding cameras and the vehicle cannot be calibrated. Moreover, the cameras installed around the vehicle generally only extract the vanishing points of the road lane lines, but the vanishing points in one direction are not sufficient constraints, and often only the pitch and yaw angles can be obtained, while the roll direction cannot be observed. That is, there is no optimization method for the rotational extrinsic parameters of the vehicle in the prior art, and the optimization method based on online calibration of camera-vehicle extrinsic parameters provided in the embodiment optimizes the rotational extrinsic parameters of the vehicle, making the online calibration more complete and smooth.

[0054] Specifically, in the embodiment, the target image is obtained, including:

[0055] An initial image is obtained, the initial image being an image corresponding to each frame of the surround-view camera of the target vehicle;

[0056] A first image is obtained and de-warping processing is performed on the first image to obtain the target image, the first image being an image satisfying a second condition in the initial image.

[0057] Referring to Figure 2 , the surround camera is placed in the middle of the roof of the target vehicle, and camera represents a camera. In this embodiment, the vehicle coordinate system is defined as: the x-axis is forward, the y-axis is left, and the z-axis is upward. The camera's own coordinate system is: the x-axis is right, the y-axis is downward, and the z-axis is forward.

[0058] First, the initial image corresponding to each frame of the surround camera of the target vehicle is obtained, and the image satisfying the second condition is extracted in the initial image. The second condition is:

[0059] When the pose of the target vehicle corresponding to the shooting time of the initial image is straight, the initial image is output as the first image.

[0060] Specifically, during the recording process of the surround camera, the vehicle pose corresponding to each frame is saved synchronously to facilitate the extraction of the straight line segment of the trajectory, so as to extract the monocular vanishing point in the straight line segment of the trajectory and add the monocular constraint.

[0061] According to the vehicle pose, it is judged whether each of the initial images is in the straight line segment of the trajectory, that is, if the target vehicle is straight in the lane, it can be considered that the vehicle and the lane line are parallel, at this time, it can be considered that the world coordinate system of the lane line is consistent with the target vehicle, and the image shot by the surround camera at this time is obtained as the first image.

[0062] Referring to Figure 3 , Figure 3 is an online calibration flowchart of the external parameter, after the first image is extracted, the first image is subjected to distortion removal processing to obtain the target image.

[0063] The extraction of the vanishing point in the target image includes:

[0064] Extracting the straight line segment in the target image, clustering the straight line segment in the target image and calculating the intersection point to obtain the vanishing point in the target image.

[0065] In this embodiment, the multiple straight line segments in the target image can be extracted by the LSD (Line Segment Detection) straight line extraction algorithm. In more embodiments, the straight line segment in the target image can be extracted by other straight line detection method.

[0066] After extracting the plurality of straight line segments in the target image, the extracted straight line segments can be clustered and the intersection points can be calculated to obtain a plurality of vanishing points by using JLinkage or Ransac algorithm. The vanishing points in the target image include a first vanishing point, a second vanishing point and a third vanishing point. Since the vanishing points are obtained in the omnidirectional view of the target vehicle driving in a straight line, it can be considered that the vehicle is parallel to the lane line, in other words, the world coordinate system of the lane line is consistent with the coordinate system of the vehicle. That is, at this time, the world coordinate system and the vehicle coordinate system are defined in the same way, that is, the x-axis is forward, the y-axis is left, and the z-axis is upward. The first vanishing point can be regarded as a vanishing point near the x-axis, the second vanishing point can be regarded as a vanishing point near the y-axis, and the third vanishing point can be regarded as a vanishing point near the z-axis. Since the vanishing point near the z-axis is a vanishing point in the vertical direction of the earth, for example, a vanishing point extracted from a vertical line of a building. However, the building is vertical to the earth but not necessarily vertical to the ground in the current frame. Specifically, because the ground may have a slope, the vertical axis of the building and the vertical axis of the ground coordinate system are not the same axis sometimes, so the vanishing point in this direction cannot be used for constraint and needs to be removed. That is, the third vanishing point should be removed. The first vanishing point and the second vanishing point satisfying the first condition can be used to optimize the rotation extrinsic parameter of the target vehicle.

[0067] Specifically, the optimization method based on camera and vehicle extrinsic parameter online calibration further includes the following steps:

[0068] S200, obtaining the first vanishing point and the second vanishing point satisfying the first condition and adding them to the first constraint set and the second constraint set respectively.

[0069] The first vanishing point and the second vanishing point satisfying the first condition are added to the first constraint set and the second constraint set respectively, including:

[0070] obtaining a target cost function, and dividing the vanishing points into the first vanishing point, the second vanishing point and the third vanishing point according to the target cost function, the first vanishing point being a vanishing point corresponding to a lane line parallel to the driving direction of the target vehicle on the ground, the second vanishing point being a vanishing point corresponding to a lane line perpendicular to the driving direction of the target vehicle on the ground, and the third vanishing point being a vanishing point corresponding to a straight line segment perpendicular to the ground;

[0071] extracting the first vanishing point satisfying the first condition and adding it to the first constraint set;

[0072] extracting the second vanishing point satisfying the first condition and adding it to the second constraint set.

[0073] The target cost function is:

[0074]

[0075] wherein e is a residual error, K1 is an intrinsic matrix of the omnidirectional camera, vp i is a position of a vanishing point in the target image, p vehicle is a homogeneous coordinate corresponding to the vanishing point in a world coordinate system, is the rotation extrinsic parameter.

[0076] Specifically, in the embodiment, the first vanishing point, the second vanishing point and the third vanishing point are distinguished by the target cost function, and the third vanishing point is eliminated. At the same time, only the first vanishing point and the second vanishing point satisfying the first condition are reserved.

[0077] In the embodiment, the target cost function is:

[0078]

[0079] wherein e is a residual error, K1 is an intrinsic matrix of the omnidirectional camera, vp i is a position of a vanishing point in the target image, p vehicle is a homogeneous coordinate corresponding to the vanishing point in a world coordinate system, is the rotation extrinsic parameter. By calculating the result of , it can be judged that the current vanishing point is the first vanishing point, the second vanishing point or the third vanishing point.

[0080] In the embodiment, the first condition is that the residual error is less than or equal to 0.02. In more embodiments, according to the different needs of accuracy in actual situation, the first condition can be set to be that the residual error is less than or equal to other threshold value. For example, in the case of low accuracy requirement, the threshold value can be set to 0.03, 0.04 or higher value; in the case of high accuracy requirement, the threshold value can be set to 0.015, 0.01 or lower value.

[0081] Specifically, referring to Figure 4 , Figure 4 is a coordinate system diagram of the omnidirectional camera and the target vehicle, which shows a matrix corresponding to the theoretical rotation extrinsic parameter, that is, assuming that the camera is perfectly installed, the residual error e = 0,

[0082] Similarly, referring to Figure 4 It can be seen that since the first vanishing point is a vanishing point near the x-axis in the vehicle coordinate system, corresponding to the z-axis of the omnidirectional camera coordinate axis, the homogeneous coordinate corresponding to the first vanishing point is pvehicle (1) = [1 0 0] T or p vehicle (1) = [-1 0 0] T It can be seen that in the case of the residual error e = 0, Therefore, for the first vanishing point, it is required that or, Since in the present embodiment, the first condition is that the residual error is less than or equal to 0.02, when the calculation result of is within the range of 0.02 from or , it can be considered that the first vanishing point satisfies the first condition.

[0083] Similarly, since the second vanishing point is a vanishing point near the y-axis in the vehicle coordinate system, corresponding to the x-axis of the panoramic camera coordinate axis, the homogeneous coordinates corresponding to the second vanishing point are p vehicle (2) = [0 1 0] T or p vehicle (2) = [0 -1 0] T It can be seen that in the case of the residual error e = 0, Therefore, for the second vanishing point, it is required that or, Since in the present embodiment, the first condition is that the residual error is less than or equal to 0.02, when the calculation result of is within the range of 0.02 from or , it can be considered that the second vanishing point satisfies the first condition.

[0084] Similarly, since the third vanishing point is a vanishing point near the z-axis in the vehicle coordinate system, corresponding to the y-axis of the panoramic camera coordinate axis, the homogeneous coordinates corresponding to the third vanishing point are p vehicle (3) = [0 0 1] T or p vehicle (3) = [0 0 -1] T It can be seen that in the case of the residual error e = 0, Therefore, for the third vanishing point, it is required that or, At this time, when the calculation result of is similar to or , it can be considered that the vanishing point is the third vanishing point, and it is directly removed.

[0085] With reference to Figure 5 , Figure 5 The straight line is perpendicular to the forward lane line on the ground in some embodiments.

[0086] With reference to Figure 1 , the optimization method based on camera and vehicle extrinsic parameter line calibration further comprises the steps of:

[0087] S300, re-performing the step of acquiring the target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, to obtain a target first constraint set and a target second constraint set;

[0088] S400, optimizing the rotation extrinsic parameter of the target vehicle according to the target first constraint set and the target second constraint set.

[0089] Specifically, through the above steps S100 and S200, after processing and rejecting each of the target images, the number of vanishing points in two perpendicular directions on the ground is determined in real time. When the number of vanishing points in each direction reaches the preset threshold, the next step of unified optimization can be performed. Otherwise, it is considered that the constraints are not enough, and the optimization calibration cannot be started yet. Return to step S100 and continue to process the next target image until the number of vanishing points is sufficient. The preset threshold can be set according to the user's requirement for accuracy. In this embodiment, the preset threshold can be 30, that is, at least 30 first vanishing points and at least 30 second vanishing points are required for the next optimization processing. In more embodiments, the preset threshold can be 40-50 or higher. The higher the preset threshold, the more accurate the optimization of the rotation extrinsic parameter. The rotation extrinsic parameter obtained by optimization is more accurate.

[0090] It can be seen that in this embodiment, the algorithm is only optimized when a sufficient number of high-quality vanishing point constraints are reached, which can ensure the robustness of the algorithm. Otherwise, insufficient constraints or low-quality constraints (a large number of vanishing points that do not meet the first condition) may cause the algorithm to fail to converge or enter a local optimum, resulting in an unsatisfactory calibration result.

[0091] In summary, the embodiment provides an optimization method for camera and vehicle extrinsic parameter online calibration, by acquiring a target image, extracting vanishing points in the target image, the target image being an image captured by a panoramic camera when a target vehicle is walking in a straight line, the vanishing points in the target image including a first vanishing point, a second vanishing point and a third vanishing point, the panoramic camera being placed in the middle position of the roof of the target vehicle, the first vanishing point and the second vanishing point satisfying a first condition are acquired and added to a first constraint set and a second constraint set respectively, then the step of acquiring a target image is re-executed until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, target first and second constraint sets are obtained, and finally, the rotation extrinsic parameter of the target vehicle is optimized according to the target first and second constraint sets. Through the optimization method for camera and vehicle extrinsic parameter online calibration provided in the embodiment, enough vanishing points can be extracted as constraints, and the rotation extrinsic parameter of the target vehicle is optimized through online calibration, solving the problem that the vehicle extrinsic parameter online calibration method in the prior art is not perfect and the calibration result has large precision deviation, so that the user obtains more accurate sensor data when using the target vehicle, and driving is safer.

[0092] It should be understood that, although each step in the flowchart shown in the drawings of the present application specification is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of the steps in the present application has no strict sequence limitation, and these steps can be executed in other orders. Moreover, at least part of the steps of the present application can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] Embodiment two

[0095] Based on the above-mentioned embodiments, the present application further provides an optimization device for online calibration of camera and vehicle external parameters, a functional module schematic diagram of which is shown in Figure 6 The optimization device for online calibration of camera and vehicle external parameters comprises:

[0096] An image acquisition module, which is configured to acquire a target image and extract vanishing points in the target image. The target image is an image captured by a panoramic camera when a target vehicle is walking in a straight line. The vanishing points in the target image include a first vanishing point, a second vanishing point and a third vanishing point. The panoramic camera is placed in the middle position of the roof of the target vehicle, as described in detail in Embodiment One.

[0097] A collection module, which is configured to acquire the first vanishing point and the second vanishing point satisfying a first condition and add them to a first constraint set and a second constraint set respectively, as described in detail in Embodiment One.

[0098] A detection module, which is configured to re-execute the step of acquiring a target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, so as to obtain a target first constraint set and a target second constraint set, as described in detail in Embodiment One.

[0099] An optimization module configured to optimize the rotational extrinsic parameters of the target vehicle according to the target first constraint set and the target second constraint set, as described in detail in Embodiment One.

[0100] Embodiment Three

[0101] Based on the optimization method for online calibration of camera and vehicle extrinsic parameters described in Embodiment One, the present application also provides a terminal, the principle block diagram of which can be as shown in Figure 7 The terminal includes a memory 20 and a processor 10, the memory 20 stores an optimization program for online calibration of camera and vehicle extrinsic parameters, and the processor 10, when executing the computer program, can at least implement the following steps:

[0102] Obtaining a target image and extracting vanishing points in the target image, the target image being an image captured by a panoramic camera when a target vehicle is walking in a straight line, the vanishing points in the target image including a first vanishing point, a second vanishing point and a third vanishing point, and the panoramic camera being placed in the middle position of the roof of the target vehicle;

[0103] Obtaining the first vanishing point and the second vanishing point satisfying the first condition and adding them to the first constraint set and the second constraint set, respectively;

[0104] Re-executing the step of obtaining a target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, to obtain a target first constraint set and a target second constraint set;

[0105] Optimizing the rotational extrinsic parameters of the target vehicle according to the target first constraint set and the target second constraint set.

[0106] The step of obtaining a target image includes:

[0107] Obtaining an initial image, the initial image being an image corresponding to each frame of the panoramic camera of the target vehicle;

[0108] Obtaining a first image and performing de-distortion processing on the first image to obtain the target image, the first image being an image satisfying a second condition in the initial image.

[0109] The second condition is:

[0110] When the pose of the target vehicle corresponding to the image capturing time of the initial image is straight driving, outputting the initial image as the first image.

[0111] The step of extracting vanishing points in the target image includes:

[0112] Extracting straight line segments in the target image, clustering the straight line segments in the target image and calculating intersection points to obtain vanishing points in the target image.

[0113] The first vanishing point and the second vanishing point satisfying the first condition are obtained and added to a first constraint set and a second constraint set, respectively, including:

[0114] A target cost function is obtained, and the vanishing points are divided into the first vanishing point, the second vanishing point and the third vanishing point according to the target cost function, the first vanishing point is a vanishing point corresponding to a lane line on the ground parallel to the driving direction of the target vehicle, the second vanishing point is a vanishing point corresponding to a lane line on the ground perpendicular to the driving direction of the target vehicle, and the third vanishing point is a vanishing point corresponding to a straight line segment perpendicular to the ground;

[0115] The first vanishing point satisfying the first condition is extracted and added to the first constraint set;

[0116] The second vanishing point satisfying the first condition is extracted and added to the second constraint set.

[0117] The target cost function is:

[0118]

[0119] Wherein e is a residual error, K1 is an intrinsic matrix of the panoramic camera, vp i is the position of the vanishing point in the target image, p vehicle is the homogeneous coordinates of the vanishing point in the world coordinate system, is the rotation extrinsic parameter.

[0120] The first condition is that the residual error is less than or equal to 0.02.

[0121] Embodiment four

[0122] The application also provides a storage medium, the storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the steps of the camera and vehicle extrinsic parameter online calibration optimization method described in the above embodiments.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An optimization method based on online calibration of camera and vehicle extrinsic parameters, characterized in that, include: Acquire a target image and extract the vanishing points in the target image. The target image is an image captured by a panoramic camera when the target vehicle is traveling in a straight line. The vanishing points in the target image include a first vanishing point, a second vanishing point, and a third vanishing point. The panoramic camera is positioned at the center of the roof of the target vehicle. Obtain the first vanishing point and the second vanishing point that satisfy the first condition, and add them to the first constraint set and the second constraint set, respectively. Repeat the steps of acquiring the target image until the number of vanishing points in the first constraint set and the second constraint set reaches a preset threshold, and obtain the first constraint set and the second constraint set of the target; The rotational extrinsic parameters of the target vehicle are optimized based on the first set of target constraints and the second set of target constraints. The step of obtaining the first vanishing point and the second vanishing point that satisfy the first condition, and adding them to the first constraint set and the second constraint set respectively, includes: Obtain the target cost function, and divide the vanishing point into a first vanishing point, a second vanishing point, and a third vanishing point according to the target cost function. The first vanishing point is the vanishing point corresponding to the lane line on the ground that is parallel to the direction of travel of the target vehicle. The second vanishing point is the vanishing point corresponding to the lane line on the ground that is perpendicular to the direction of travel of the target vehicle. The third vanishing point is the vanishing point corresponding to the straight line segment that is perpendicular to the ground. Extract the first vanishing point that satisfies the first condition and add it to the first constraint set; Extract the second vanishing point that satisfies the first condition and add it to the second constraint set; the objective cost function is: ； Where e is the residual, Let be the intrinsic parameter matrix of the panoramic camera. The location of the vanishing point in the target image. These are the homogeneous coordinates of the vanishing point in the world coordinate system. The rotational extrinsic parameter; The first condition is that the residual is less than or equal to 0.

02.

2. The optimized method for online calibration of camera and vehicle extrinsic parameters according to claim 1, characterized in that, The acquisition of the target image includes: Acquire an initial image, which is the image corresponding to each frame of the panoramic camera of the target vehicle; A first image is acquired and distortion correction is performed on the first image to obtain the target image, wherein the first image is the image in the initial image that satisfies the second condition.

3. The optimized method for online calibration of camera and vehicle extrinsic parameters according to claim 2, characterized in that, The second condition is: When the target vehicle's pose is straight when the initial image is captured, the initial image is output as the first image.

4. The optimized method for online calibration of camera and vehicle extrinsic parameters according to claim 1, characterized in that, Extracting the vanishing points in the target image includes: Extract straight line segments from the target image, cluster the straight line segments in the target image and calculate the intersection points to obtain the vanishing points in the target image.

5. An optimization device for online calibration of camera and vehicle extrinsic parameters, characterized in that, The device includes: An image acquisition module is used to acquire a target image and extract the vanishing points in the target image. The target image is an image captured by a panoramic camera when the target vehicle is traveling in a straight line. The vanishing points in the target image include a first vanishing point, a second vanishing point, and a third vanishing point. The panoramic camera is positioned at the center of the roof of the target vehicle. The collection module is used to obtain the first vanishing point and the second vanishing point that satisfy the first condition, and add them to the first constraint set and the second constraint set respectively; The detection module is used to re-execute the step of acquiring the target image until the number of vanishing points in the first constraint set and the second constraint set respectively reaches a preset threshold, so as to obtain the first constraint set and the second constraint set of the target; An optimization module is used to optimize the rotational extrinsic parameters of the target vehicle based on the first set of target constraints and the second set of target constraints. The step of obtaining the first vanishing point and the second vanishing point that satisfy the first condition, and adding them to the first constraint set and the second constraint set respectively, includes: Obtain the target cost function, and divide the vanishing point into a first vanishing point, a second vanishing point, and a third vanishing point according to the target cost function. The first vanishing point is the vanishing point corresponding to the lane line on the ground that is parallel to the direction of travel of the target vehicle. The second vanishing point is the vanishing point corresponding to the lane line on the ground that is perpendicular to the direction of travel of the target vehicle. The third vanishing point is the vanishing point corresponding to the straight line segment that is perpendicular to the ground. Extract the first vanishing point that satisfies the first condition and add it to the first constraint set; Extract the second vanishing point that satisfies the first condition and add it to the second constraint set; the objective cost function is: ； Where e is the residual, Let be the intrinsic parameter matrix of the panoramic camera. The location of the vanishing point in the target image. These are the homogeneous coordinates of the vanishing point in the world coordinate system. The rotational extrinsic parameter; The first condition is that the residual is less than or equal to 0.

02.

6. A terminal, characterized in that, The terminal includes: a processor and a storage medium communicatively connected to the processor. The storage medium is adapted to store multiple instructions, and the processor is adapted to call the instructions in the storage medium to execute the steps of the optimization method for online calibration of camera and vehicle extrinsic parameters as described in any one of claims 1-4.

7. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the optimization method for online calibration of camera and vehicle extrinsic parameters as described in any one of claims 1-4.

Images