Camera dynamic calibration method and device and vehicle

Abstract

The invention provides a camera dynamic calibration method and device and a vehicle, and the method comprises the steps: in a data preparation stage, based on a road environment image collected by a target camera in a vehicle in a vehicle driving process, obtaining target data needing to be provided in the data preparation stage, the target data comprising a first feature point set, and in a camera calibration stage, the target data comprising a second feature point set; the second feature point set needed to be used in the camera calibration stage is obtained based on the target data, the external parameter calibration result of the target camera is determined based on the second feature point set, the second feature point set comprises the first feature point set, a calibration plate and an artificial marker do not need to be used in the whole calibration process, and the calibration efficiency of the target camera is greatly improved; meanwhile, according to the method, different feature point sets can be respectively obtained in different stages of dynamic calibration of the target camera according to calibration requirements, so that the calibration precision of the target camera can be effectively ensured.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a camera dynamic calibration method, device and vehicle. Background Art

[0002] With the popularization of intelligent driving technology, the demand for sensor calibration is increasing. Calibration can achieve spatial synchronization between sensors and car bodies, which is the basis and important underlying technology for automatic driving. The calibration methods of vehicle cameras usually include static calibration and dynamic calibration. Static calibration requires determining the external parameters of the camera through a single frame image in a calibration room or a fixed site, while dynamic calibration can complete the calibration through multiple frames of images in an environment that does not require advance arrangement (such as roads). That is, dynamic calibration does not rely on a specific calibration room compared to static calibration, has lower cost, more flexible implementation, and better environmental adaptability. Therefore, dynamic calibration is widely used.

[0003] At present, in the process of dynamic camera calibration, there is a problem that the camera cannot be calibrated quickly and accurately, thereby failing to ensure the performance and safety of vehicle intelligent driving. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a camera dynamic calibration method, device and vehicle to solve the problem that the camera cannot be calibrated quickly and accurately in the prior art.

[0005] To achieve the above technical objectives, the present application provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present specification provides a camera dynamic calibration method, including:

[0007] In the data preparation stage, based on the road environment image captured by the target camera in the vehicle during the vehicle driving process, the target data required to be provided in the data preparation stage is obtained, and the target data includes a first feature point set;

[0008] In the camera calibration stage, based on the target data, a second feature point set required for the camera calibration stage is obtained, and based on the second feature point set, an extrinsic calibration result of the target camera is determined; the second feature point set includes the first feature point set.

[0009] In one implementation, the acquiring of the target data required to be provided in the data preparation stage based on the road environment image captured by the target camera in the vehicle during the vehicle driving process includes:

[0010] Determining a target calibration strategy for the target camera based on image features in the road environment image;

[0011] Based on the target calibration strategy, determining the target feature point set type corresponding to the data preparation stage from the candidate feature point set types corresponding to the data preparation stage;

[0012] Based on the road environment image and the target feature point set type corresponding to the data preparation stage, the first feature point set is acquired.

[0013] In one implementation, the candidate feature point set type corresponding to the data preparation stage includes a first sub-feature point set and a second sub-feature point set, the first sub-feature point set includes edge points on a straight line segment, and the second sub-feature point set includes obstacle corner points.

[0014] In one embodiment, the target data further includes a target image;

[0015] The step of obtaining target data required to be provided in the data preparation stage based on the road environment image captured by the target camera in the vehicle during the vehicle driving process includes:

[0016] Based on the required image features in the camera calibration phase, the road environment image is segmented to obtain the target image, and the required image features are used to determine a feature point set in the second feature point set that is outside the first feature point set.

[0017] In one embodiment, the camera calibration phase includes an initial calibration phase and a calibration optimization phase;

[0018] The step of obtaining, based on the target data, a second feature point set required for use in the camera calibration phase includes:

[0019] Based on the target calibration strategy, determining the target feature point set type corresponding to the initial calibration stage from the candidate feature point set types corresponding to the initial calibration stage, and, based on the target calibration strategy, determining the target feature point set type corresponding to the calibration optimization stage from the candidate feature point set types corresponding to the calibration optimization stage;

[0020] Based on the target data and the target feature point set type corresponding to the initial calibration stage, the third feature point set corresponding to the initial calibration stage is determined, and based on the target data and the target feature point set type corresponding to the calibration optimization stage, the fourth feature point set corresponding to the calibration optimization stage is determined; wherein the second feature point set includes the third feature point set and the fourth feature point set.

[0021] In one embodiment, the candidate feature point set types corresponding to the initial calibration stage include a third sub-feature point set and a fourth sub-feature point set, the third sub-feature point set includes a vanishing point set and an edge point set, the edge point set includes edge points on a straight line segment; the fourth sub-feature point set includes the central inner points of the virtual grid.

[0022] In one embodiment, the candidate feature point set types corresponding to the calibration optimization stage include a fifth sub-feature point set and a sixth sub-feature point set, the fifth sub-feature point set includes obstacle corner points, and the sixth sub-feature point set includes intersection points of a bird's-eye view virtual marking line and a straight line segment.

[0023] In one implementation, determining the extrinsic calibration result of the target camera based on the second feature point set includes:

[0024] In the initial calibration stage, one or more groups of initial extrinsic parameter values ​​of the target camera are determined based on the third feature point set;

[0025] In the calibration optimization stage, the initial extrinsic parameter value is iteratively optimized based on the fourth feature point set;

[0026] Based on the iterative optimization results of each group of the initial extrinsic parameter values, the extrinsic parameter calibration results of the target camera are determined.

[0027] In a second aspect, an embodiment of the present specification provides a camera dynamic calibration device, including:

[0028] A first processing module is used to obtain target data required to be provided in the data preparation stage based on a road environment image captured by a target camera in the vehicle during the vehicle's driving process, wherein the target data includes a first feature point set;

[0029] The second processing module is used to obtain a second feature point set required for use in the camera calibration stage based on the target data during the camera calibration stage, and determine an external parameter calibration result of the target camera based on the second feature point set; the second feature point set includes the first feature point set.

[0030] In a third aspect, an embodiment of the present specification provides a vehicle, including a camera and a camera dynamic calibration device as described above.

[0031] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the camera dynamic calibration method as described in any one of the above items is implemented.

[0032] In a fifth aspect, an embodiment of the present specification provides a computer program product or a computer program, wherein the computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium; the processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, it implements the camera dynamic calibration method as described in any one of the above items.

[0033] It can be seen from the above technical scheme that the embodiments of the present application provide a camera dynamic calibration method, device and vehicle. In the data preparation stage, the method obtains the target data required to be provided in the data preparation stage based on the road environment image collected by the target camera in the vehicle during the vehicle driving process, and the target data includes a first feature point set. In the camera calibration stage, the second feature point set required for the camera calibration stage is obtained based on the target data, and the external parameter calibration result of the target camera is determined based on the second feature point set. The second feature point set includes the first feature point set. No calibration plate and artificial markers are required in the entire calibration process, which greatly improves the calibration efficiency of the target camera. At the same time, according to the calibration requirements, different feature point sets can be obtained at different stages of dynamic calibration of the target camera through the method of the present application, so that the calibration accuracy of the target camera can be effectively guaranteed. Therefore, the target camera can be dynamically calibrated quickly and accurately through the method of the present application, thereby improving the performance and safety of vehicle intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0035] Figure 1 A schematic flow chart of a camera dynamic calibration method provided for an implementation of this specification.

[0036] Figure 2 A schematic diagram of the intersection of a virtual marking line and a straight line segment in a bird's-eye view is provided for an implementation manner of this specification.

[0037] Figure 3 A schematic diagram of the structure of a camera dynamic calibration device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0038] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of this specification shall have the common meanings understood by persons with ordinary skills in the field to which this specification belongs. The words "first", "second" and similar words used in the embodiments of this specification do not indicate any order, quantity or importance, but are only used to avoid confusion of constituent elements.

[0039] Unless the context requires otherwise, throughout the specification, "plurality" means "at least two", and "including" is interpreted as an open, inclusive meaning, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" are intended to indicate that a particular feature, structure, material or characteristic associated with the embodiment or example is included in at least one embodiment or example of the specification. The schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0040] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0041] Overview

[0042] As described in the background technology, with the popularization of intelligent driving technology for automobiles, the demand for sensor calibration is increasing. Calibration can achieve spatial synchronization between sensors and automobile bodies, which is the basis and important underlying technology for automatic driving of automobiles. The calibration methods of vehicle cameras usually include static calibration and dynamic calibration. Static calibration requires determining the external parameters of the camera through a single-frame image in a calibration room or a fixed site, while dynamic calibration can complete the calibration through multiple frames of images in an environment that does not require advance arrangement (such as roads). That is, dynamic calibration does not rely on a specific calibration room compared to static calibration, has lower cost, more flexible implementation, and better environmental adaptability. Therefore, dynamic calibration is widely used.

[0043] At present, in the process of dynamic camera calibration, the online calibration method of multiple cameras under non-common view cameras requires the use of a calibration board. The online calibration method of multiple cameras under common view cameras follows the pre-set common viewpoint optimization external parameters, which cannot guarantee the camera calibration efficiency and the accuracy of the calibration results, and thus cannot guarantee the performance and safety of vehicle intelligent driving.

[0044] In order to solve the problem that traditional methods cannot calibrate cameras quickly and accurately, in the technical solution of the present application, in the data preparation stage, based on the road environment image collected by the target camera in the vehicle during the driving process of the vehicle, the target data required for the data preparation stage is obtained, and the target data includes a first feature point set, and in the camera calibration stage, the second feature point set required for the camera calibration stage is obtained based on the target data, and the external parameter calibration result of the target camera is determined based on the second feature point set, and the second feature point set includes the first feature point set. In the entire calibration process, there is no need to use calibration plates and artificial markers, which greatly improves the calibration efficiency of the target camera; at the same time, according to the calibration requirements, different feature point sets can be obtained at different stages of dynamic calibration of the target camera through the method of the present application, so that the calibration accuracy of the target camera can be effectively guaranteed. Therefore, the target camera can be dynamically calibrated quickly and accurately through the method of the present application, thereby improving the performance and safety of vehicle intelligent driving.

[0045] Based on the above-mentioned inventive concept, the camera dynamic calibration method provided in the implementation manner of this specification is exemplarily described below.

[0046] Exemplary Methods

[0047] The embodiments of this specification provide a camera dynamic calibration method, such as Figure 1 As shown, including:

[0048] S101. In a data preparation stage, based on a road environment image captured by a target camera in the vehicle during vehicle driving, target data required for the data preparation stage is obtained, wherein the target data includes a first feature point set.

[0049] Specifically, the vehicle may be a vehicle with an intelligent driving function, and the target camera in the vehicle may be a camera to be dynamically calibrated, for example, the target camera may be a surround view camera or a single camera.

[0050] The process of dynamically calibrating the target camera may include a data preparation stage and a camera calibration stage. The data preparation stage may be used to provide target data for the camera calibration stage. The target data may include data required for use in the camera calibration process. For example, the target data may include a first feature point set, which may be a feature point set obtained from a road environment image that needs to be continuously collected, such as a feature point set determined by tracking the trajectory of the feature points. In addition, the target data may also include a target image used to calculate the feature point set required for use in the camera calibration process, etc., which may be specifically set according to actual needs.

[0051] In implementation, in the data preparation stage, the road environment image captured by the target camera can be acquired in real time during the vehicle driving process, and the target data required for the camera calibration stage in the data preparation stage can be acquired based on the road environment image. The road environment image can include the characteristics of the road on which the vehicle is driving, and can also include the environmental characteristics around the vehicle, such as obstacles, etc. It can be understood that when the target camera is a surround view camera, the road environment image to be acquired needs to be a multi-frame multi-channel synchronous image.

[0052] Therefore, in the process of dynamic calibration of the target camera, there is no need to use calibration plates and artificial markers, which greatly improves the calibration efficiency of the target camera.

[0053] S102. In a camera calibration phase, based on the target data, a second feature point set required for use in the camera calibration phase is obtained, and an extrinsic calibration result of the target camera is determined based on the second feature point set; the second feature point set includes the first feature point set.

[0054] Specifically, in the camera calibration stage, the second feature point set required for the camera calibration stage can be obtained based on the target data provided in the data preparation stage, and the second feature point set can be used to perform external parameter calibration on the target camera. For example, the second feature point set may include the first feature point set, and may also include a feature point set calculated based on the first feature point set, and may also include a feature point set calculated based on the target image in the target data.

[0055] In implementation, the target camera may be calibrated with external parameters based on the second feature point set to obtain the extrinsic calibration result of the target camera. For example, the target camera may be calibrated with external parameters using the second feature point set based on the target calibration strategy. The target calibration strategy may be a pre-set calibration strategy, or a calibration strategy adaptively determined based on the road environment image collected by the target camera, or a calibration strategy determined in response to the user's input operation on the calibration strategy, thereby effectively ensuring the flexibility of the target camera calibration and the accuracy of the calibration result.

[0056] It can be understood that the first feature point set and the second feature point set can be feature point sets determined based on the target calibration strategy, so that in the process of extrinsic calibration of the target camera according to the second feature point set, the validity of the calibration result can be effectively guaranteed.

[0057] In addition, the camera calibration stage can be run online, that is, the external parameter calibration can be performed according to the target data provided in real time in the data preparation stage to improve the calibration efficiency of the target camera; in addition, the camera calibration stage can also be run offline, that is, during the vehicle driving process, only the data preparation stage can be executed and the target data can be stored, thereby effectively reducing the calibration failure caused by abnormal exit during the target camera calibration process due to failure to meet the calibration conditions, thereby improving the probability of successful calibration of the target camera.

[0058] After determining the external parameter calibration result of the target camera, the evaluation information of the external parameter calibration result can also be obtained, and whether the external parameter calibration result meets the calibration requirements can be determined based on the evaluation information. For example, the evaluation information of the external parameter calibration result can include calibration errors (such as reprojection errors), and can also include the stitching of images collected by multiple cameras. Among them, it can be determined whether the external parameter calibration result meets the calibration requirements based on the evaluation information. When the external parameter calibration result meets the calibration requirements, the calibration file or calibration parameters of the target camera can be updated based on the external parameter calibration result. When the external parameter calibration result does not meet the calibration requirements, the data preparation stage and the camera calibration stage can be repeatedly executed until the calibration exit conditions are met. The calibration exit conditions can include that the external parameter calibration result meets the calibration requirements, and can also include that the road environment, the driving mode of the vehicle, etc. do not meet the calibration conditions of the target camera.

[0059] In a feasible implementation manner, the target data required to be provided in the data preparation stage is obtained based on the road environment image captured by the target camera in the vehicle during the vehicle driving process, including:

[0060] Determining a target calibration strategy for the target camera based on image features in the road environment image;

[0061] Based on the target calibration strategy, determining the target feature point set type corresponding to the data preparation stage from the candidate feature point set types corresponding to the data preparation stage;

[0062] Based on the road environment image and the target feature point set type corresponding to the data preparation stage, the first feature point set is acquired.

[0063] Specifically, the image features in the road environment image may be features in the road environment image that can be used to perform external parameter calibration on the target camera. For example, the image features may include straight line features and obstacle features, etc. The straight line features may include lane lines, storage lines, stop lines, etc. In implementation, the target calibration strategy of the target camera may be determined based on the image features in the road environment image. For example, the target calibration strategy may be determined based on the image features in the road environment image and a predetermined correspondence between the type of the image features and the calibration strategy.

[0064] The candidate feature point set types corresponding to the data preparation stage may include the types of feature point sets that need to be provided in the data preparation stage under each optional calibration strategy. In implementation, the target feature point set type corresponding to the data preparation stage may be determined from the candidate feature point set types corresponding to the data preparation stage based on the target calibration strategy. For example, the target feature point set type corresponding to the data preparation stage may be determined based on the target calibration strategy and a predetermined correspondence between the calibration strategy and the feature point set type corresponding to the data preparation stage.

[0065] Among them, the first feature point set can be determined according to the road environment image collected in real time by the target camera during vehicle driving and the target feature point set type corresponding to the data preparation stage, that is, the type of the first feature point set is the target feature point set type.

[0066] Therefore, the calibration strategy of the target camera and the first feature point set required by the calibration strategy can be adaptively updated according to the image features in the road environment image, and the accuracy of the calibration results can be effectively improved during the process of extrinsic calibration of the target camera during the camera calibration phase. Considering the variability of the road environment during the dynamic driving of the vehicle, the accuracy of the calibration results of the target camera can be guaranteed under different road environments through the method of this embodiment, which greatly improves the environmental adaptability of the camera calibration, and the camera calibration process does not need to be limited to a specific road environment, thereby effectively improving the success probability of the camera calibration.

[0067] In a feasible implementation, the candidate feature point set type corresponding to the data preparation stage includes a first sub-feature point set and a second sub-feature point set, the first sub-feature point set includes edge points on a straight line segment, and the second sub-feature point set includes obstacle corner points.

[0068] Specifically, the candidate feature point set type corresponding to the data preparation stage may include a first sub-feature point set, and the first sub-feature point set may include edge points on a straight line segment. In the implementation, in the data preparation stage, the first sub-feature point set may be obtained based on a continuous multi-frame road environment image collected by a target camera during vehicle driving. For example, a straight line segment in the first frame of the road environment image, such as a lane line, a storage line, a stop line, etc., may be extracted by linear detection, and a plurality of candidate edge points may be selected from the detected straight line segments. For example, a plurality of candidate edge points may be obtained at a predetermined interval, or a plurality of candidate edge points may be obtained randomly; wherein, each candidate edge point may be tracked in the multi-frame road environment image, for example, each candidate edge point may be tracked by an optical flow tracking method, and each edge point in the first sub-feature point set may be determined based on each candidate edge point whose trajectory is successfully tracked. For example, each candidate edge point whose trajectory is successfully tracked may be directly used as each edge point in the first sub-feature point set, or each edge point in the first sub-feature point set may be determined from each candidate edge point whose trajectory is successfully tracked based on a linear geometric constraint, so that the first sub-feature point set may be determined quickly and accurately in the data preparation stage.

[0069] The candidate feature point set type corresponding to the data preparation stage may also include a second sub-feature point set, and the second sub-feature point set may include obstacle corner points. In implementation, in the data preparation stage, the second sub-feature point set may be obtained based on the continuous multi-frame road environment images collected by the target camera during the vehicle's driving process. For example, the obstacle corner points in the first frame of the road environment image may be detected by a corner point detector and used as candidate corner points; wherein, each candidate corner point may be tracked in the multi-frame road environment image, for example, each candidate corner point may be tracked by an optical flow tracking method or a corner point descriptor, and each candidate corner point with a successful trajectory tracking is used as an obstacle corner point in the second sub-feature point set, so that the second sub-feature point set can be quickly and accurately determined in the data preparation stage.

[0070] In a feasible implementation manner, the target data further includes a target image;

[0071] The step of obtaining target data required to be provided in the data preparation stage based on the road environment image captured by the target camera in the vehicle during the vehicle driving process includes:

[0072] Based on the required image features in the camera calibration phase, the road environment image is segmented to obtain the target image, and the required image features are used to determine a feature point set in the second feature point set that is outside the first feature point set.

[0073] Specifically, the target data may further include a target image, and the second image may be used to determine a feature point set other than the first feature point set in the second feature point set during the camera calibration phase.

[0074] The required image features in the camera calibration phase may include image features required to be used in the process of determining a feature point set other than the first feature point set in the second feature point set in the camera calibration phase.

[0075] In implementation, in the data preparation stage, each road environment image can be segmented based on the required image features in the camera calibration stage to obtain a target image. For example, when the target camera is a surround camera, the target image can include the seam line and sensor images on both sides of the seam line. In the camera calibration stage, the second feature point set can be determined based on the required image features in the target image.

[0076] Therefore, image segmentation can effectively screen out invalid images in the road environment image captured by the target camera during vehicle driving. In the camera calibration stage, the feature point set other than the first feature point set in the second feature point set can be quickly and effectively obtained according to the target image, further improving the external parameter calibration efficiency of the target camera and the accuracy of the calibration results.

[0077] In a feasible implementation manner, the camera calibration phase includes an initial calibration phase and a calibration optimization phase;

[0078] The step of obtaining, based on the target data, a second feature point set required for use in the camera calibration phase includes:

[0079] Based on the target calibration strategy, determining the target feature point set type corresponding to the initial calibration stage from the candidate feature point set types corresponding to the initial calibration stage, and, based on the target calibration strategy, determining the target feature point set type corresponding to the calibration optimization stage from the candidate feature point set types corresponding to the calibration optimization stage;

[0080] Based on the target data and the target feature point set type corresponding to the initial calibration stage, the third feature point set corresponding to the initial calibration stage is determined, and based on the target data and the target feature point set type corresponding to the calibration optimization stage, the fourth feature point set corresponding to the calibration optimization stage is determined; wherein the second feature point set includes the third feature point set and the fourth feature point set.

[0081] Specifically, the camera calibration phase may include an initial calibration phase and a calibration optimization phase, wherein the initial calibration phase may be used to determine the initial extrinsic parameter values ​​of the target camera, and the calibration optimization phase may be used to iteratively optimize the initial extrinsic parameter values ​​of the target camera to obtain the extrinsic parameter calibration result of the target camera.

[0082] In the camera calibration phase, the second feature point set can be determined based on the target calibration strategy determined in the data preparation phase. For example, the target calibration strategy may include a first strategy and a second strategy, wherein the first strategy may be a strategy to be adopted in the process of determining the initial extrinsic parameter value of the target camera in the initial calibration phase, and the second strategy may be a strategy to be adopted in the process of iteratively optimizing the initial extrinsic parameter value of the target camera in the calibration optimization phase.

[0083] The candidate feature point set types corresponding to the initial calibration stage may include the types of feature point sets that need to be provided in the initial calibration stage under each optional first strategy. In implementation, in the initial calibration stage, the target feature point set type corresponding to the initial calibration stage may be determined from the candidate feature point set types corresponding to the initial calibration stage based on the first strategy in the target calibration strategy. For example, the target feature point set type corresponding to the initial calibration stage may be determined based on the first strategy in the target calibration strategy and the predetermined correspondence between the first strategy and the feature point set type corresponding to the initial calibration stage. Thus, the third feature point set corresponding to the initial calibration stage may be determined based on the target data and the target feature point set type corresponding to the initial calibration stage. For example, the third feature point set may be selected from the first feature point set, and the third feature point set may also be determined based on the target image.

[0084] The candidate feature point set types corresponding to the calibration optimization stage may include the types of feature point sets that need to be provided in the calibration optimization stage under each optional second strategy. In implementation, in the calibration optimization stage, the target feature point set type corresponding to the calibration optimization stage may be determined from the candidate feature point set types corresponding to the calibration optimization stage based on the second strategy in the target calibration strategy. For example, the target feature point set type corresponding to the calibration optimization stage may be determined based on the second strategy in the target calibration strategy and the predetermined correspondence between the second strategy and the feature point set type corresponding to the calibration optimization stage. Thus, the fourth feature point set corresponding to the calibration optimization stage may be determined based on the target data and the target feature point set type corresponding to the calibration optimization stage. For example, the fourth feature point set may be selected from the first feature point set, and the fourth feature point set may also be determined based on the target image.

[0085] The second feature point set may include the third feature point set and the fourth feature point set at the same time. Thus, through the method of this embodiment, the calibration strategy of the target camera and the second feature point set can be adaptively updated under different road environments, thereby ensuring the accuracy of the calibration results of the target camera under different road environments, greatly improving the environmental adaptability of the camera calibration, and the camera calibration process does not need to be limited to a specific road environment, thereby effectively improving the success probability of the camera calibration.

[0086] In a feasible implementation, the candidate feature point set types corresponding to the initial calibration stage include a third sub-feature point set and a fourth sub-feature point set, the third sub-feature point set includes a vanishing point set and an edge point set, the edge point set includes edge points on a straight line segment; the fourth sub-feature point set includes the central inner points of the virtual grid.

[0087] Specifically, the candidate feature point set type corresponding to the initial calibration stage may include a third sub-feature point set, and the third sub-feature point set may include a vanishing point set and an edge point set. The vanishing point set may include multiple vanishing points. In implementation, the target data may also include straight line segments extracted by linear detection of the road environment image captured by the target camera. In the initial calibration stage, the vanishing points may be calculated based on the parallel straight line segments in the target data to form a vanishing point set in the third sub-feature point set. The edge point set may include edge points on the straight line segments. In implementation, the first sub-feature point set provided in the data preparation stage may be directly used as the vanishing point set in the third sub-feature point set. Thus, in the initial calibration stage, the third sub-feature point set can be determined quickly and effectively.

[0088] The candidate feature point set type corresponding to the initial calibration stage may also include a fourth sub-feature point set, and the fourth sub-feature point set may include the central inner point of the virtual grid. In implementation, in the initial calibration stage, the BEV (Bird's Eye View) area of ​​a predetermined area may be divided into grids of equal size (i.e., virtual grids), and the BEV area of ​​a predetermined area may include a BEV area of ​​a predetermined area centered at the center point of the rear axle of the vehicle. Considering that the obstacle corner points in the second sub-feature point set are all feature corner points matched between frames, for any frame target image in the continuous multi-frame target image, the corresponding obstacle corner points can be converted from the image coordinate system to the world coordinate system by a preset conversion coefficient according to the position of the obstacle corner points in the second sub-feature point set in the target image, and projected to the virtual grid according to the position coordinates in the world coordinate system, thereby obtaining the matched inter-frame grid feature corner points, and using the matched inter-frame grid feature corner points as the fourth sub-feature point set, so that the fourth sub-feature point set can be quickly and effectively determined in the initial calibration stage.

[0089] It can be understood that the third sub-feature point set and the fourth sub-feature point set can be used for different first strategies respectively.

[0090] In a feasible implementation, the candidate feature point set types corresponding to the calibration optimization stage include a fifth sub-feature point set and a sixth sub-feature point set, the fifth sub-feature point set includes obstacle corner points, and the sixth sub-feature point set includes intersection points of the bird's-eye view virtual marking line and the straight line segment.

[0091] Specifically, the candidate feature point set type corresponding to the calibration optimization stage may include a fifth sub-feature point set, and the fifth sub-feature point set may include obstacle corner points. In implementation, in the calibration optimization stage, the second sub-feature point set provided in the data preparation stage may be directly used as the fifth sub-feature point set. Thus, in the calibration optimization stage, the fifth sub-feature point set can be determined quickly and effectively.

[0092] The candidate feature point set type corresponding to the calibration optimization stage may also include a sixth sub-feature point set, which may include the intersection points of the bird's-eye view virtual marking line and the straight line segment. For example, at any time, the intersection points of the bird's-eye view virtual marking line and the straight line segment may be as follows: Figure 2 As shown, Figure 2 In the figure, Bev1, Bev2 and Bev3 are virtual marking lines of the bird's-eye view, and Lane1, Lane2, Lane3 and Lane4 are straight line segments extracted by linear detection.

[0093] In implementation, the target data may also include straight line segments extracted by linear detection of the road environment image captured by the target camera. In the calibration optimization stage, the intersection points of the straight line segments in the target data and the virtual marking lines of the bird's-eye view may be obtained to form the sixth sub-feature point set. Thus, in the calibration optimization stage, the sixth sub-feature point set can be determined quickly and effectively.

[0094] It can be understood that the fifth sub-feature point set and the sixth sub-feature point set can be used for different second strategies respectively.

[0095] In a feasible implementation manner, determining the extrinsic calibration result of the target camera based on the second feature point set includes:

[0096] In the initial calibration stage, one or more groups of initial extrinsic parameter values ​​of the target camera are determined based on the third feature point set;

[0097] In the calibration optimization stage, the initial extrinsic parameter value is iteratively optimized based on the fourth feature point set;

[0098] Based on the iterative optimization results of each group of the initial extrinsic parameter values, the extrinsic parameter calibration results of the target camera are determined.

[0099] Specifically, in the initial calibration stage, one or more sets of initial extrinsic parameter values ​​of the target camera can be determined by the third feature point set, and the initial extrinsic parameter values ​​can include the initial calibration value of the rotation matrix of the target camera. The third feature point set can include a third sub-feature point set and / or a fourth sub-feature point set.

[0100] In implementation, when the third feature point set only includes the third sub-feature point set, a set of initial extrinsic parameter values ​​can be determined based on the vanishing point set and the edge point set in the third sub-feature point set. For example, the initial calibration values ​​of two degrees of freedom in the rotation matrix of the target camera can be calculated based on the vanishing point set in the third sub-feature point set, and the initial calibration value of the remaining degree of freedom in the rotation matrix of the target camera can be calculated based on the edge point set in the third sub-feature point set. Among them, the three degrees of freedom in the rotation matrix of the target camera can be Pitch (pitch angle), Yaw (heading angle) and Roll (roll angle), respectively. Specifically, the remaining degree of freedom can be determined based on the type of the target camera. For example, if the target camera is a forward-looking camera, the remaining degree of freedom is Roll, that is, the initial calibration values ​​of Pitch and Yaw are calculated based on the vanishing point set in the third sub-feature point set, and the initial calibration value of Roll is calculated based on the edge point set in the third sub-feature point set.

[0101] When the third feature point set only includes the fourth sub-feature point set, a set of initial extrinsic parameter values ​​may be determined based on the fourth sub-feature point set. For example, the initial extrinsic parameter values ​​of the target camera may be calculated based on the matched inter-frame grid feature corner points in the fourth sub-feature point set.

[0102] In the case where the third feature point set includes both the third sub-feature point set and the fourth sub-feature point set, a set of initial extrinsic parameter values ​​can be determined based on the third sub-feature point set, and a set of initial extrinsic parameter values ​​can be determined based on the fourth sub-feature point set to obtain two sets of initial extrinsic parameter values. In addition, the two sets of initial extrinsic parameter values ​​can be weighted summed to obtain a set of initial extrinsic parameter values. In the process of weighted summing of the two sets of initial extrinsic parameter values, the weighting coefficient can be determined based on the image features in the road environment image. For example, the weighting coefficient can be determined based on the ratio of the number of straight line features and obstacle features in the road environment image, so that the weights of the initial calibration values ​​of the two calibration strategies can be adaptively updated according to the road environment. Therefore, the accuracy of the calibration results of the target camera can be guaranteed under different road environments through the method of this embodiment, which greatly improves the environmental adaptability of the camera calibration, and the camera calibration process does not need to be limited to a specific road environment, so that the success probability of the camera calibration can be effectively improved. At the same time, by performing iterative optimization based on high-precision initial extrinsic parameter values, compared with iterative optimization based on the initial design values ​​during the installation of the target camera, it is possible to effectively avoid falling into the local optimum during the iterative optimization process, thereby further improving the calibration accuracy of the target camera.

[0103] In the calibration optimization stage, the initial extrinsic parameter values ​​may also be iteratively optimized based on the fourth feature point set, wherein the fourth feature point set may include the fifth sub-feature point set and / or the sixth sub-feature point set.

[0104] In implementation, when the fourth feature point set only includes the fifth sub-feature point set, each group of initial extrinsic parameter values ​​can be iteratively optimized based on the fifth sub-feature point set to obtain iterative optimization results of each group of initial extrinsic parameter values. For example, when the target camera is a surround camera, for any iteration in the iterative optimization process, the corner points of the obstacles in the fifth sub-feature point set can be triangulated to obtain three-dimensional points, and based on the extrinsic parameter values ​​of the target camera in the current iteration, the three-dimensional points of the matching point pairs are reprojected to the reference frame and the current frame respectively, so as to determine the reprojection error according to the reprojection result. If the reprojection error is greater than the predetermined error value, the extrinsic parameter values ​​of the target camera are optimized based on the optimization results of the extrinsic parameter values ​​of the target camera in the previous iteration and the next iteration is performed. If the reprojection error is less than or equal to the predetermined error value, the iterative optimization ends. At this time, the optical center parallax angle of the two cameras calculated when generating the three-dimensional point can be saved, and at the same time, the coordinates of the three-dimensional point in the world coordinate system can be stored to facilitate the review of the algorithm problems in the dynamic calibration process. The smaller the parallax angle of the optical center of the two cameras, the greater the reprojection error. Therefore, the positive and negative depth values ​​of the three-dimensional points and the size of the parallax angle of the optical center of the two cameras can effectively avoid the three-dimensional points obtained by triangulation measurement having infinite one-dimensional coordinates. Therefore, through iterative optimization, the reprojection error of the three-dimensional points restored by the pixels in the target image can be kept within a reasonable range, thereby ensuring the accuracy of the iterative optimization results of the initial external parameter values ​​of the target camera.

[0105] In the case where the fourth feature point set only includes the sixth sub-feature point set, each group of initial extrinsic parameter values ​​can be iteratively optimized based on the sixth sub-feature point set to obtain iterative optimization results for each group of initial extrinsic parameter values. For example, when the target camera is a surround camera, for any iteration in the iterative optimization process, a seam line can be constructed based on the optimization results of the extrinsic parameter values ​​of the target camera in the previous iteration and the intersection points of the virtual marking line of the bird's-eye view and the straight line segment in the sixth sub-feature point set, and whether the iterative optimization end condition is met is determined based on the constructed seam line. The iterative optimization end condition may include image geometric alignment, and may also include that the function value of the energy function of the overlapping area is less than or equal to a predetermined function value. For example, when the images are geometrically aligned and the function value of the energy function of the overlapping area is less than the predetermined function value, it is determined that the iterative optimization end condition is met. When the images are not geometrically aligned and / or the function value of the energy function of the overlapping area is greater than the predetermined function value, it is determined that the iterative optimization end condition is not met. If the iterative optimization end condition is not met, the extrinsic parameter values ​​of the target camera are optimized based on the optimization results of the extrinsic parameter values ​​of the target camera in the previous iteration and the next iteration is performed. If the iterative optimization end condition is met, the iterative optimization ends. Thus, according to the iterative optimization results of the extrinsic parameter values ​​of the target camera, the images captured by different cameras in the surround camera can be seamlessly spliced.

[0106] When the fourth feature point set includes both the fifth sub-feature point set and the sixth sub-feature point set, the initial extrinsic parameter values ​​of the target camera can be iteratively optimized based on the fifth sub-feature point set to obtain a first iterative optimization result of the extrinsic parameter values ​​of the target camera, and the first iterative optimization result can be iteratively optimized based on the sixth sub-feature point set to obtain a final iterative optimization result of the target camera.

[0107] Among them, when a group of initial extrinsic parameter values ​​is included, the iterative optimization result of the initial extrinsic parameter values ​​can be used as the extrinsic parameter calibration result of the target camera. When multiple groups of initial extrinsic parameter values ​​are included, the iterative optimization result with the best effect (such as the smallest reprojection error or the best stitching) among the iterative optimization results of the multiple groups of initial extrinsic parameter values ​​can be used as the extrinsic parameter calibration result of the target camera, thereby effectively ensuring the accuracy of the extrinsic parameter calibration result of the target camera.

[0108] Exemplary Devices

[0109] In an exemplary embodiment of the present specification, a camera dynamic calibration device is also provided, such as Figure 3 As shown, including:

[0110] The first processing module 301 is used to obtain target data required to be provided in the data preparation stage based on the road environment image collected by the target camera in the vehicle during the vehicle driving process, wherein the target data includes a first feature point set;

[0111] The second processing module 302 is used to obtain a second feature point set required for use in the camera calibration stage based on the target data during the camera calibration stage, and determine an extrinsic calibration result of the target camera based on the second feature point set; the second feature point set includes the first feature point set.

[0112] In a feasible implementation manner, the first processing module 301 is specifically configured to:

[0113] Determining a target calibration strategy for the target camera based on image features in the road environment image;

[0114] Based on the target calibration strategy, determining the target feature point set type corresponding to the data preparation stage from the candidate feature point set types corresponding to the data preparation stage;

[0115] Based on the road environment image and the target feature point set type corresponding to the data preparation stage, the first feature point set is acquired.

[0116] In a feasible implementation, the candidate feature point set type corresponding to the data preparation stage includes a first sub-feature point set and a second sub-feature point set, the first sub-feature point set includes edge points on a straight line segment, and the second sub-feature point set includes obstacle corner points.

[0117] In a feasible implementation manner, the target data further includes a target image; and the first processing module 301 is specifically configured to:

[0118] Based on the required image features in the camera calibration phase, the road environment image is segmented to obtain the target image, and the required image features are used to determine a feature point set in the second feature point set that is outside the first feature point set.

[0119] In a feasible implementation manner, the camera calibration phase includes an initial calibration phase and a calibration optimization phase; the second processing module 302 is specifically used for:

[0120] Based on the target calibration strategy, determining the target feature point set type corresponding to the initial calibration stage from the candidate feature point set types corresponding to the initial calibration stage, and, based on the target calibration strategy, determining the target feature point set type corresponding to the calibration optimization stage from the candidate feature point set types corresponding to the calibration optimization stage;

[0121] Based on the target data and the target feature point set type corresponding to the initial calibration stage, the third feature point set corresponding to the initial calibration stage is determined, and based on the target data and the target feature point set type corresponding to the calibration optimization stage, the fourth feature point set corresponding to the calibration optimization stage is determined; wherein the second feature point set includes the third feature point set and the fourth feature point set.

[0122] In a feasible implementation, the candidate feature point set types corresponding to the initial calibration stage include a third sub-feature point set and a fourth sub-feature point set, the third sub-feature point set includes a vanishing point set and an edge point set, the edge point set includes edge points on a straight line segment; the fourth sub-feature point set includes the central inner points of the virtual grid.

[0123] In a feasible implementation, the candidate feature point set types corresponding to the calibration optimization stage include a fifth sub-feature point set and a sixth sub-feature point set, the fifth sub-feature point set includes obstacle corner points, and the sixth sub-feature point set includes intersection points of the bird's-eye view virtual marking line and the straight line segment.

[0124] In a feasible implementation manner, the second processing module 302 is specifically configured to:

[0125] In the initial calibration stage, one or more groups of initial extrinsic parameter values ​​of the target camera are determined based on the third feature point set;

[0126] In the calibration optimization stage, the initial extrinsic parameter value is iteratively optimized based on the fourth feature point set;

[0127] Based on the iterative optimization results of each group of the initial extrinsic parameter values, the extrinsic parameter calibration results of the target camera are determined.

[0128] The camera dynamic calibration device provided in this embodiment belongs to the same application concept as the camera dynamic calibration method provided in the above embodiments of this application, and can execute the camera dynamic calibration method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the camera dynamic calibration method. For technical details not fully described in this embodiment, please refer to the specific processing content of the camera dynamic calibration method provided in the above embodiments of this application, and will not be repeated here.

[0129] Exemplary Devices

[0130] In an exemplary embodiment of the present specification, an electronic device is also provided, the electronic device comprising at least one processor and at least one memory, the memory storing a computer program, and the computer program, when executed by the processor, implements the camera dynamic calibration method as described in any of the above embodiments.

[0131] Example Vehicles

[0132] In an exemplary embodiment of the present specification, a vehicle is also provided, comprising a camera and a camera dynamic calibration device as described in any one of the above embodiments.

[0133] Exemplary computer program products and storage media

[0134] In addition to the above-mentioned methods and devices, the camera dynamic calibration method provided in the embodiments of this specification may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the camera dynamic calibration method according to various embodiments of this specification described in the above “Exemplary Method” section of this specification.

[0135] The computer program product can be written in any combination of one or more programming languages ​​to write program codes for executing the operations of the embodiments of this specification, and the programming languages ​​include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as "C" language or similar programming languages.

[0136] In addition, an embodiment of the present specification also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the camera dynamic calibration method according to various embodiments of the present specification described in the above "Exemplary Method" section of the present specification.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments only express several implementation methods of this specification, and the descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the solutions provided by the embodiments of this specification. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this specification, which all belong to the protection scope of this specification. Therefore, the protection scope of the patent of this specification shall be based on the attached claims.

Claims

1. A camera dynamic calibration method, characterized in that: include: In the data preparation stage, based on the road environment image captured by the target camera in the vehicle during the vehicle driving process, the target data required to be provided in the data preparation stage is obtained, and the target data includes a first feature point set; In the camera calibration stage, based on the target data, a second feature point set required for the camera calibration stage is obtained, and based on the second feature point set, an extrinsic calibration result of the target camera is determined; the second feature point set includes the first feature point set.

2. The method according to claim 1, characterized in that The step of obtaining target data required to be provided in the data preparation stage based on the road environment image captured by the target camera in the vehicle during the vehicle driving process includes: Determining a target calibration strategy for the target camera based on image features in the road environment image; Based on the target calibration strategy, determining the target feature point set type corresponding to the data preparation stage from the candidate feature point set types corresponding to the data preparation stage; Based on the road environment image and the target feature point set type corresponding to the data preparation stage, the first feature point set is acquired.

3. The method according to claim 2, characterized in that The candidate feature point set types corresponding to the data preparation stage include a first sub-feature point set and a second sub-feature point set, the first sub-feature point set includes edge points on a straight line segment, and the second sub-feature point set includes obstacle corner points.

4. The method according to claim 1, characterized in that: The target data also includes a target image; The step of obtaining target data required to be provided in the data preparation stage based on the road environment image captured by the target camera in the vehicle during the vehicle driving process includes: Based on the required image features in the camera calibration phase, the road environment image is segmented to obtain the target image, and the required image features are used to determine a feature point set in the second feature point set that is outside the first feature point set.

5. The method according to claim 2, characterized in that: The camera calibration phase includes an initial calibration phase and a calibration optimization phase; The step of obtaining a second feature point set required for use in the camera calibration phase based on the target data includes: Based on the target calibration strategy, determining the target feature point set type corresponding to the initial calibration stage from the candidate feature point set types corresponding to the initial calibration stage, and, based on the target calibration strategy, determining the target feature point set type corresponding to the calibration optimization stage from the candidate feature point set types corresponding to the calibration optimization stage; Based on the target data and the target feature point set type corresponding to the initial calibration stage, the third feature point set corresponding to the initial calibration stage is determined, and based on the target data and the target feature point set type corresponding to the calibration optimization stage, the fourth feature point set corresponding to the calibration optimization stage is determined; wherein the second feature point set includes the third feature point set and the fourth feature point set.

6. The method according to claim 5, characterized in that The candidate feature point set types corresponding to the initial calibration stage include a third sub-feature point set and a fourth sub-feature point set, the third sub-feature point set includes a vanishing point set and an edge point set, the edge point set includes edge points on a straight line segment; the fourth sub-feature point set includes the central inner points of the virtual grid.

7. The method according to claim 5, characterized in that The candidate feature point set types corresponding to the calibration optimization stage include a fifth sub-feature point set and a sixth sub-feature point set, the fifth sub-feature point set includes obstacle corner points, and the sixth sub-feature point set includes intersection points of a bird's-eye view virtual marking line and a straight line segment.

8. The method according to claim 5, characterized in that The determining the extrinsic calibration result of the target camera based on the second feature point set includes: In the initial calibration stage, one or more groups of initial extrinsic parameter values ​​of the target camera are determined based on the third feature point set; In the calibration optimization stage, the initial extrinsic parameter value is iteratively optimized based on the fourth feature point set; Based on the iterative optimization results of each group of the initial extrinsic parameter values, the extrinsic parameter calibration results of the target camera are determined.

9. A camera dynamic calibration device, characterized in that: include: A first processing module is used to obtain target data required to be provided in the data preparation stage based on a road environment image captured by a target camera in the vehicle during the vehicle's driving process, wherein the target data includes a first feature point set; The second processing module is used to obtain a second feature point set required for use in the camera calibration stage based on the target data during the camera calibration stage, and determine an external parameter calibration result of the target camera based on the second feature point set; the second feature point set includes the first feature point set.

10. A vehicle, characterized in that: It comprises a camera and the camera dynamic calibration device as claimed in claim 9.

Images