Camera calibration method, device, electronic device and readable storage medium

Abstract

The present disclosure provides a camera calibration method, device, electronic device and readable storage medium, which relate to technical fields such as autonomous driving and intelligent transportation. The camera calibration method includes: obtaining the camera intrinsic parameters of the camera to be calibrated; obtaining a detection image sequence based on at least one driving image taken by the camera to be calibrated; performing feature extraction on at least one driving image in the detection image sequence to obtain a target feature set; performing extrinsic calibration on the camera to be calibrated based on the camera height of the camera to be calibrated, the camera intrinsic parameters and the target feature set to obtain the camera extrinsic parameters of the camera to be calibrated. The present disclosure can simplify the steps of camera extrinsic calibration, expand the scope of application of camera extrinsic calibration, and improve the efficiency and accuracy of camera extrinsic calibration.

Description

Technical Field

[0001] The present disclosure relates to the field of technical data processing, and in particular to the technical fields of autonomous driving, intelligent transportation, etc. A camera calibration method, apparatus, electronic device, and readable storage medium are provided. Background Art

[0002] With the continuous development of the internet, crowdsourcing data collection is gaining increasing attention. For example, in autonomous driving scenarios, crowdsourcing is used to collect road image data. However, due to the low requirements for the installation of acquisition devices such as cameras, the cameras can be installed in arbitrary locations, making it difficult to accurately calibrate the cameras used in crowdsourcing. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a camera calibration method is provided, comprising: obtaining camera intrinsic parameters of a camera to be calibrated; obtaining a detection image sequence based on at least one driving image captured by the camera to be calibrated; performing feature extraction on at least one driving image in the detection image sequence to obtain a target feature set; and performing extrinsic calibration on the camera to be calibrated based on the camera height of the camera to be calibrated, the camera intrinsic parameters, and the target feature set to obtain camera extrinsic parameters of the camera to be calibrated.

[0004] According to a second aspect of the present disclosure, a camera calibration device is provided, comprising: an acquisition unit for acquiring camera intrinsic parameters of a camera to be calibrated; a processing unit for obtaining a detection image sequence based on at least one driving image taken by the camera to be calibrated; an extraction unit for performing feature extraction on at least one driving image in the detection image sequence to obtain a target feature set; and a calibration unit for performing extrinsic parameter calibration on the camera to be calibrated based on the camera height of the camera to be calibrated, the camera intrinsic parameters, and the target feature set to obtain the camera extrinsic parameters of the camera to be calibrated.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0008] It can be seen from the above technical solutions that the present invention completes the extrinsic calibration of the camera to be calibrated based on the acquired camera intrinsic parameters, the target feature set extracted from the retrieved image sequence, and the camera height of the camera to be calibrated. This can simplify the steps of camera extrinsic calibration, expand the scope of application of camera extrinsic calibration, and improve the efficiency and accuracy of camera extrinsic calibration.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0011] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0012] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0013] Figure 3 is a block diagram of an electronic device used to implement the camera calibration method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, and various details of the embodiments of the present disclosure are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and mechanisms are omitted in the following description.

[0015] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the camera calibration method of this embodiment specifically includes the following steps:

[0016] S101: Obtain the camera intrinsic parameters of the camera to be calibrated.

[0017] The camera intrinsic parameters obtained by executing S101 in this embodiment are specifically parameters related to the characteristics of the camera to be calibrated, such as the focal length, pixel size and other parameters of the camera to be calibrated.

[0018] In this embodiment, when executing S101 to obtain the camera intrinsic parameters of the camera to be calibrated, the parameters input by the input end can be used as the camera intrinsic parameters of the camera to be calibrated; or the parameters corresponding to the attribute information of the camera to be calibrated (such as the camera model, camera name, and other attribute information) obtained by the input end from the network can be used as the camera intrinsic parameters of the camera to be calibrated.

[0019] In this embodiment, when executing S101 to obtain the camera intrinsic parameters of the camera to be calibrated, the following method can also be used: obtaining at least one image taken by the camera to be calibrated; based on the at least one image obtained, using a preset algorithm to calibrate the intrinsic parameters of the camera to be calibrated, to obtain the camera intrinsic parameters of the camera to be calibrated. The preset algorithm used in this embodiment can be the Zhang Zhengyou checkerboard calibration method.

[0020] That is, this embodiment can obtain existing parameters as camera intrinsic parameters, and can also obtain camera intrinsic parameters through real-time processing, thereby improving the flexibility in obtaining camera intrinsic parameters.

[0021] S102: Obtain a detection image sequence according to at least one driving image captured by the camera to be calibrated.

[0022] After executing S101 to obtain the camera intrinsic parameters of the camera to be calibrated, this embodiment executes S102 to obtain a detection image sequence based on at least one driving image captured by the camera to be calibrated. The detection image sequence obtained by executing S102 in this embodiment includes at least one driving image.

[0023] It can be understood that the driving image used when executing S102 in this embodiment is an image located in front of the vehicle captured by a camera to be calibrated installed on the vehicle while the vehicle is driving.

[0024] Specifically, when executing S102 and obtaining a detection image sequence based on at least one driving image taken by the camera to be calibrated, this embodiment may adopt an optional implementation method as follows: according to the timestamp, align the position information of the camera to be calibrated when shooting with the captured driving image to obtain a first image sequence. Each driving image in the first image sequence obtained in this embodiment has corresponding position information; and the driving images in the first image sequence are filtered according to the position information to obtain a detection image sequence.

[0025] Among them, when executing S102, this embodiment aligns the position information of the camera to be calibrated when shooting with the captured driving image according to the timestamp, a correspondence between the position information with the same timestamp and the driving image can be established, thereby avoiding the problem of inconsistency between the position information and the driving image caused by the inconsistency between the acquisition frequency of the positioning system on the vehicle and the camera to be calibrated.

[0026] When executing S102 in this embodiment to filter the driving images in the first image sequence according to the position information, at least one driving image whose position information is within a preset distance threshold can be used as a detection image sequence based on the position information of each driving image in the first image sequence. That is, the detection image sequence obtained by executing S102 in this embodiment is a loop image sequence when the vehicle is driving in a certain area.

[0027] That is to say, this embodiment first obtains a first image sequence containing driving images and their position information based on the timestamp, and then filters the driving images in the first image sequence according to the position information to obtain a detection image sequence. On the one hand, it can ensure that each driving image in the detection image sequence corresponds to the position information, and on the other hand, it can improve the accuracy of the images contained in the detection image sequence.

[0028] In addition, when executing S102 to obtain a detection image sequence based on at least one driving image taken by the camera to be calibrated, this embodiment does not need to perform an operation to align the position information with the driving image according to the timestamp, and can directly obtain the detection image sequence based on at least one driving image taken by the camera to be calibrated.

[0029] S103 : performing feature extraction on at least one driving image in the detection image sequence to obtain a target feature set.

[0030] After executing S102 to obtain the detection image sequence, this embodiment executes S103 to perform feature extraction on at least one driving image in the obtained detection image sequence, thereby obtaining a target feature set corresponding to the detection image sequence.

[0031] In order to improve the accuracy of the extracted target feature set, when executing S103, this embodiment performs feature extraction on at least one driving image in the image sequence to be detected to obtain the target feature set, and an optional implementation method that can be adopted is: performing preset processing on at least one driving image in the image sequence to be detected. The preset processing in this embodiment includes de-distortion processing, Gaussian filtering processing, image enhancement processing, etc.; performing feature extraction on at least one driving image that has undergone preset processing to obtain the target feature set.

[0032] Specifically, when executing S103 to extract features from at least one driving image in the detection image sequence to obtain a target feature set, this embodiment may employ the following optional implementation methods: extracting line features from each driving image in the detection image sequence. This embodiment may employ the LSD (Line Segment Detector) algorithm to extract line features from the driving images; fitting the line features of each driving image in the detection image sequence to obtain a straight line fitting result corresponding to each driving image. This embodiment may employ the RANSAC (Random Sample Consensus) algorithm to fit the line features of the driving images; and constructing a straight line set based on the straight line fitting result corresponding to each driving image, and using the constructed straight line set as the target feature set.

[0033] In this embodiment, when executing S103 to construct a line set according to the line fitting results corresponding to each driving image, the line fitting results selected from each driving image and ranked in the top N positions in length can be used to construct the line set, where N is a positive integer greater than or equal to 1.

[0034] That is to say, this embodiment can construct a target feature set based on the line features extracted from each driving image in the detection image sequence, and then use the straight line fitting results corresponding to each driving image contained in the target feature set to complete the calibration of the camera to be calibrated in the subsequent process.

[0035] In this embodiment, the straight line fitting result in the target feature set obtained by executing S103 is used to calibrate the three external parameters of the camera to be calibrated, namely the yaw angle, pitch angle and roll angle.

[0036] In some application scenarios, such as crowdsourced data collection, the position of the camera mounted on the vehicle for image acquisition may be uncertain (e.g., different heights, front and back positions, etc.). Therefore, it is necessary to calibrate the offset parameters of the camera to be calibrated relative to the vehicle positioning system. This offset parameter calibration must be performed after the yaw, pitch, and roll angles have been calibrated.

[0037] Therefore, when executing S103 and using the constructed straight line set as the target feature set, this embodiment may also include the following contents: extracting point features of each driving image in the detection image sequence. This embodiment may use the ORB (Oriented FAST and Rotated BRIEF) algorithm to extract point features in the driving images; performing feature matching on the point features of each driving image in the detection image sequence; constructing a point pair set based on the matching results between the point features; and using the constructed point pair set and the straight line set as the target feature set.

[0038] Among them, when executing S103 to construct a point pair set based on the matching results between point features, this embodiment can filter point features based on the offset distance between different driving images of the same point feature, so that the filtered same point features located in different driving images are respectively formed into point pairs to obtain a point pair set, in which one point pair contains two point features.

[0039] When executing S103 to filter point features based on the offset distances of the same point feature between different driving images, this embodiment can first obtain the offset distances of the point feature between different images, then obtain the division result between the smallest offset distance and the second smallest offset distance, and finally, after determining that the obtained division result is less than or equal to a preset threshold, retain the point feature, otherwise discard the point feature.

[0040] That is to say, the target feature set obtained in this embodiment will include both a straight line set and a point pair set, so that after the calibration of the yaw angle, pitch angle and roll angle of the camera to be calibrated is completed according to the straight line set, the calibration of the offset parameters of the camera to be calibrated relative to the vehicle positioning system is completed according to the point pair set.

[0041] S104 , performing extrinsic calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera internal parameters, and the target feature set to obtain camera extrinsics of the camera to be calibrated.

[0042] In this embodiment, the camera extrinsic parameters of the camera to be calibrated obtained by executing S104 are specifically parameters of the camera to be calibrated in the world coordinate system, such as the pose of the camera to be calibrated, the position of the camera to be calibrated, etc.

[0043] After executing S103 to obtain the target feature set, this embodiment executes S104 to perform extrinsic calibration on the camera to be calibrated according to the height of the camera to be calibrated, the camera intrinsic parameters of the camera to be calibrated, and the target feature set of the camera to be calibrated, thereby obtaining the camera extrinsic parameters of the camera to be calibrated.

[0044] If the target feature set obtained by executing S103 in this embodiment only includes a straight line set, executing S104 in this embodiment will obtain camera extrinsics including yaw angle, pitch angle, and roll angle; if the target feature set obtained by executing S103 in this embodiment includes both a straight line set and a point pair set, executing S104 in this embodiment will obtain yaw angle, pitch angle, roll angle, and offset parameters x0 (offset distance along the x direction) and y0 (offset distance along the y direction) of the camera to be calibrated relative to the vehicle positioning system.

[0045] Specifically, when executing S104, this embodiment performs extrinsic calibration on the camera to be calibrated according to the camera height and the target feature set of the camera to be calibrated, an optional implementation method that can be adopted is: with the optical center of the camera to be calibrated as the origin, a camera imaging model is constructed according to the first preset coordinate system, the camera height and the camera intrinsic parameters. The first preset coordinate system in this embodiment can be, for example, the carrier coordinate system of the RFU; using the constructed camera imaging model, the image in the detection image sequence and the set of straight lines corresponding to the image in the target feature set are projected onto the ground to generate a bird's-eye view; when it is determined that the straight line in the generated bird's-eye view does not meet the first constraint condition, the yaw angle, pitch angle and roll angle in the camera imaging model are adjusted, and then the step of generating the bird's-eye view is executed until it is determined that the straight line in the generated bird's-eye view meets the first constraint condition, and the current yaw angle, pitch angle and roll angle of the camera imaging model are obtained, which are used as the target yaw angle, target pitch angle and target roll angle, respectively; and the camera extrinsics of the camera to be calibrated are obtained according to the obtained target yaw angle, target pitch angle and target roll angle.

[0046] The first constraint used in this embodiment when executing S104 may include at least one of a constraint that lane lines in the bird's-eye view remain parallel and a constraint that zebra crossings in the bird's-eye view remain horizontal.

[0047] The camera imaging model corresponding to the first preset coordinate system constructed by executing S104 in this embodiment can be expressed as follows:

[0048]

[0049] In the formula: (u, v) represents the column coordinates and row coordinates of a pixel in the image; K represents the matrix corresponding to the camera intrinsic parameter; yaw represents the yaw angle of the camera to be calibrated; roll represents the roll angle of the camera to be calibrated; pitch represents the pitch angle of the camera to be calibrated; x w represents the x-coordinate of the pixel point (u, v) in the image in the first preset coordinate system; y w represents the coordinate of the pixel point (u, v) in the image in the y direction in the first preset coordinate system; H represents the camera height; h is a preset parameter.

[0050] That is to say, this embodiment generates a bird's-eye view of the corresponding image by constructing a camera imaging model corresponding to the first preset coordinate system, and then continuously adjusts the values ​​of the yaw angle, pitch angle and roll angle in the camera imaging model according to whether the straight line in the generated bird's-eye view meets the preset constraints, until the straight line in the bird's-eye view generated by the camera imaging model meets the preset constraints, and obtains the values ​​of the yaw angle, pitch angle and roll angle in the current camera imaging model as the camera extrinsic parameters. Therefore, it is possible to complete the automatic calibration of the camera extrinsic parameters in the case that the camera to be calibrated is not installed properly in the crowdsourcing scenario.

[0051] If the target feature set obtained by executing S103 in this embodiment includes a point pair set, then when executing S104 in this embodiment, the camera extrinsic parameters of the camera to be calibrated are obtained according to the obtained target yaw angle, target pitch angle, and target roll angle. An optional implementation method that can be adopted is: according to the point pair set in the target feature set, two target images corresponding to a target point pair in the point pair set are obtained from the detection image sequence; according to the obtained target yaw angle, target pitch angle, and target roll angle, the two obtained target images are respectively projected onto the ground using a camera imaging model to generate two bird's-eye views; according to the position information of the two target images and the offset between the point features of the corresponding target point pairs in the two bird's-eye views, the offset parameters of the camera to be calibrated relative to the vehicle positioning system are obtained; and the obtained target yaw angle, target pitch angle, target roll angle, and offset parameters are used as the camera extrinsic parameters of the camera to be calibrated.

[0052] Among them, when executing S104, this embodiment obtains the offset parameters of the camera to be calibrated relative to the vehicle positioning system based on the position information of the two target images and the point features between the point features of the corresponding target point pairs in the two bird's-eye views. First, the position information deviation can be obtained based on the position information of the two target images, and then the offset distance between the point features of the corresponding target point pairs in the two bird's-eye views can be obtained. Finally, a distance constraint is performed based on the obtained position information deviation and the offset distance to obtain the offset parameters of the camera to be calibrated relative to the vehicle positioning system.

[0053] In addition, when executing S104 to obtain the offset distance between the point features of corresponding target point pairs in the two bird's-eye views, this embodiment also maps the two obtained bird's-eye views to a second preset coordinate system (for example, a UTM coordinate system), and then obtains the offset distance based on the mapping results of the bird's-eye views.

[0054] In this embodiment, the offset parameters obtained by executing S104 may be two, one is an offset parameter along the x direction, and the other is an offset parameter along the y direction; if the x direction is the front-back direction, the y direction is the left-right direction; if the x direction is the left-right direction, the y direction is the front-back direction.

[0055] That is to say, after obtaining the external parameters of the target yaw angle, target pitch angle and target roll angle, this embodiment further obtains the offset parameters of the camera to be calibrated relative to the vehicle positioning system based on the obtained external parameters, and then uses the obtained target yaw angle, target pitch angle, target roll angle and offset parameters as the camera extrinsic parameters of the camera to be calibrated. While improving the accuracy of the obtained camera extrinsic parameters, it can also improve the diversity of the camera extrinsic parameters.

[0056] After executing S104 to obtain the camera extrinsic parameters of the camera to be calibrated, this embodiment can construct a camera imaging model based on the obtained camera extrinsic parameters, and then use the constructed camera imaging model to generate a bird's-eye view of the image taken by the camera to be calibrated. After the generated bird's-eye view is processed with offset parameters, the bird's-eye view is spliced ​​according to the vehicle's driving trajectory to obtain a high-definition and accurate road image.

[0057] This embodiment can also identify and update landmark information on the road (such as lane lines, guide arrows, speed reduction markings, etc.) based on the generated road image.

[0058] Figure 2 Schematic diagram of the second embodiment of the present disclosure. Figure 2 As shown, the camera calibration device 200 of this embodiment includes:

[0059] The acquisition unit 201 is used to acquire the camera intrinsic parameters of the camera to be calibrated.

[0060] The camera intrinsic parameters acquired by the acquisition unit 201 are specifically parameters related to the characteristics of the camera to be calibrated, such as the focal length, pixel size and other parameters of the camera to be calibrated.

[0061] When acquiring the camera intrinsic parameters of the camera to be calibrated, the acquisition unit 201 can use the parameters input by the input end as the camera intrinsic parameters of the camera to be calibrated; or it can use the parameters corresponding to the attribute information of the camera to be calibrated (such as the camera model, camera name, and other attribute information) obtained from the network by the input end as the camera intrinsic parameters of the camera to be calibrated.

[0062] When acquiring the camera intrinsic parameters of the camera to be calibrated, the acquisition unit 201 may also adopt the following method: acquiring at least one image taken by the camera to be calibrated; and performing intrinsic parameter calibration on the camera to be calibrated based on the at least one image acquired, using a preset algorithm to obtain the camera intrinsic parameters of the camera to be calibrated. The preset algorithm used in this embodiment may be the Zhang Zhengyou checkerboard calibration method.

[0063] That is, the acquisition unit 201 may acquire existing parameters as camera intrinsic parameters, or may acquire camera intrinsic parameters by real-time processing, thereby improving flexibility in acquiring camera intrinsic parameters.

[0064] The processing unit 202 is configured to obtain a detection image sequence based on at least one driving image captured by the camera to be calibrated.

[0065] In this embodiment, after the acquisition unit 201 acquires the camera intrinsic parameters of the camera to be calibrated, the processing unit 202 obtains a detection image sequence based on at least one driving image captured by the camera to be calibrated. The detection image sequence obtained by the processing unit 202 includes at least one driving image.

[0066] Specifically, when the processing unit 202 obtains a detection image sequence based on at least one driving image taken by the camera to be calibrated, the optional implementation method that can be adopted is: according to the timestamp, the position information of the camera to be calibrated when shooting is aligned with the captured driving image to obtain a first image sequence; according to the position information, the driving images in the first image sequence are filtered to obtain a detection image sequence.

[0067] Among them, when the processing unit 202 aligns the position information of the camera to be calibrated when shooting with the captured driving image according to the timestamp, it can establish a correspondence between the position information with the same timestamp and the driving image, thereby avoiding the problem of inconsistency between the position information and the driving image caused by the inconsistency between the acquisition frequency of the positioning system on the vehicle and the camera to be calibrated.

[0068] When the processing unit 202 filters the driving images in the first image sequence according to the position information, it can select at least one driving image whose position information is within a preset distance threshold as a detection image sequence based on the position information of each driving image in the first image sequence. That is, the detection image sequence obtained by executing S102 in this embodiment is a loop image sequence when the vehicle is driving in a certain area.

[0069] That is to say, the processing unit 202 first obtains a first image sequence containing driving images and their position information based on the timestamp, and then filters the driving images in the first image sequence according to the position information to obtain a detection image sequence. On the one hand, it can ensure that each driving image in the detection image sequence corresponds to the position information, and on the other hand, it can improve the accuracy of the images contained in the detection image sequence.

[0070] In addition, when the processing unit 202 obtains the detection image sequence based on at least one driving image taken by the camera to be calibrated, it is not necessary to perform the operation of aligning the position information with the driving image according to the timestamp, and can directly obtain the detection image sequence based on at least one driving image taken by the camera to be calibrated.

[0071] The extraction unit 203 is configured to perform feature extraction on at least one driving image in the detection image sequence to obtain a target feature set.

[0072] In this embodiment, after the processing unit 202 obtains the detection image sequence, the extraction unit 203 performs feature extraction on at least one driving image in the obtained detection image sequence, thereby obtaining a target feature set corresponding to the detection image sequence.

[0073] In order to improve the accuracy of the extracted target feature set, when the extraction unit 203 performs feature extraction on at least one driving image in the image sequence to be detected to obtain the target feature set, an optional implementation method that can be adopted is: performing preset processing on at least one driving image in the image sequence to be detected; performing feature extraction on at least one driving image that has undergone preset processing to obtain the target feature set.

[0074] Specifically, when the extraction unit 203 performs feature extraction on at least one driving image in the detection image sequence to obtain a target feature set, the optional implementation method that can be adopted is: extracting the line features of each driving image in the detection image sequence; fitting the line features of each driving image in the detection image sequence to obtain a straight line fitting result corresponding to each driving image; constructing a straight line set according to the straight line fitting result corresponding to each driving image, and using the constructed straight line set as the target feature set.

[0075] When constructing a line set based on the line fitting results corresponding to each driving image, the extraction unit 203 may construct the line set using the line fitting results of the first N digits in length selected from each driving image, where N is a positive integer greater than or equal to 1.

[0076] That is to say, the extraction unit 203 can construct a target feature set based on the line features extracted from each driving image in the detection image sequence, and then use the straight line fitting results corresponding to each driving image contained in the target feature set to complete the calibration of the camera to be calibrated in the subsequent process.

[0077] The straight line fitting result in the target feature set obtained by the extraction unit 203 is used to calibrate the three external parameters of the camera to be calibrated, namely the yaw angle, pitch angle and roll angle.

[0078] In some application scenarios, such as crowdsourced data collection, the position of the camera mounted on the vehicle for image acquisition may be uncertain (e.g., different heights, front and back positions, etc.). Therefore, it is necessary to calibrate the offset parameters of the camera to be calibrated relative to the vehicle positioning system. This offset parameter calibration must be performed after the yaw, pitch, and roll angles have been calibrated.

[0079] Therefore, when the extraction unit 203 uses the constructed straight line set as the target feature set, it can also include the following contents: extracting point features of each driving image in the detection image sequence; performing feature matching on the point features of each driving image in the detection image sequence; constructing a point pair set based on the matching results between the point features; and using the constructed point pair set and the straight line set as the target feature set.

[0080] Among them, when the extraction unit 203 constructs a point pair set based on the matching results between point features, the point features can be filtered according to the offset distance between different driving images of the same point feature, so that the filtered same point features located in different driving images are respectively formed into point pairs to obtain a point pair set, in which one point pair contains two point features.

[0081] When the extraction unit 203 filters point features based on the offset distance between different driving images of the same point feature, it can first obtain the offset distance of the point feature between different images, and then obtain the division result between the smallest offset distance and the second smallest offset distance. Finally, after determining that the obtained division result is less than or equal to a preset threshold, the point feature is retained, otherwise the point feature is discarded.

[0082] That is to say, the target feature set obtained by the extraction unit 203 will include both a straight line set and a point pair set, so that after the calibration of the yaw angle, pitch angle and roll angle of the camera to be calibrated is completed according to the straight line set, the calibration of the offset parameters of the camera to be calibrated relative to the vehicle positioning system is completed according to the point pair set.

[0083] The calibration unit 204 is configured to perform extrinsic calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera intrinsic parameters and the target feature set, to obtain the camera extrinsic parameters of the camera to be calibrated.

[0084] The camera extrinsic parameters of the camera to be calibrated obtained by the calibration unit 204 are specifically parameters of the camera to be calibrated in the world coordinate system, such as the pose of the camera to be calibrated, the position of the camera to be calibrated, etc.

[0085] In this embodiment, after the processing unit 203 obtains the target feature set, the calibration unit 204 performs extrinsic calibration on the camera to be calibrated according to the height of the camera to be calibrated, the intrinsic parameters of the camera to be calibrated, and the target feature set of the camera to be calibrated, thereby obtaining the camera extrinsic parameters of the camera to be calibrated.

[0086] If the target feature set obtained by the extraction unit 203 only includes a straight line set, the calibration unit 204 will obtain camera external parameters including yaw angle, pitch angle and roll angle; if the target feature set obtained by the extraction unit 203 includes both a straight line set and a point pair set, the calibration unit 204 will obtain yaw angle, pitch angle, roll angle, and offset parameters x0 (offset distance along the x direction) and y0 (offset distance along the y direction) of the camera to be calibrated relative to the vehicle positioning system.

[0087] Specifically, when the calibration unit 204 performs extrinsic parameter calibration on the camera to be calibrated according to the camera height and the target feature set of the camera to be calibrated, an optional implementation method that can be adopted is: taking the optical center of the camera to be calibrated as the origin, constructing a camera imaging model according to the first preset coordinate system, the camera height and the camera intrinsic parameters; using the constructed camera imaging model, projecting the image in the detection image sequence and the set of straight lines corresponding to the image in the target feature set to the ground to generate a bird's-eye view; when it is determined that the straight lines in the generated bird's-eye view do not meet the first constraint condition, adjusting the yaw angle, pitch angle and roll angle in the camera imaging model, and then executing the step of generating a bird's-eye view until it is determined that the straight lines in the generated bird's-eye view meet the first constraint condition, obtaining the current yaw angle, pitch angle and roll angle of the camera imaging model, and using them as the target yaw angle, target pitch angle and target roll angle respectively; and obtaining the camera extrinsic parameters of the camera to be calibrated according to the obtained target yaw angle, target pitch angle and target roll angle.

[0088] The first constraint condition used by the calibration unit 204 may include at least one of a constraint that lane lines in the bird's-eye view remain parallel and a constraint that zebra crossings in the bird's-eye view remain horizontal.

[0089] That is to say, the calibration unit 204 generates a bird's-eye view of the corresponding image by constructing a camera imaging model corresponding to the first preset coordinate system, and then continuously adjusts the values ​​of the yaw angle, pitch angle and roll angle in the camera imaging model according to whether the straight line in the generated bird's-eye view meets the preset constraints, until the straight line in the bird's-eye view generated by the camera imaging model meets the preset constraints, and obtains the values ​​of the yaw angle, pitch angle and roll angle in the current camera imaging model as the camera extrinsic parameters. Therefore, it is possible to complete the automatic calibration of the camera extrinsic parameters in the case that the camera to be calibrated is not installed properly in the crowdsourcing scenario.

[0090] If the target feature set obtained by the extraction unit 203 includes a point pair set, the calibration unit 204 obtains the camera extrinsic parameters of the camera to be calibrated based on the obtained target yaw angle, target pitch angle and target roll angle. An optional implementation method that can be adopted is: based on the point pair set in the target feature set, two target images corresponding to a target point pair in the point pair set are obtained from the detection image sequence; based on the obtained target yaw angle, target pitch angle and target roll angle, the two obtained target images are projected onto the ground using a camera imaging model to generate two bird's-eye views; based on the position information of the two target images and the offset between the point features of the corresponding target point pairs in the two bird's-eye views, the offset parameters of the camera to be calibrated relative to the vehicle positioning system are obtained; and the obtained target yaw angle, target pitch angle, target roll angle and offset parameters are used as the camera extrinsic parameters of the camera to be calibrated.

[0091] Among them, when the calibration unit 204 obtains the offset parameters of the camera to be calibrated relative to the vehicle positioning system based on the position information of the two target images and the point features between the point features of the corresponding target point pairs in the two bird's-eye views, it can first obtain the position information deviation based on the position information of the two target images, then obtain the offset distance between the point features of the corresponding target point pairs in the two bird's-eye views, and finally perform distance constraints based on the obtained position information deviation and the offset distance to obtain the offset parameters of the camera to be calibrated relative to the vehicle positioning system.

[0092] The calibration unit 204 may obtain two offset parameters, one along the x direction and the other along the y direction. If the x direction is the front-to-back direction, the y direction is the left-to-right direction; if the x direction is the left-to-right direction, the y direction is the front-to-back direction.

[0093] That is to say, after obtaining the external parameters of the target yaw angle, target pitch angle and target roll angle, the calibration unit 204 further obtains the offset parameters of the camera to be calibrated relative to the vehicle positioning system based on the obtained external parameters, and then uses the obtained target yaw angle, target pitch angle, target roll angle and offset parameters as the camera extrinsic parameters of the camera to be calibrated. While improving the accuracy of the obtained camera extrinsic parameters, it can also improve the diversity of the camera extrinsic parameters.

[0094] After obtaining the camera extrinsic parameters of the camera to be calibrated, the calibration unit 204 can construct a camera imaging model based on the obtained camera extrinsic parameters, and then use the constructed camera imaging model to generate a bird's-eye view of the image taken by the camera to be calibrated. After the generated bird's-eye view is processed with offset parameters, the bird's-eye view is spliced ​​according to the vehicle's driving trajectory to obtain a high-definition and accurate road image.

[0095] The calibration unit 204 may also identify and update landmark information on the road (eg, lane lines, guide arrows, speed reduction markings, etc.) based on the generated road image.

[0096] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0097] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0098] like Figure 3, which is a block diagram of an electronic device for a camera calibration method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example purposes only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0099] like Figure 3 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0100] Various components in device 300 are connected to I / O interface 305, including: an input unit 306, such as a keyboard, mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, optical disk, etc.; and a communication unit 309, such as a network card, modem, wireless communication transceiver, etc. The communication unit 309 allows device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0101] The computing unit 301 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the camera calibration method. For example, in some embodiments, the camera calibration method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 308.

[0102] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the camera calibration method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the camera calibration method in any other appropriate manner (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable camera calibration device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0109] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0110] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A camera calibration method, comprising: Get the camera intrinsic parameters of the camera to be calibrated; Obtaining a detection image sequence according to at least one driving image taken by the camera to be calibrated; Performing feature extraction on at least one driving image in the detection image sequence to obtain a target feature set, wherein the target feature set includes a line set and a point pair set; Performing extrinsic calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera intrinsic parameters, the straight line set, and the point pair set to obtain the camera extrinsic parameters of the camera to be calibrated, wherein the straight line set is used to obtain a target yaw angle, a target pitch angle, and a target roll angle in the camera extrinsic parameters, and the point pair set is used to obtain an offset parameter in the camera extrinsic parameters after obtaining the target yaw angle, the target pitch angle, and the target roll angle; The step of extracting features from at least one driving image in the detection image sequence to obtain a target feature set includes: Extracting line features of each driving image in the detection image sequence; performing fitting processing on the line features of each driving image in the detection image sequence to obtain a straight line fitting result corresponding to each driving image; and constructing the straight line set based on the straight line fitting result corresponding to each driving image; Extracting point features of each driving image in the detection image sequence; screening the point features according to the offset distance between the same point features in different driving images; and constructing the point pair set based on the point pairs respectively formed by the same point features located in different driving images after screening.

2. The method according to claim 1, wherein The step of obtaining a detection image sequence based on at least one driving image captured by the camera to be calibrated comprises: Aligning the position information of the camera to be calibrated at the time of shooting with the captured driving image according to the timestamp to obtain a first image sequence; The driving images in the first image sequence are screened according to the position information to obtain the detection image sequence.

3. The method according to any one of claims 1 to 2, wherein The step of performing extrinsic parameter calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera intrinsic parameters, the line set, and the point pair set to obtain the camera extrinsic parameters of the camera to be calibrated includes: Taking the optical center of the camera to be calibrated as the origin, constructing a camera imaging model according to a first preset coordinate system, the camera height and the camera intrinsic parameters; Using the camera imaging model, projecting the images in the detection image sequence and the set of straight lines corresponding to the images in the target feature set onto the ground to generate a bird's-eye view; If it is determined that the straight line in the bird's-eye view does not satisfy the first constraint condition, after adjusting the yaw angle, pitch angle, and roll angle in the camera imaging model, proceeding to the step of generating the bird's-eye view until it is determined that the straight line in the bird's-eye view satisfies the first constraint condition, obtaining the current yaw angle, pitch angle, and roll angle of the camera imaging model as the target yaw angle, target pitch angle, and target roll angle; The camera extrinsic parameters of the camera to be calibrated are obtained according to the target yaw angle, target pitch angle and target roll angle.

4. The method according to claim 3, wherein: Obtaining the camera extrinsic parameters of the camera to be calibrated according to the target yaw angle, target pitch angle, and target roll angle includes: According to the point pair set in the target feature set, two target images corresponding to a target point pair in the point pair set are obtained from the detection image sequence; projecting the two target images onto the ground using the camera imaging model according to the target yaw angle, target pitch angle, and target roll angle to generate two bird's-eye views; Obtaining an offset parameter of the camera to be calibrated relative to the vehicle positioning system based on the position information of the two target images and the offset distance between the point features of the corresponding target point pairs in the two bird's-eye views; The target yaw angle, target pitch angle, target roll angle and the offset parameter are used as camera extrinsic parameters of the camera to be calibrated.

5. A camera calibration device, comprising: An acquisition unit, used to obtain the camera intrinsic parameters of the camera to be calibrated; a processing unit, configured to obtain a detection image sequence based on at least one driving image captured by the camera to be calibrated; an extraction unit, configured to perform feature extraction on at least one driving image in the detection image sequence to obtain a target feature set, wherein the target feature set includes a line set and a point pair set; a calibration unit, configured to perform extrinsic calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera intrinsic parameters, the line set, and the point pair set, to obtain the camera extrinsic parameters of the camera to be calibrated, wherein the line set is used to obtain a target yaw angle, a target pitch angle, and a target roll angle from the camera extrinsic parameters, and the point pair set is used to obtain an offset parameter from the camera extrinsic parameters after obtaining the target yaw angle, the target pitch angle, and the target roll angle; The extraction unit, when extracting features from at least one driving image in the detection image sequence to obtain a target feature set, specifically performs the following steps: Extracting line features of each driving image in the detection image sequence; performing fitting processing on the line features of each driving image in the detection image sequence to obtain a straight line fitting result corresponding to each driving image; and constructing the straight line set based on the straight line fitting result corresponding to each driving image; Extracting point features of each driving image in the detection image sequence; screening the point features according to the offset distance between the same point features in different driving images; and constructing the point pair set based on the point pairs respectively formed by the same point features located in different driving images after screening.

6. The device according to claim 5, wherein When obtaining a detection image sequence based on at least one driving image captured by the camera to be calibrated, the processing unit specifically performs the following steps: Aligning the position information of the camera to be calibrated at the time of shooting with the captured driving image according to the timestamp to obtain a first image sequence; The driving images in the first image sequence are screened according to the position information to obtain the detection image sequence.

7. The device according to any one of claims 5 to 6, wherein: When the calibration unit performs extrinsic parameter calibration on the camera to be calibrated according to the camera height of the camera to be calibrated, the camera intrinsic parameters, the line set, and the point pair set to obtain the camera extrinsic parameters of the camera to be calibrated, the calibration unit specifically executes: Taking the optical center of the camera to be calibrated as the origin, constructing a camera imaging model according to a first preset coordinate system, the camera height and the camera intrinsic parameters; Using the camera imaging model, projecting the images in the detection image sequence and the set of straight lines corresponding to the images in the target feature set onto the ground to generate a bird's-eye view; If it is determined that the straight line in the bird's-eye view does not satisfy the first constraint condition, after adjusting the yaw angle, pitch angle, and roll angle in the camera imaging model, proceeding to the step of generating the bird's-eye view until it is determined that the straight line in the bird's-eye view satisfies the first constraint condition, obtaining the current yaw angle, pitch angle, and roll angle of the camera imaging model as the target yaw angle, target pitch angle, and target roll angle; The camera extrinsic parameters of the camera to be calibrated are obtained according to the target yaw angle, target pitch angle and target roll angle.

8. The device according to claim 7, wherein When the calibration unit obtains the camera extrinsic parameters of the camera to be calibrated according to the target yaw angle, the target pitch angle, and the target roll angle, the calibration unit specifically performs the following steps: According to the point pair set in the target feature set, two target images corresponding to a target point pair in the point pair set are obtained from the detection image sequence; projecting the two target images onto the ground using the camera imaging model according to the target yaw angle, target pitch angle, and target roll angle to generate two bird's-eye views; Obtaining an offset parameter of the camera to be calibrated relative to the vehicle positioning system based on the position information of the two target images and the offset distance between the point features of the corresponding target point pairs in the two bird's-eye views; The target yaw angle, target pitch angle, target roll angle and the offset parameter are used as camera extrinsic parameters of the camera to be calibrated.

9. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images