Vehicle-mounted all-round view camera calibration system and method based on feature invariance
By using a rotating platform and a calibration platform of constant feature during vehicle maintenance, combined with BEV transformation and feature processing, the fast and accurate calibration of the vehicle-mounted surround-view camera is achieved, and the calibration time, high cost and accurate in the existing technology is solved. The problem of low sex ensures the vehicle's accurate perception of the outer scene.
Patent Information
- Application Number
- CN202411029715.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-19
AI Technical Summary
The existing vehicle-mounted surround-view camera calibration methods have problems such as long time, high cost and low accuracy during vehicle maintenance, especially due to the placement of calibration layout, lighting conditions and operational complexity.
Using a vehicle-mounted surround-view camera calibration system and method based on feature invariance, using a rotating platform and a calibration platform of multiple invariant features, images are collected from different angles through the vehicle-mounted camera, BEV transformation and feature processing are performed, instance segmentation, and network model extraction are performed. Feature points, calculate external parameter deviations to achieve fast and accurate calibration.
It realizes the calibration of the vehicle-mounted surround-view camera quickly and accurately during vehicle maintenance, ensuring the vehicle's accurate perception of the peripheral scene, reducing calibration costs and time, and improving calibration accuracy.
Smart Images

Figure CN118982587A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of camera calibration, and in particular to a vehicle-mounted surround view camera calibration system based on feature invariance and a vehicle-mounted surround view camera calibration method based on feature invariance. Background Art
[0002] With the continuous development of autonomous driving technology, on-board cameras have become an indispensable key sensing device on vehicles. For on-board surround view cameras, it is usually necessary to jointly calibrate multiple on-board cameras so that multiple images taken by multiple on-board cameras at the same time can be accurately stitched into a panoramic image around the vehicle. However, as vehicles are used, vehicle collision accidents are inevitable, resulting in damage to individual on-board cameras in the vehicle. At this time, multiple on-board cameras need to be re-calibrated jointly to ensure the accuracy of the panoramic image.
[0003] In the prior art, the calibration methods of the on-board surround-view camera during vehicle maintenance usually include factory calibration and 4S shop calibration. If the factory calibration is performed, on the one hand, it takes a long time and the car owner needs to wait for a long time; on the other hand, because the car factory usually operates on an assembly line, for this special situation, both the car factory and the car owner will incur higher costs.
[0004] In 4S stores, calibration is usually done by laying multiple calibration cloths around the vehicle and using image corners to complete the calibration. This calibration method is not only easily affected by factors such as the placement, flatness, and contamination of the calibration cloths, but is also affected by the lighting conditions in the calibration scene. The operation is complex and the accuracy is low. Summary of the invention
[0005] The purpose of this application is to quickly and accurately calibrate a vehicle-mounted surround view camera during vehicle maintenance.
[0006] The technical solution of the first aspect of the present application is: a vehicle-mounted surround view camera calibration system based on feature invariance is provided, the system includes: a calibration platform, at least one feature object and a rotating platform are arranged in the calibration platform, wherein the rotating platform is used to place a vehicle to be calibrated, and at least two vehicle-mounted cameras are arranged in the vehicle to be calibrated, and the vehicle-mounted cameras include at least one vehicle-mounted camera to be calibrated and at least one calibrated vehicle-mounted camera; an image processing unit, the image processing unit is configured to perform BEV transformation on images collected by the vehicle-mounted cameras when the vehicle to be calibrated is located at at least two different angles after the rotating platform is rotated, so as to generate a surround view sampling image; a feature processing unit, the feature processing unit is configured to extract image features in the surround view sampling image and determine the feature values of each image feature; a camera calibration unit, the camera calibration unit is configured to calibrate the external parameters of the vehicle-mounted camera to be calibrated based on the feature values.
[0007] In any of the above technical solutions, further, when the rotating platform rotates, it drives the vehicle to be calibrated to rotate, and adjusts the angle of the vehicle to be calibrated in the calibration platform so that the on-board camera can capture images at different angles, wherein the center of the vehicle to be calibrated coincides with the center of the circle when the rotating platform rotates.
[0008] In any of the above technical solutions, further, the feature processing unit is configured to: use instance segmentation to perform instance segmentation on each surround view sampling image respectively to determine the segmentation target; count the pixel distribution of each segmentation target in each surround view sampling image, calculate the center point coordinates of each segmentation target, and mark the center point coordinates as feature values; the camera calibration unit is configured to: pair the center point coordinates of each segmentation target in any two surround view sampling images to form a center point coordinate pair, and calculate the center coordinate deviation of the center point coordinate pair, traverse the extrinsic parameter calibration parameters, and when it is determined that the center coordinate deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
[0009] In any of the above technical solutions, further, the feature processing unit is also configured to: count the number of pixels of each segmented target in each surround sampling image, and use the number of pixels as the instance area of the corresponding segmented target, and record the instance area as the feature value; the camera calibration unit is also configured to: calculate the area deviation of the instance areas of the two segmented targets in any pair of center point coordinates, traverse the extrinsic calibration parameters, and when it is determined that the area deviation and / or the center coordinate deviation is the smallest, use the corresponding extrinsic calibration parameters as the optimal extrinsic parameters.
[0010] In any of the above technical solutions, further, the feature processing unit is also configured to: use the network model to extract features from each surround sampling image in turn to obtain a feature map, and remove similar feature points in each feature map, retain independent feature points, and record the independent feature points as feature values; the camera calibration unit is also configured to: calculate the similarity of the independent feature points retained between each feature map, form similar feature point pairs, and calculate the feature deviation of the similar feature point pairs, traverse the extrinsic parameter calibration parameters, and when it is determined that the feature deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
[0011] The technical solution of the second aspect of the present application is: a method for calibrating a vehicle-mounted surround view camera based on feature invariance is provided, the method comprising: step 1: when the vehicle to be calibrated rotates a preset angle driven by a rotating platform, image acquisition is performed, and the image acquired by the vehicle-mounted camera is subjected to BEV transformation to generate a surround view sampling image, wherein the vehicle-mounted camera is arranged on the vehicle to be calibrated, and the vehicle-mounted camera comprises at least one vehicle-mounted camera to be calibrated and at least one calibrated vehicle-mounted camera; step 2: extracting image features in the surround view sampling image, determining the feature values of each image feature, and calibrating the external parameters of the vehicle-mounted camera to be calibrated based on the feature values.
[0012] In any of the above technical solutions, further, step 2 includes: step 201: using instance segmentation to perform instance segmentation on each surround view sampling image and determine the segmentation target; step 202: counting the pixel distribution of each segmentation target in each surround view sampling image, calculating the center point coordinates of each segmentation target, and marking the center point coordinates as feature values; step 203: pairing the center point coordinates of each segmentation target in any two surround view sampling images to form a center point coordinate pair, and calculating the center coordinate deviation of the center point coordinate pair, traversing the external parameter calibration parameters, and when it is determined that the center coordinate deviation is the smallest, the corresponding external parameter calibration parameter is used as the optimal external parameter, wherein the optimal external parameter is used to calibrate the external parameter of the vehicle-mounted camera to be calibrated.
[0013] In any of the above technical solutions, further, step 2 also includes: step 204: counting the number of pixels of each segmented target in each surround sampling image, and taking the number of pixels as the instance area of the corresponding segmented target, and recording the instance area as the eigenvalue; step 205: calculating the area deviation of the instance areas of the two segmented targets in any pair of center point coordinates, traversing the extrinsic parameter calibration parameters, and when it is determined that the area deviation and / or the center coordinate deviation is the smallest, the corresponding extrinsic parameter calibration parameters are used as the optimal extrinsic parameters.
[0014] In any of the above technical solutions, further, step 2 includes: step 211: using the network model, extracting features from each surround sampling image in turn to obtain a feature map, and removing similar feature points in each feature map, retaining independent feature points, and recording the independent feature points as feature values; step 212: calculating the similarity of the independent feature points retained between each feature map, forming similar feature point pairs, and calculating the feature deviation of the similar feature point pairs, traversing the extrinsic parameter calibration parameters, and when it is determined that the feature deviation is the smallest, using the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
[0015] In any of the above technical solutions, further, the center of the vehicle to be calibrated coincides with the center of a circle when the rotating platform rotates.
[0016] The beneficial effects of this application are:
[0017] The technical solution in this application is based on the premise that the scene features remain unchanged, and the various features in the BEV images obtained by the vehicle at different angles in the same scene should be consistent. Therefore, when calibrating the on-board camera to be calibrated in the vehicle to be calibrated, the vehicle to be calibrated is placed on a calibration platform with a rotating platform and multiple unchanged feature objects, and the on-board camera to be calibrated and the calibrated on-board camera on the vehicle are used to simultaneously obtain the image of the current scene and perform BEV transformation, and then rotate to different angles to repeat the above process.
[0018] After acquiring multiple surround view sampling images of the same scene at different angles, the matching relationship between the features between the images is constructed, and the corresponding relationship between the external parameters is constructed using the network. Therefore, by adjusting the external parameters of the vehicle-mounted camera to be calibrated, the various features in the image can be made consistent, that is, the vehicle can accurately perceive the peripheral scene information, and realize fast and accurate calibration of the vehicle-mounted surround view camera during vehicle maintenance.
[0019] In order to ensure that the distance and size of obstacles in the scene can be accurately perceived, this application also calculates the deviation of the center point coordinates and the instance area (number of pixels) through instance segmentation, so as to obtain accurate external parameters of the on-board camera to be calibrated. At the same time, based on the above scheme, this application can also form similar feature point pairs through feature point elimination and matching, and perform more accurate calibration on the feature point information in the surround view sampling image. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:
[0021] Figure 1 is a schematic block diagram of a vehicle-mounted surround view camera calibration system based on feature invariance according to an embodiment of the present application;
[0022] Figure 2 is a schematic diagram of a calibration platform according to an embodiment of the present application;
[0023] Figure 3 It is a schematic flowchart of a method for calibrating a vehicle-mounted surround view camera based on feature invariance according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0025] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.
[0026] Embodiment 1:
[0027] like Figure 1 and Figure 2 As shown, this embodiment provides a vehicle-mounted surround view camera calibration system 10 based on feature invariance, the system comprising:
[0028] A calibration platform 100, wherein the calibration platform 100 is provided with at least one feature object 101 and a rotating platform 102, wherein the rotating platform 102 is used to place a vehicle 103 to be calibrated, wherein the vehicle 103 to be calibrated is provided with at least two vehicle cameras, and the vehicle cameras include at least one vehicle camera to be calibrated and at least one calibrated vehicle camera;
[0029] The image processing unit 200 is configured to perform BEV transformation on the images collected by the vehicle-mounted camera when the vehicle 103 to be calibrated is located at at least two different angles after the rotating platform 102 rotates, so as to generate surround sampling images;
[0030] A feature processing unit 300, the feature processing unit 300 is configured to extract image features in the surround sampling image and determine a feature value of each image feature;
[0031] The camera calibration unit 400 is configured to calibrate the external parameters of the vehicle-mounted camera to be calibrated based on the characteristic values.
[0032] Specifically, the on-board surround view camera has become an indispensable sensing device in autonomous vehicles. Usually, 6-8 on-board cameras are installed at different angles of the vehicle. Then the manufacturer uses a series of calibration methods to determine the external parameters (position) of each on-board camera to form a rotation matrix and an offset matrix. Finally, the perspective transformation, image fusion and other methods are used to generate a BEV image centered on the vehicle to achieve perception of the vehicle's surrounding environment.
[0033] However, during the driving process, the vehicle will inevitably have some collisions and damages, which will cause the external parameters of the vehicle-mounted camera to change. Therefore, the vehicle-mounted camera to be calibrated in the vehicle to be calibrated 103 needs to be recalibrated.
[0034] The calibration system 10 in this embodiment is provided with a calibration platform 100, on which a plurality of feature objects 101 can be fixedly provided. The number, shape, size and installation position of the feature objects 101 can be set according to requirements, but during the calibration process, it is necessary to ensure that the feature objects 101 do not change.
[0035] The calibration platform 100 is also provided with a rotatable rotating platform 102. When the rotating platform 102 rotates, it drives the vehicle 103 to be calibrated to rotate, and adjusts the angle of the vehicle 103 to be calibrated in the calibration platform 100, so that the on-board camera can collect images at different angles, wherein the center of the vehicle 103 to be calibrated coincides with the center of the circle when the rotating platform 102 rotates. In other words, when the rotating platform 102 rotates, the vehicle 103 to be calibrated above it rotates accordingly, and the center of the vehicle coincides with the center of the circle when the rotating platform 102 rotates. Therefore, even if the vehicle rotates, the distance between each feature 101 and the center of the vehicle remains unchanged.
[0036] Those skilled in the art can understand that, from the perspective of the BEV, the size of the feature 101 perceived by the vehicle should not change due to the different angles of the vehicle when perceiving it.
[0037] Therefore, this embodiment calibrates the on-board camera to be calibrated in the vehicle to be calibrated 103 based on the characteristic that the features remain unchanged under the BEV viewing angle.
[0038] During the calibration process, after the rotating platform 102 rotates an angle (such as 10°, 20°, etc. counterclockwise), a group of sample images are obtained through the onboard cameras (onboard cameras to be calibrated and calibrated onboard cameras) on the to-be-calibrated vehicle 103. After that, the group of sample images is sent to the image processing unit 200, which performs BEV transformation on each group of sample images collected at different angles based on the current set external parameters of each onboard camera to generate surround view sample images.
[0039] It should be noted that, for the vehicle-mounted camera to be calibrated, the camera extrinsic parameters at the corresponding position when the vehicle leaves the factory can be used as the initial values of the extrinsic parameters used during calibration.
[0040] Those skilled in the art can understand that if the extrinsic parameters of a vehicle-mounted camera (the vehicle-mounted camera to be calibrated) are abnormal, the result reflected in the BEV image is that the image of the area corresponding to the vehicle-mounted camera will be abnormal, while the images of other areas will not be affected. That is, if the extrinsic parameters of the vehicle-mounted camera at the right front of the vehicle are abnormal, then the image of the upper right area in the BEV image will be abnormal.
[0041] When the vehicle 103 to be calibrated rotates with the rotating platform 102, such as 180°, the image area captured by the on-board camera (with normal external parameters) at its left rear and after BEV transformation should be the correct area. Therefore, combined with the feature invariance under the BEV perspective, the correct image features in the BEV image at this time can be used to correct the previous erroneous image features. The determination of image features is closely related to the external parameters of the on-board camera, thereby realizing the external parameter calibration of the on-board camera to be calibrated.
[0042] The calibration system 10 in this embodiment is further provided with an image processing unit 200, which performs BEV transformation. For an image sampled at any angle, the image sampled is set to {I i |1≤i≤N}, where N is the number of vehicle cameras, including the vehicle cameras to be calibrated and the calibrated vehicle cameras.
[0043] First, according to the external parameters {R i ,t i |1≤i≤N} calculate the corresponding perspective transformation matrix H i , the corresponding calculation formula is:
[0044] H i =K i [R i |t i ]K i -1
[0045] In the formula, K i is the internal parameter of the i-th vehicle camera, corresponding to the i-th image I i .
[0046] Then, use the perspective transformation matrix H i For image I i Perform perspective transformation to obtain the image BEV from the BEV perspective i , the corresponding calculation formula is:
[0047] BEV i =wrapPerspective(I i ,H i )
[0048] Finally, for the image BEV i Perform image fusion and use an image stitching algorithm, such as a weighted average method or a multi-band fusion algorithm, to process the overlapping areas of each image to obtain the final surround sampling image.
[0049] Afterwards, the image processing unit 200 sends the surround sampling image to the feature processing unit 300, which extracts image features in the surround sampling image, such as each feature object 101 on the calibration platform 100, and calculates its feature values in the surround sampling image, such as length, width, area, volume, center point coordinates, etc.
[0050] The camera calibration unit 400 then obtains multiple feature pairs based on the above image features through feature matching, and then uses the matched feature pairs to iteratively optimize the external parameters of the vehicle-mounted camera to be calibrated until the termination condition is met, the optimal external parameters are obtained, and the calibration is completed.
[0051] In the process of optimizing the extrinsic parameters, the corresponding extrinsic calibration parameters can be randomly selected within the set extrinsic parameter variation range, and the difference of the eigenvalues corresponding to different extrinsic calibration parameters can be calculated. When the difference is 0 or close to 0, it is considered that the current extrinsic calibration parameters are the true extrinsic parameters of the vehicle camera to be calibrated, and the extrinsic calibration is completed.
[0052] It should be noted that, since the number of feature objects 101 in the calibration platform 100 is fixed, the number of image features in the surround sampling image under the BEV perspective is also a fixed value, that is, the number of feature pairs obtained each time the matching is performed is fixed. Therefore, if there are image features that cannot be matched successfully, it can be determined that it is caused by abnormal external parameters of the corresponding on-board camera, that is, external parameters that need to be calibrated. At this time, such image features can be automatically / manually marked as feature pairs.
[0053] During the autonomous driving process, the main task of BEV perception is to obtain information such as the distance between the obstacle in front and the vehicle, the size of the obstacle itself, and the spacing between multiple obstacles. Based on this, this embodiment can select the above information as calibration parameters so that the calibrated vehicle camera can perceive accurate external environment information.
[0054] In any of the above embodiments, further, in order to accurately perceive the distance information of the obstacle, the feature processing unit 300 is configured to: adopt an instance segmentation method to perform instance segmentation on each surround view sampling image respectively to determine the segmentation target, wherein the instance segmentation model SAM can be selected to perform instance segmentation on each surround view sampling image during instance segmentation; count the pixel distribution of each segmentation target in each surround view sampling image, calculate the center point coordinates of each segmentation target, and mark the center point coordinates as feature values;
[0055] Assume that the bth instance in the ath surround sampling image can be expressed as {(x ab,k ,y ab,k )|1≤k≤P ab}, the number of pixels is P ab, where 1≤a≤M, M is the number of surround sampling images, and 1≤b≤B, B is the number of instances, that is, the number of feature objects 101. For the bth instance in the ath surround sampling image, the calculation formula for the coordinates of its center point is:
[0056]
[0057] In the formula, is the center point coordinate.
[0058] Therefore, the camera calibration unit 400 is configured to pair the center point coordinates of each segmented target in any two surround sampling images (such as using the Hungarian algorithm) to form a center point coordinate pair, and calculate the center coordinate deviation of the center point coordinate pair, traverse the extrinsic calibration parameters, and when it is determined that the center coordinate deviation is the smallest, use the corresponding extrinsic calibration parameters as the optimal extrinsic parameters, wherein the optimal extrinsic parameters are used to calibrate the extrinsic parameters of the vehicle-mounted camera to be calibrated.
[0059] Specifically, the coordinate pair of the center point after matching is set to It represents the center point coordinate pair matched by instance b in the a1th and a2th surround sampling images. Therefore, the corresponding relationship between the camera extrinsic parameters and the center coordinate deviation can be established through the network, and the camera extrinsic parameters (extrinsic calibration parameters) with the minimum center coordinate deviation can be calculated. The corresponding calculation formula is:
[0060]
[0061] In the formula, (R 0 ,t 0 ) is the initial value of the external parameter, E 2Dcenter is the center coordinate deviation, N1[] represents the corresponding network model, and its input is the surround sampling image C BEV And the initial value of the external parameter, through instance segmentation, coordinate calculation method to output the corresponding center point coordinates, these center point coordinates as the center coordinate deviation E 2Dcen By adjusting the external calibration parameters, the center coordinate deviation E is minimized. 2Dcenter , in order to calculate the optimal external parameter (R * ,t * ).
[0062] In any of the above embodiments, further, in order to accurately perceive the size information of the obstacle, the feature processing unit 300 is further configured to: count the number of pixels of each segmented target in each surround sampling image, and use the number of pixels as the instance area of the corresponding segmented target, and record the instance area as the feature value;
[0063] Therefore, the camera calibration unit 400 is further configured to calculate the area deviation of the instance areas of the two segmented targets in any pair of center point coordinates, traverse the external parameter calibration parameters, and when it is determined that the area deviation or the center coordinate deviation is the smallest, use the corresponding external parameter calibration parameter as the optimal external parameter, wherein the corresponding calculation formula is:
[0064]
[0065]
[0066] In the formula, E 2Dar is the area deviation, area′ a1,b 、area′ a2,b are the instance areas of instance b in the a1th and a2th surround sampling images respectively.
[0067] It should be noted that the corresponding external parameter calibration parameters can also be calculated when the area deviation and the center coordinate deviation are both minimized, and the specific process will not be repeated here.
[0068] In order to perceive the information in the surrounding environment more precisely, after the obstacle position information is perceived, the feature point information in the surround sampling image is used for more accurate calibration. In any of the above embodiments, further, the feature processing unit 300 is also configured to: use the network model to extract features from each surround sampling image in turn to obtain a feature map, and remove similar feature points in each feature map, retain independent feature points, and record the independent feature points as feature values;
[0069] Specifically, the surround sampling image under the BEV perspective is set to {BEV j |0≤j≤M}, where M is the number of surround sampling images, which is determined by the number of times the rotating platform 102 rotates. j Input to the trained network model for feature extraction, such as U-Net network, and take the feature map of the last layer of the network with the same resolution as the original image (surround sampling image) as the final output feature map, denoted as feature map {F j |0≤j≤M}.
[0070] For any feature map F j For each feature point, sample it to obtain a B×B neighborhood, and determine whether there is a point in the neighborhood whose similarity to its cosine is greater than the threshold θ. 0 If a feature point exists, remove the similar feature point, otherwise, keep the feature point as an independent feature point. In this way, P independent feature points are selected from the feature map to ensure that each feature point is unique in its own neighborhood.
[0071] Therefore, the camera calibration unit 400 is further configured to calculate the similarity of the independent feature points retained between the feature maps to form similar feature point pairs, calculate the feature deviation of the similar feature point pairs, traverse the extrinsic calibration parameters, and when it is determined that the feature deviation is the smallest (0 or close to 0), use the corresponding extrinsic calibration parameters as the optimal extrinsic parameters, wherein the optimal extrinsic parameters are used to calibrate the extrinsic parameters of the vehicle-mounted camera to be calibrated.
[0072] Specifically, a method similar to the above process can be used to calculate any feature map F j The pth independent feature point in the image is determined, and a B×B neighborhood is determined at its position. The independent feature points retained in this neighborhood in other feature maps are determined in turn. The feature point with the largest cosine similarity value is formed into a similar feature point pair, which is recorded as Indicates that the jth feature map F j The independent feature point p and the mth (m≠j)th feature map F m Similar feature point pairs composed of independent feature points q in .
[0073] Therefore, the corresponding relationship between the camera extrinsic parameters and the independent feature points can be established through the network, and the camera extrinsic parameters (extrinsic parameter calibration parameters) with the minimum feature deviation of similar feature point pairs composed of independent feature points can be calculated. The corresponding calculation formula is:
[0074]
[0075] In the formula, (R 0 ,t 0 ) is the initial value of the external parameter, E point is the feature deviation, N3[] represents the corresponding network model, whose input is the feature map F and the initial value of the external parameter, and the output is the independent feature point; the coordinates of these feature points are used as the feature deviation E point By adjusting the external calibration parameters, the characteristic deviation E is minimized point , calculate the optimal external parameter (R * ,t * ).
[0076] Embodiment 2:
[0077] like Figure 2 and Figure 3 As shown, this embodiment shows a vehicle-mounted surround view camera calibration method based on feature invariance, the method comprising:
[0078] Step 1: After the vehicle 103 to be calibrated rotates a preset angle driven by the rotating platform 102, image acquisition is performed, and the image acquired by the vehicle-mounted camera is subjected to BEV transformation to generate a surround sampling image, wherein the vehicle-mounted camera is arranged on the vehicle 103 to be calibrated, and the vehicle-mounted camera includes at least one vehicle-mounted camera to be calibrated and at least one calibrated vehicle-mounted camera;
[0079] Step 2: Extract image features from the surround view sampling image, determine the eigenvalues of each image feature, and calibrate the external parameters of the vehicle camera to be calibrated based on the eigenvalues.
[0080] Specifically, in this embodiment, a calibration platform 100 is provided, on which a plurality of feature objects 101 can be fixedly provided. The number, shape, size and installation position of the feature objects 101 can be set according to requirements. However, during the calibration process, it is necessary to ensure that the feature objects 101 do not change.
[0081] The calibration platform 100 is also provided with a rotatable rotating platform 102. When the rotating platform 102 rotates, it drives the vehicle 103 to be calibrated to rotate, and adjusts the angle of the vehicle 103 to be calibrated in the calibration platform 100, so that the on-board camera can collect images at different angles, wherein the center of the vehicle 103 to be calibrated coincides with the center of the circle when the rotating platform 102 rotates. In other words, when the rotating platform 102 rotates, the vehicle 103 to be calibrated above it rotates accordingly, and the center of the vehicle coincides with the center of the circle when the rotating platform 102 rotates. Therefore, even if the vehicle rotates, the distance between each feature 101 and the center of the vehicle remains unchanged.
[0082] Those skilled in the art can understand that, from the perspective of the BEV, the size of the feature 101 perceived by the vehicle should not change due to the different angles of the vehicle when perceiving it.
[0083] Therefore, this embodiment calibrates the on-board camera to be calibrated in the vehicle to be calibrated 103 based on the characteristic that the features remain unchanged under the BEV viewing angle.
[0084] During the calibration process, each time the rotating platform 102 rotates an angle (such as 10°, 20°, etc. counterclockwise), a set of sample images is acquired through the onboard cameras (onboard cameras to be calibrated and calibrated onboard cameras) on the to-be-calibrated vehicle 103. Afterwards, the BEV transformation is performed on each set of sample images acquired at different angles based on the currently set external parameters of each onboard camera to generate surround view sample images.
[0085] First, according to the external parameters {R i ,t i |1≤i≤N} calculate the corresponding perspective transformation matrix H i, the corresponding calculation formula is:
[0086] H i =K i [R i |t i ]K i -1
[0087] In the formula, k i is the internal parameter of the i-th vehicle camera, corresponding to the i-th image i i .
[0088] Then, use the perspective transformation matrix H i For image I i Perform perspective transformation to obtain the image BEV from the BEV perspective i , the corresponding calculation formula is:
[0089] BEV i =wrapPerspective(I i ,H i )
[0090] Finally, for the image BEV i Perform image fusion and use an image stitching algorithm, such as a weighted average method or a multi-band fusion algorithm, to process the overlapping areas of each image to obtain the final surround sampling image.
[0091] Afterwards, image features in the surround sampling image are extracted, such as each feature object 101 on the calibration platform 100, and their feature values in the surround sampling image, such as length, width, area, volume, center point coordinates, etc., are calculated.
[0092] Based on the above image features, multiple feature pairs are obtained through feature matching. The matched feature pairs are then used to iteratively optimize the external parameters of the vehicle camera to be calibrated until the termination conditions are met, the optimal external parameters are obtained, and the calibration is completed.
[0093] In the process of optimizing the extrinsic parameters, the corresponding extrinsic calibration parameters can be randomly selected within the set extrinsic parameter variation range, and the difference of the eigenvalues corresponding to different extrinsic calibration parameters can be calculated. When the difference is 0 or close to 0, it is considered that the current extrinsic calibration parameters are the true extrinsic parameters of the vehicle camera to be calibrated, and the extrinsic calibration is completed.
[0094] In any of the above embodiments, further, step 2 includes:
[0095] Step 201: using instance segmentation to perform instance segmentation on each surround view sampling image to determine the segmentation target;
[0096] Step 202: Count the pixel distribution of each segmented target in each surround sampling image, calculate the center point coordinates of each segmented target, and mark the center point coordinates as feature values;
[0097] Step 203: Pair the center point coordinates of each segmented target in any two surround sampling images to form a center point coordinate pair, calculate the center coordinate deviation of the center point coordinate pair, traverse the extrinsic parameter calibration parameters, and when it is determined that the center coordinate deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, where the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
[0098] Specifically, the bth instance in the ath surround sampling image can be expressed as {(x ab,k ,y ab,k )|1≤k≤P ab}, the number of pixels is P ab , where 1≤a≤M, M is the number of surround sampling images, and 1≤b≤B, B is the number of instances, that is, the number of feature objects 101. For the bth instance in the ath surround sampling image, the calculation formula for the coordinates of its center point is:
[0099]
[0100] In the formula, is the center point coordinate.
[0101] Set the center point coordinate pair after matching to It represents the center point coordinate pair matched by instance b in the a1th and a2th surround sampling images. Therefore, the corresponding relationship between the camera extrinsic parameters and the center coordinate deviation can be established through the network, and the camera extrinsic parameters (extrinsic calibration parameters) with the minimum center coordinate deviation can be calculated. The corresponding calculation formula is:
[0102]
[0103] In the formula, (R 0 ,t 0 ) is the initial value of the external parameter, E 2Dcenter is the center coordinate deviation, N1[] represents the corresponding network model, and its input is the surround sampling image C BEV And the initial value of the external parameter, through instance segmentation, coordinate calculation method to output the corresponding center point coordinates, these center point coordinates as the center coordinate deviation E 2Dcenter By adjusting the external calibration parameters, the center coordinate deviation E is minimized. 2Dcenter , in order to calculate the optimal external parameter (R * ,t * ).
[0104] In any of the above embodiments, further, step 2 also includes:
[0105] Step 204: Count the number of pixels of each segmentation target in each surround sampling image, and use the number of pixels as the instance area of the corresponding segmentation target, and record the instance area as the feature value;
[0106] Step 205: Calculate the area deviation of the instance areas of the two segmented targets in any pair of center point coordinates, traverse the external parameter calibration parameters, and when it is determined that the area deviation and / or the center coordinate deviation is the smallest, use the corresponding external parameter calibration parameters as the optimal external parameters, where the corresponding calculation formula is:
[0107]
[0108]
[0109] In the formula, E 2Darea is the area deviation, area′ a1,b 、area′ a2,b are the instance areas of instance b in the a1th and a2th surround sampling images respectively.
[0110] In any of the above embodiments, further, step 2 includes:
[0111] Step 211: using the network model, extracting features from each surround sampling image in turn to obtain a feature map, and removing similar feature points in each feature map, retaining independent feature points, and recording the independent feature points as feature values;
[0112] Step 212: Calculate the similarity of the independent feature points retained between the feature maps to form similar feature point pairs, and calculate the feature deviation of the similar feature point pairs, traverse the extrinsic parameter calibration parameters, and when it is determined that the feature deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
[0113] Specifically, the surround sampling image under the BEV perspective is set to {BEV j |0≤j≤M}, where M is the number of surround sampling images, which is determined by the number of times the rotating platform 102 rotates. j Input to the trained network model for feature extraction, such as U-Net network, and take the feature map of the last layer of the network with the same resolution as the original image (surround sampling image) as the final output feature map, denoted as feature map {F j |0≤j≤M}.
[0114] For any feature map F jFor each feature point, sample it to obtain a B×B neighborhood, and determine whether there is a point in the neighborhood whose similarity to its cosine is greater than the threshold θ. 0 If a feature point exists, remove the similar feature point, otherwise, keep the feature point as an independent feature point. In this way, P independent feature points are selected from the feature map to ensure that each feature point is unique in its own neighborhood.
[0115] A similar process can be used to calculate the feature map F j The pth independent feature point in the image is determined, and a B×B neighborhood is determined at its position. The independent feature points retained in this neighborhood in other feature maps are determined in turn. The feature point with the largest cosine similarity value is formed into a similar feature point pair, which is recorded as Indicates that the jth feature map F j The independent feature point p and the mth (m≠j)th feature map F m Similar feature point pairs composed of independent feature points q in .
[0116] Therefore, the corresponding relationship between the camera extrinsic parameters and the independent feature points can be established through the network, and the camera extrinsic parameters (extrinsic parameter calibration parameters) with the minimum feature deviation of similar feature point pairs composed of independent feature points can be calculated. The corresponding calculation formula is:
[0117]
[0118] In the formula, (R 0 ,t 0 ) is the initial value of the external parameter, E point is the feature deviation, N3[] represents the corresponding network model, whose input is the feature map F and the initial value of the external parameter, and the output is the independent feature point; the coordinates of these feature points are used as the feature deviation E point By adjusting the external calibration parameters, the characteristic deviation E is minimized point , calculate the optimal external parameter (R * ,t * ).
[0119] In summary, when calibrating the on-board camera to be calibrated in the vehicle to be calibrated, the present embodiment obtains multiple surround-view sampling images of the same scene and different angles by placing the vehicle to be calibrated on a calibration platform with a rotating platform and multiple invariant features, constructs a matching relationship between each feature between images, and uses a network to construct a correspondence between external parameters and features. Therefore, by adjusting the external parameters of the on-board camera to be calibrated, the various features in the image can be made consistent, that is, the vehicle can accurately perceive the peripheral scene information, and realize fast and accurate calibration of the on-board surround-view camera during vehicle maintenance.
[0120] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.
[0121] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.
[0122] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.
[0123] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.
Claims
1. A vehicle-mounted surround view camera calibration system based on feature invariance, It is characterized in that The system comprises: A calibration platform, wherein at least one feature and a rotating platform are arranged on the calibration platform, wherein the rotating platform is used to place a vehicle to be calibrated, wherein the vehicle to be calibrated is provided with at least two vehicle-mounted cameras, wherein the vehicle-mounted cameras include at least one vehicle-mounted camera to be calibrated and at least one calibrated vehicle-mounted camera; An image processing unit, wherein the image processing unit is configured to perform BEV transformation on the images collected by the vehicle-mounted camera when the vehicle to be calibrated is located at at least two different angles after the rotating platform rotates, so as to generate a surround view sampling image; A feature processing unit, wherein the feature processing unit is configured to extract image features from the surround view sampling image and determine a feature value of each image feature; A camera calibration unit is configured to calibrate the external parameters of the vehicle-mounted camera to be calibrated based on the characteristic value.
2. The vehicle-mounted surround view camera calibration system based on feature invariance as claimed in claim 1, It is characterized in that When the rotating platform rotates, the vehicle to be calibrated is driven to rotate, and the angle of the vehicle to be calibrated on the calibration platform is adjusted so that the vehicle-mounted camera can collect images at different angles. The center of the vehicle to be calibrated coincides with the center of a circle of the rotating platform when it rotates.
3. The vehicle-mounted surround view camera calibration system based on feature invariance as claimed in claim 1, It is characterized in that The feature processing unit is configured as follows: Using instance segmentation, each of the surround view sampling images is segmented to determine a segmentation target; Counting the pixel distribution of each segmented target in each of the surround sampling images, calculating the center point coordinates of each segmented target, and marking the center point coordinates as the feature value; The camera calibration unit is configured as follows: The center point coordinates of each segmented target in any two of the surround view sampling images are paired to form a center point coordinate pair, and the center coordinate deviation of the center point coordinate pair is calculated, and the extrinsic parameter calibration parameters are traversed. When it is determined that the center coordinate deviation is the smallest, the corresponding extrinsic parameter calibration parameter is used as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
4. The vehicle-mounted surround view camera calibration system based on feature invariance as claimed in claim 3, It is characterized in that The feature processing unit is further configured to: Counting the number of pixels of each of the segmented objects in each of the surround sampling images, and using the number of pixels as the instance area of the corresponding segmented object, and recording the instance area as the feature value; The camera calibration unit is further configured to: Calculate the area deviation of the instance areas of the two segmented targets in any pair of the center point coordinates, traverse the extrinsic parameter calibration parameters, and when it is determined that the area deviation and / or the center coordinate deviation is the smallest, use the corresponding extrinsic parameter calibration parameters as the optimal extrinsic parameter.
5. The vehicle-mounted surround view camera calibration system based on feature invariance as claimed in claim 1, It is characterized in that The feature processing unit is further configured to: Using the network model, extracting features from each of the surround sampling images in turn to obtain a feature map, and removing similar feature points in each feature map, retaining independent feature points, and recording the independent feature points as the feature values; The camera calibration unit is further configured to: The similarity of the independent feature points retained between the feature maps is calculated to form similar feature point pairs, and the feature deviation of the similar feature point pairs is calculated, and the extrinsic parameter calibration parameters are traversed. When it is determined that the feature deviation is the smallest, the corresponding extrinsic parameter calibration parameter is used as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
6. A vehicle-mounted surround view camera calibration method based on feature invariance, It is characterized in that The method comprises: Step 1: When the vehicle to be calibrated rotates a preset angle driven by the rotating platform, image acquisition is performed, and the image acquired by the vehicle-mounted camera is subjected to BEV transformation to generate a surround sampling image, wherein the vehicle-mounted camera is set on the vehicle to be calibrated, and the vehicle-mounted camera includes at least one vehicle-mounted camera to be calibrated and at least one calibrated vehicle-mounted camera; Step 2: extracting image features from the surround view sampling image, determining feature values of each image feature, and calibrating the external parameters of the vehicle-mounted camera to be calibrated based on the feature values.
7. The vehicle-mounted surround view camera calibration method based on feature invariance as claimed in claim 6, It is characterized in that The step 2 includes: Step 201: using instance segmentation to perform instance segmentation on each of the surround view sampling images to determine a segmentation target; Step 202: Counting the pixel distribution of each segmented object in each of the surround sampling images, calculating the center point coordinates of each segmented object, and marking the center point coordinates as the feature value; Step 203: Pair the center point coordinates of each segmented target in any two of the surround view sampling images to form a center point coordinate pair, calculate the center coordinate deviation of the center point coordinate pair, traverse the extrinsic parameter calibration parameters, and when it is determined that the center coordinate deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameter of the vehicle-mounted camera to be calibrated.
8. The vehicle-mounted surround view camera calibration method based on feature invariance as claimed in claim 7, It is characterized in that The step 2 also includes: Step 204: Counting the number of pixels of each segmentation target in each of the surround sampling images, and using the number of pixels as the instance area of the corresponding segmentation target, and recording the instance area as the feature value; Step 205: Calculate the area deviation of the instance areas of the two segmented targets in any pair of the center point coordinates, traverse the extrinsic parameter calibration parameters, and when it is determined that the area deviation and / or the center coordinate deviation is the smallest, use the corresponding extrinsic parameter calibration parameters as the optimal extrinsic parameter.
9. The vehicle-mounted surround view camera calibration method based on feature invariance as claimed in claim 6, It is characterized in that The step 2 includes: Step 211: using the network model, extracting features from each of the surround view sampling images in turn to obtain a feature map, and removing similar feature points in each feature map, retaining independent feature points, and recording the independent feature points as the feature values; Step 212: Calculate the similarity of the independent feature points retained between the feature maps to form similar feature point pairs, and calculate the feature deviation of the similar feature point pairs, traverse the extrinsic parameter calibration parameters, and when it is determined that the feature deviation is the smallest, use the corresponding extrinsic parameter calibration parameter as the optimal extrinsic parameter, wherein the optimal extrinsic parameter is used to calibrate the extrinsic parameters of the vehicle-mounted camera to be calibrated.
10. The vehicle-mounted surround view camera calibration method based on feature invariance according to any one of claims 6 to 9, It is characterized in that The center of the vehicle to be calibrated coincides with the center of a circle when the rotating platform rotates.
