A method and system for inter-vehicle static calibration based on surround view cameras

Abstract

The application discloses a kind of inter-vehicle camera-based static calibration method and system of workshop, it is related to target calibration technical field.The method includes the following steps: obtaining the vehicle to be calibrated image obtained by surround view camera, pre-processing to-be-calibrated image;Parameter correction model and distortion correction model are constructed;Parameter correction model is used to correct the parameter of to-be-calibrated image, and calibration image is obtained;Calibration image is processed using distortion correction model, eliminate image distortion, and obtain the calibration image after correction;After registration, fusion and splicing, the final image is obtained after calibration image after correction.This application reduces the cost requirement by using online calibration method, and uses double neural network to optimize the parameters of calibration process, improves the accuracy of camera calibration.

Description

Technical Field

[0001] This invention relates to the field of target calibration technology, and in particular to a static calibration method and system for a workshop based on a surround-view camera. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In image measurement and machine vision applications, to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. The parameters of this geometric model are the camera parameters. However, during camera imaging, lens distortion can occur, affecting the accuracy of the camera parameters and consequently the image generation quality. Since the degree of distortion varies from lens to lens, camera calibration can correct this lens distortion, generating a corrected image from which a 3D scene can be reconstructed.

[0004] The Around View Monitor (AVM) system, as an assisted autonomous driving technology, is an upgrade from the rearview camera system. It uses four wide-angle fisheye cameras and algorithms to synthesize video data, creating a bird's-eye view of the vehicle's surroundings. This 360-degree panoramic, ultra-wide-angle, seamlessly stitched real-time image is then displayed on an in-vehicle screen, achieving blind spot avoidance and safety assistance. To obtain more accurate real-time image information, camera calibration is required before the vehicle leaves the factory. Currently, OEMs require a panoramic image calibration workshop for AVM off-line calibration testing. The AVM system off-line calibration workshop is typically 10-12m long and 6-8m wide, with the vehicle positioned in the center. The cameras capture panoramic images of the vehicle's surroundings. During calibration, the OEM's assembly line operates in a streamlined manner, allowing workshop staff to install guide rails in the calibration area to quickly locate the vehicle in the center before calibration. However, calibration sites occupy a certain amount of space, and a significant amount of money is required to meet site planning and lighting requirements. In addition, the height difference between the guide rails and the calibration interface in the calibration workshop, as well as the potential for misalignment on the on-site road surface, all affect the calibration results.

[0005] Therefore, how to overcome the calibration deficiencies of existing calibration workshops and achieve low-cost, fast, and accurate camera calibration before vehicles leave the factory has become an urgent problem to be solved by existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a static calibration method and system for workshops based on surround-view cameras. The method adopts online calibration to reduce cost requirements and utilizes dual neural networks to optimize parameters in the calibration process, thereby improving the accuracy of camera calibration.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] The first aspect of this invention provides a static calibration method for a workshop based on a surround-view camera, comprising the following steps:

[0009] Acquire the vehicle image to be calibrated from the surround view camera, and preprocess the image to be calibrated;

[0010] Construct parameter correction models and distortion correction models;

[0011] The parameter correction model is used to correct the parameters of the image to be calibrated, and the calibration image is obtained.

[0012] The calibration image is processed using a distortion correction model to eliminate image distortion and obtain a corrected calibration image.

[0013] The final image is obtained by registering, fusing, and stitching the corrected calibration images.

[0014] Furthermore, the specific steps for obtaining the vehicle image to be calibrated from the surround-view camera are as follows:

[0015] The experimental vehicle enters the leveled calibration site;

[0016] Acquire checkerboard images of the experimental vehicle from different positions and angles using multiple surround-view cameras.

[0017] Furthermore, the preprocessing of the images to be calibrated includes image cropping, thresholding, denoising, and normalization.

[0018] Furthermore, the parameter correction model consists of two neural network models: the first neural network model is used to map the pixel coordinates of the image to be calibrated to camera coordinates, and the second neural network model is used to map the camera coordinates to world coordinates.

[0019] Furthermore, the first neural network model is a binary neural network, and the second neural network model is a fully connected neural network.

[0020] Furthermore, the specific steps for processing the calibration image using the distortion correction model are as follows:

[0021] Calculate the camera's internal parameters;

[0022] A deep learning model is used as a distortion correction model to learn the correction relationship between the calibration image and the camera's internal parameters and to correct distortion.

[0023] Furthermore, the specific steps for registering, fusing, and stitching the corrected calibration images to obtain the final image are as follows:

[0024] The calibrated image after correction is subjected to projective transformation to obtain the projective transformation parameters.

[0025] Features are extracted from the overhead view using projective transformation parameters and coarse matching is performed. Initial values ​​of the homography matrix are then fitted.

[0026] Based on the initial value of the homography matrix, multiple images are fused and stitched together using an image perspective transformation algorithm to obtain the final image.

[0027] A second aspect of the present invention provides a workshop static calibration system based on a surround-view camera, comprising:

[0028] The data acquisition module is configured to acquire the vehicle image to be calibrated obtained from the surround view camera and preprocess the image to be calibrated.

[0029] The model building module is configured to build parametric correction models and distortion correction models;

[0030] The camera calibration module is configured to use a parameter correction model to perform parameter correction on the image to be calibrated, thereby obtaining a calibration image.

[0031] The distortion correction module is configured to process the calibration image using a distortion correction model to eliminate image distortion and obtain a corrected calibration image.

[0032] The image stitching module is configured to register, fuse, and stitch the calibrated images to obtain the final image.

[0033] A third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the workshop static calibration method based on a surround-view camera as described in the first aspect of the present invention.

[0034] A fourth aspect of the present invention provides an apparatus including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the workshop static calibration method based on a surround-view camera as described in the first aspect of the present invention.

[0035] The above one or more technical solutions have the following beneficial effects:

[0036] This invention discloses a static calibration method and system for vehicle surround-view cameras. Through multi-step processing of parameter correction and distortion correction, the calibration accuracy of the vehicle surround-view camera can be significantly improved, ensuring the accuracy of subsequent image processing. The distortion correction model can effectively eliminate image distortion, making the corrected calibration image more realistically reflect the actual scene and improving image usability. This invention utilizes a deep learning model for distortion correction, which can learn complex distortion correction relationships and provide better correction results than traditional methods. This invention also uses a combination of binary neural networks and fully connected neural networks, which can effectively handle the mapping from pixel coordinates to camera coordinates and then to world coordinates, improving the accuracy and efficiency of calibration.

[0037] The entire calibration process of this invention is automated through a neural network model, reducing manual intervention and improving the efficiency and consistency of the calibration process. Furthermore, image fusion from multiple surround-view cameras reduces blind spots from individual cameras, enhancing the reliability and security of the entire system. This solution is also adaptable to image acquisition from different locations and angles, exhibiting strong adaptability and flexibility.

[0038] The automated calibration method of this invention reduces labor costs and special site costs, requiring only a flat site. It also improves the repeatability and reproducibility of calibration, resulting in significant economic benefits for large-scale production and application.

[0039] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0041] Figure 1 This is a flowchart of the workshop static calibration method based on a surround-view camera in Embodiment 1 of the present invention. Detailed Implementation

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] Example 1:

[0045] Embodiment 1 of the present invention provides a static calibration method for a workshop based on a surround-view camera, such as... Figure 1 As shown, it includes the following steps:

[0046] Step 1: Obtain the vehicle image to be calibrated from the surround view camera and preprocess the image.

[0047] Step 1.1: Obtain the vehicle image to be calibrated from the surround view camera.

[0048] Step 1.1.1: The experimental vehicle enters the leveled calibration site.

[0049] Step 1.1.2: Acquire checkerboard images of the experimental vehicle from different positions and angles using multiple surround-view cameras.

[0050] Step 1.2: Preprocess the image to be calibrated, including image cropping, thresholding, denoising, and normalization.

[0051] In one specific implementation, the region containing the calibrated target needs to be cropped first to reduce the impact of irrelevant information on subsequent processing.

[0052] The acquired raw image data is then converted into grayscale images, which helps reduce computational complexity and allows us to focus on the texture and shape information of the image.

[0053] Thresholding techniques are used to enhance the contrast of feature points or checkerboard patterns, making the feature points more prominent and facilitating subsequent feature point extraction.

[0054] Image denoising reduces noise and improves image quality. This can be achieved using various filters, such as Gaussian filters.

[0055] Normalizing image data ensures that pixel values ​​fall within a smaller range, which is helpful for subsequent image processing and feature extraction.

[0056] Step 2: Construct the parameter correction model and the distortion correction model.

[0057] Step 2.1: The parameter correction model consists of two neural network models. The first neural network model is used to map the pixel coordinates of the image to be calibrated to camera coordinates, and the second neural network model is used to map the camera coordinates to world coordinates.

[0058] In one specific implementation, the first neural network model is a binary neural network. This model takes the pixel coordinates of the image as input and outputs the corresponding camera coordinates.

[0059] Due to the highly homogenous scene, the high similarity of captured images, numerous limitations, and limited sample size, a binary neural network was chosen as the first neural network. By using binary weights and activation functions, the computational complexity and number of parameters of the model are reduced, making it suitable for edge deployment. In resource-constrained environments, it can learn the mapping relationship from 2D image coordinates to 3D camera coordinates. The binary neural network is a multi-layered feedforward neural network, containing an input layer, hidden layers, and an output layer.

[0060] The second neural network model is a fully connected neural network. This model takes the camera coordinates output from the first network as input and maps them to world coordinates in the world coordinate system. Through multiple layers of neurons and weights, the fully connected neural network can learn the complex mapping relationship from camera coordinates to world coordinates.

[0061] Step 2.2: The distortion correction model is a deep learning model.

[0062] In this embodiment, a training set is constructed by actually capturing images in different scenes and their distortion parameters to train the deep learning model.

[0063] In addition, to enhance the correlation between the parametric correction model and the distortion correction model, the training set is expanded using the original and output images of the parametric correction model along with their distortion parameters. This increases the accuracy of localization in the current field.

[0064] Step 3: Use the parameter correction model to perform parameter correction on the image to be calibrated to obtain the calibration image.

[0065] Step 3.1: Input the preprocessed image to be calibrated into the first neural network. The binary neural network is a multi-layered feedforward neural network, containing an input layer, hidden layers, and an output layer. The input layer receives pixel coordinates, the hidden layers are responsible for extracting and combining features, and the output layer outputs camera coordinates. It learns the non-linear relationship between image pixel coordinates and camera coordinates through training, thereby predicting the camera coordinates corresponding to a given pixel coordinate. Historical images of the current site or synthetic images simulating the current site are used as the training set to train the binary neural network.

[0066] Step 3.2: The fully connected neural network learns the mapping relationship from camera coordinates to world coordinates, enabling it to predict the position in the world coordinate system based on the camera coordinates. The fully connected neural network is a multi-layered feedforward neural network that learns the non-linear relationship between camera coordinates and world coordinates through training, thereby predicting the world coordinates corresponding to a given camera coordinate. Specifically, the output of a binary neural network is used as the training set to train the fully connected neural network.

[0067] Step 4: Use the distortion correction model to process the calibration image, eliminate image distortion, and obtain the corrected calibration image.

[0068] Step 4.1: Calculate camera intrinsic parameters. Calculate camera intrinsic parameters, such as focal length, principal point, and distortion coefficients, using the calibration function in the Camera Calibration Toolbox.

[0069] Step 4.2: Use a deep learning model as a distortion correction model to learn the correction relationship between the calibration image and the camera's internal parameters and perform distortion correction.

[0070] In one specific implementation, the deep learning model includes a distortion line perception module, a line guidance parameter module, and a correction module.

[0071] The distortion line awareness module is used to extract distortion lines from a given pair of calibration images; these lines should be straight in the corrected image. This embodiment uses a pyramid residual module (RPM) and a stacked hourglass network to learn distortion lines from the input image.

[0072] The line-guided parameter estimation module takes the distorted line and calibration image as input and attempts to predict the distortion parameters of the surround-view camera. In this embodiment, a multi-scale perceptron combined with local and global learning is used to remove the nonlinear distortion distribution in the calibration image.

[0073] The correction module acts as a connector between distortion parameters and geometric constraints. It corrects distorted lines to obtain a distortion-free calibration image.

[0074] In this embodiment, existing images and corresponding distortion parameters and warp curves are used as training sets. Furthermore, the images output by the parameter correction module can be used to synthesize corresponding warp curves using synthesis techniques to obtain the corresponding distortion parameters as supplementary data for the training set.

[0075] Step 5: The corrected calibration images are registered, fused, and stitched together to obtain the final image. Through projective transformation and homography matrix calculation, accurate image registration and seamless stitching are achieved, which is crucial for constructing a panoramic view around the vehicle.

[0076] Step 5.1: Perform a projective transformation on the calibrated image after correction to obtain the projective transformation parameters.

[0077] Step 5.2: Extract features from the overhead view using projective transformation parameters and perform coarse matching, and fit the initial value of the homography matrix.

[0078] In one specific implementation, the projective transformation parameters are used to calculate the initial values ​​of the homography matrix. The initial values ​​of the homography matrix are calculated using corresponding point pairs found in the images (e.g., identical corner points in two images). This matrix describes the transformation relationship from one image to another.

[0079] Specifically, the coarse matching process refers to finding matching feature point pairs by comparing feature points in different images after feature extraction. Once matching point pairs are found, they can be used to calculate the initial value of the homography matrix. Generally, at least four matching point pairs are needed to calculate the homography matrix. This can be achieved using the `findHomography` function in OpenCV, which accepts a set of points from both the source and target images and calculates the homography matrix.

[0080] The initial value of the homography matrix is ​​used as an input parameter in image registration, fusion, and stitching processes. This matrix describes how points in one image are transformed into the coordinate system of another image, thereby achieving image alignment and stitching. In practical applications, this matrix can be used in the `warpPerspective` function, which performs perspective transformation on images based on the homography matrix, thereby achieving image correction and stitching.

[0081] Step 5.3: Based on the initial value of the homography matrix, use the image perspective transformation algorithm to fuse and stitch together multiple images to obtain the final image.

[0082] In one specific implementation, during image registration, fusion, and stitching, projective transformation parameters (homography matrix) are used as input to transform images from multiple perspectives into a unified coordinate system, thereby achieving seamless image stitching. This process involves perspective transformation of the images, that is, transforming points in one image to the coordinate system of another image according to the homography matrix.

[0083] Example 2:

[0084] Embodiment 2 of the present invention provides a workshop static calibration system based on a surround-view camera, comprising:

[0085] The data acquisition module is configured to acquire the vehicle image to be calibrated obtained from the surround view camera and preprocess the image to be calibrated.

[0086] The model building module is configured to build parametric correction models and distortion correction models;

[0087] The camera calibration module is configured to use a parameter correction model to perform parameter correction on the image to be calibrated, thereby obtaining a calibration image.

[0088] The distortion correction module is configured to process the calibration image using a distortion correction model to eliminate image distortion and obtain a corrected calibration image.

[0089] The image stitching module is configured to register, fuse, and stitch the calibrated images to obtain the final image.

[0090] Example 3:

[0091] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, it implements the steps in the workshop static calibration method based on a surround-view camera as described in Embodiment 1 of the present invention.

[0092] Example 4:

[0093] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the workshop static calibration method based on a surround-view camera as described in Embodiment 1 of the present invention.

[0094] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0095] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0096] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A static calibration method for a workshop based on a surround-view camera, characterized in that, Includes the following steps: Acquire the vehicle image to be calibrated from the surround view camera, and preprocess the image to be calibrated; Construct parameter correction models and distortion correction models; The parameter correction model consists of two neural network models. The first neural network model is a binary neural network, which is used to map the pixel coordinates of the image to be calibrated to camera coordinates and is deployed on the edge. The second neural network model is a fully connected neural network, which is used to map the camera coordinates to world coordinates. The parameter correction model is used to correct the parameters of the image to be calibrated, and the calibration image is obtained. The calibration image is processed using a distortion correction model to eliminate image distortion and obtain a corrected calibration image. The final image is obtained by registering, fusing, and stitching the corrected calibration images.

2. The workshop static calibration method based on a surround-view camera as described in claim 1, characterized in that, The specific steps for obtaining the vehicle image to be calibrated from the surround-view camera are as follows: The experimental vehicle enters the leveled calibration site; Acquire checkerboard images of the experimental vehicle from different positions and angles using multiple surround-view cameras.

3. The workshop static calibration method based on a surround-view camera as described in claim 1, characterized in that, Preprocessing of the image to be calibrated includes image cropping, thresholding, denoising, and normalization.

4. The workshop static calibration method based on a surround-view camera as described in claim 1, characterized in that, The specific steps for processing the calibration image using the distortion correction model are as follows: Calculate the camera's internal parameters; A deep learning model is used as a distortion correction model to learn the correction relationship between the calibration image and the camera's internal parameters and to correct distortion.

5. The workshop static calibration method based on a surround-view camera as described in claim 1, characterized in that, The specific steps for registering, fusing, and stitching the corrected calibration images to obtain the final image are as follows: The calibrated image after correction is subjected to projective transformation to obtain the projective transformation parameters. Features are extracted from the overhead view using projective transformation parameters and coarse matching is performed. Initial values ​​of the homography matrix are then fitted. Based on the initial value of the homography matrix, multiple images are fused and stitched together using an image perspective transformation algorithm to obtain the final image.

6. A workshop static calibration system based on surround-view cameras, characterized in that, include: The data acquisition module is configured to acquire the vehicle image to be calibrated obtained from the surround view camera and preprocess the image to be calibrated. The model building module is configured to build a parameter correction model and a distortion correction model. The parameter correction model consists of two neural network models. The first neural network model is a binary neural network, which is used to map the pixel coordinates of the image to be calibrated to camera coordinates and is deployed on the edge. The second neural network model is a fully connected neural network, which is used to map the camera coordinates to world coordinates. The camera calibration module is configured to use a parameter correction model to perform parameter correction on the image to be calibrated, thereby obtaining a calibration image. The distortion correction module is configured to process the calibration image using a distortion correction model to eliminate image distortion and obtain a corrected calibration image. The image stitching module is configured to register, fuse, and stitch the calibrated images to obtain the final image.

7. A computer-readable storage medium, characterized in that, It stores multiple instructions, which are adapted to be loaded and executed by the processor of the terminal device, according to any one of claims 1-5, the workshop static calibration method based on a surround-view camera.

8. A terminal device, characterized in that, It includes a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being adapted to be loaded by the processor and executed by the processor for the workshop static calibration method based on any one of claims 1-5.

Images