Convenient AVM calibration and correction method based on fisheye camera

Abstract

The invention relates to the field of intelligent driving, and discloses a convenient AVM calibration and correction method based on a fisheye camera, and the method comprises the steps: obtaining an original calibration image, and carrying out the internal reference calibration and distortion removal, and obtaining a corrected image; selecting fisheye cameras in all directions on a calibration interface, selecting feature points according to the schematic diagram, and then performing splicing fusion processing; the method comprises the following steps of: firstly generating a pre-spliced panorama, executing fusion processing after the pre-spliced panorama meets requirements, and otherwise, performing adjustment on a calibration interface; and a direct splicing and fusion processing mode: directly carrying out splicing and fusion to obtain a panoramic fusion image, judging whether a fusion effect meets requirements or not, and if the fusion effect does not meet the requirements, adjusting feature point parameters on a calibration interface. The method can solve the problems that a traditional method is difficult to calibrate and the adjustment operation of a calibration result is complex, and improves the calibration efficiency and precision.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a convenient AVM calibration and correction method based on a fisheye camera. Background Art

[0002] With the development of intelligent driving technology and the advent of the software-defined car era, the use of Around View Monitor (AVM) systems in vehicles is becoming increasingly widespread. AVM uses cameras distributed around the vehicle to capture images and stitch multiple images into a single panoramic image, providing the driver with a complete, panoramic view of the vehicle's surroundings. This improves the driver's perception of the vehicle's surroundings during parking, driving in narrow sections, and in complex road conditions, effectively reducing traffic accidents caused by blind spots and significantly enhancing driving safety and convenience.

[0003] Although AVM has significant advantages, due to factors such as the camera installation position, angle, and lens distortion, the directly captured images have problems with geometric deformation and spatial inconsistency, and cannot be directly stitched into an ideal panoramic image. Therefore, accurate camera calibration becomes a key prerequisite for achieving high-quality panoramic images.

[0004] At present, the existing calibration methods have the following limitations in practical applications: (1) Dependence on a specific site: It is necessary to lay special calibration patterns such as high-precision checkerboards and concentric circle arrays on a specific flat site. Taking checkerboard calibration as an example, the size and angle of the squares must strictly comply with the design standards, and the checkerboard plane must be absolutely horizontal when laying. This means that calibration work can only be carried out in specific environments such as laboratories and professional workshops, and is difficult to implement in complex scenarios such as roads and parking lots where vehicles actually drive. (2) The calibration process is cumbersome and complicated: the operator needs to adjust the camera position and angle many times, repeatedly collect images, and then solve the parameters through complex mathematical calculations and image processing algorithms. The entire process is not only time-consuming, but also requires extremely high professional knowledge and skills of the operator, and cannot meet the needs of rapid calibration in actual vehicle use scenarios. (3) The adjustment link of the calibration results is complicated: During the feature point calibration process, since the operator needs to mark, the operator's subjective judgment and operation errors can easily lead to inaccurate calibration points. Therefore, after the calibration is completed, it is often necessary to fine-tune according to the actual effect. However, in the existing adjustment process, operators can only rely on experience to repeatedly modify the feature point parameters and then re-verify them. This is not only complicated to operate, but also difficult to accurately control the adjustment range, which can easily lead to over-correction or under-correction, thus affecting the imaging quality and debugging efficiency of the panoramic imaging system. Summary of the Invention

[0005] The present invention aims to provide a convenient AVM calibration and correction method based on a fisheye camera to solve the problems of difficult calibration in traditional methods and complex adjustment operations of the calibration results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a convenient AVM calibration and correction method based on a fisheye camera, comprising: Step S100: obtaining the original calibration images of each fisheye camera, and performing internal calibration and dedistortion to obtain the corrected images; Step S200: Enter the calibration interface, select the fisheye cameras corresponding to the front, back, left, and right directions respectively, and select feature points in the calibration images currently captured by each fisheye camera in real time according to the corresponding schematic diagram; Step S300: performing stitching and fusion processing on the selected views; this includes the following steps: Step S310: splicing first and then fusion processing; Step S311: preliminarily stitching the BEV views generated by different fisheye cameras to obtain a pre-stitched panoramic image, and displaying the pre-stitched panoramic image in real time on the calibration interface; Step S312: Determine whether the pre-stitching result meets the requirements. If not, adjust the feature point parameters in the calibration interface until the stitching effect meets the requirements. If it meets the requirements, perform fusion processing. Step S320: direct splicing and fusion processing; Step S321: Preliminary stitching of the BEV views generated by different fisheye cameras; Step S322: Execute fusion processing to obtain a panoramic fusion image and determine whether the fusion effect meets the requirements. If not, adjust the feature point parameters in the calibration interface until the fusion effect meets the requirements.

[0007] The principle of this solution is as follows: In practical application, the AVM system first acquires raw calibration images captured by fisheye cameras in real time and performs internal calibration and dedistortion to obtain corrected images. The calibration interface then opens. The user selects the fisheye cameras corresponding to the front, back, left, and right directions and selects feature points from the corrected images captured by each fisheye camera in real time according to the diagram on the interface. After feature point selection, the system performs stitching and fusion. This stitching and fusion process includes two methods: stitching-first, then fusion and direct stitching and fusion. 1) Stitching-first, then fusion: The system performs a preliminary stitching of the BEV views generated by the different fisheye cameras to generate a pre-stitched panorama, which is displayed in real time on the calibration interface. The user can intuitively assess the stitching results. If any issues such as image discontinuity or misalignment are detected, the user can directly adjust the feature point positions on the calibration interface, and the adjusted stitching effect is displayed in real time until the stitching meets the requirements. Once the stitching meets the requirements, the fusion process is performed to achieve a natural transition between the images. 2) Direct stitching and fusion: After completing the initial stitching, the system immediately performs fusion processing to generate a complete panoramic fusion image and displays it in real time on the calibration interface. Users can intuitively judge the final fusion effect. If there are obvious seams in the fusion area, the position of the feature points can be directly adjusted on the calibration interface, and the adjusted fusion effect is displayed in real time until the fusion effect meets the requirements.

[0008] In summary, this solution simplifies the calibration process, solves the problems of difficult calibration and complex adjustment operations in traditional methods, and improves calibration efficiency and accuracy.

[0009] The advantages of this solution are: (1) This solution breaks through the traditional "black box" adjustment operation logic and visualizes the feature point selection and feature point parameter adjustment process. Users can view the splicing or fusion effect in real time on the calibration interface and directly adjust the feature point position based on image feedback. This "what you see is what you get" approach significantly improves the transparency and controllability of the operation, thereby improving the accuracy and efficiency of the calibration result correction.

[0010] (2) This solution links calibration, stitching, and fusion. When adjusting feature points in the calibration interface, the pre-stitched panorama or panoramic fusion image will show the impact of the adjustment on stitching and fusion in real time, thereby reducing the cost of trial and error.

[0011] (3) This solution provides two processing methods: "stitching first and then fusion" and "direct stitching and fusion". Users can choose the most appropriate process according to actual needs, enhance the flexibility and adaptability of the system, and improve the calibration efficiency in different scenarios.

[0012] (4) Internal reference calibration and dedistortion can effectively correct the distortion of the original calibration image and restore the true geometric relationship, thereby providing a high-quality image foundation for subsequent panoramic stitching and display.

[0013] Preferably, as an improvement, step S100 includes the following steps: Step S110: Select appropriate calibration plates and place them around the vehicle; Step S120: Using fisheye cameras around the vehicle to shoot the calibration plate at different angles and positions to obtain original calibration images; Step S130: extracting feature points from each original calibration image; Step S140: Calculating the intrinsic parameter matrix and distortion coefficient of each fisheye camera using the distortion model and Zhang Zhengyou calibration method; Step S150: Based on the intrinsic parameter matrix and the distortion coefficient, the collected original calibration image is dedistorted to obtain a corrected image.

[0014] Beneficial Effects: Steps S110-S150 can improve the accuracy and image quality of the calibration process, laying the foundation for subsequent stitching and fusion processing. Dedistortion processing can effectively eliminate the distortion caused by the fisheye lens, making the image closer to the real scene, thereby improving the visual presentation quality of the AVM system. The corrected image obtained by dedistortion provides a reliable data foundation for stitching and fusion of multi-view images and BEV generation.

[0015] Preferably, as an improvement, the calibration plate is a checkerboard, and the original calibration image needs to meet the following conditions during the shooting process: All the chessboards in the photos need to cover the entire shooting screen and need to cover different angles; A single checkerboard image occupies about 1 / 4 to 1 / 3 of the shooting image; The checkerboard pattern must be fully displayed, with at least half a square of blank border left at the edge. Choose a checkerboard calibration cloth made of non-reflective material.

[0016] Beneficial effects: (1) The checkerboard structure is clear and the corners are easy to identify, which can provide high-quality feature point data for the calibration algorithm. Covering the entire screen and including multi-angle images helps to improve the calibration model's calculation accuracy for distortion and internal parameters. (2) Shooting at different angles can obtain rich calibration data, thereby ensuring the representativeness of the calibration data. (3) A single checkerboard occupies 1 / 4 to 1 / 3 of the screen, which not only ensures that the checkerboard details are clearly visible, but also does not limit the camera's effective field of view due to excessive proportions, which helps to obtain a more accurate spatial mapping relationship. (4) The checkerboard pattern is fully displayed and sufficient blank borders are retained at the edges, which can prevent feature point recognition errors caused by cropping or occlusion, thereby improving the reliability of the calibration results. (5) Selecting a checkerboard calibration cloth made of non-reflective material prevents the increase of image noise due to reflections, and avoids corner point recognition failures caused by overexposure or highlight areas, ensuring that the image is clear and the contrast is moderate, thereby improving the reliability of the calibration results.

[0017] Preferably, as an improvement, the number of the feature points is 4.

[0018] Benefits: A small and appropriate number of feature points makes it easier to accurately identify and match. Especially in complex environmental conditions, such as those with changing lighting or partial occlusion, fewer feature points can maintain higher stability and reliability. Furthermore, an appropriate number of feature points ensures sufficient information for accurate calibration while avoiding redundant data or noise interference introduced by too many feature points, thereby improving the accuracy of the final calibration results.

[0019] Preferably, as an improvement, step S200 includes the following steps: Step S210: Connect the relevant calibration equipment to the vehicle's OBD port to enter the calibration interface; The calibration interface is divided into three areas from left to right: an AVM view area for displaying stitching and fusion effects, a fisheye acquisition area for displaying the current correction image acquisition status of the four-way fisheye camera, and a feature point management area for calibrating and adjusting feature points.

[0020] The AVM view area includes a panoramic BEV view and a fusion button for triggering multi-view fusion, and the fusion button is located in the upper right corner; The fisheye acquisition area displays the corrected image pictures currently captured in real time by the four fisheye cameras of front view, rear view, left view and right view in sequence from top to bottom; The feature point management area includes, from top to bottom, a feature point selection area for selecting a view to be adjusted and a feature point position adjustment area for fine-tuning feature point coordinates.

[0021] Beneficial Effects: By dividing the calibration interface into different areas, users can quickly find the required operation options, reducing search and switching time and improving work efficiency. The feature point position adjustment area provides detailed coordinate fine-tuning functions, ensuring that each feature point is accurately placed, thereby improving the quality of the final stitching and fusion. At the same time, the division design of the calibration interface lays the foundation for subsequent "what you see is what you get" operations.

[0022] Preferably, as an improvement, the feature point selection area includes, from top to bottom, four views to be adjusted, namely, front view, rear view, left view and right view, and a stitching button, and the stitching button is located in the lower left corner; The view to be adjusted corresponds to the fisheye real-time acquisition area; The feature point position adjustment area includes operation buttons for selecting a feature point value of a specific feature point, inputting a shift value of a pixel offset, and controlling the moving direction of the feature point; The operation buttons include four buttons: up, down, left and right.

[0023] Beneficial effects: Feature point values, shift values and operation buttons allow users to adjust the position of each feature point very accurately, thereby ensuring the high-precision requirements of image stitching and fusion. The view to be adjusted corresponds to the real-time acquisition area of the fisheye, ensuring that users can accurately adjust the corresponding feature points while viewing the real-time image, enhancing the consistency and coherence of the operation.

[0024] Preferably, as an improvement, step S200 further includes the following steps: Step S220: introducing perspective transformation to convert the corrected image captured in real time by each fisheye camera into an independent BEV view; Step S230: Select the view to be adjusted in the feature point selection area to enter the corresponding view interface, and click to select the corresponding actual point on the correction image screen captured in real time by the fisheye camera according to the position indication of the feature points listed in the view interface.

[0025] Beneficial Effects: Perspective transformation converts the corrected image into a BEV view, making it more similar to the driver's "bird's-eye view." This helps users more intuitively understand the spatial relationships between images in different directions, thereby improving the accuracy and efficiency of spatial judgment during the calibration process. By introducing perspective transformation and feature point selection, this solution not only enhances the image's spatial representation and user-friendly operation, but also significantly improves calibration accuracy and interaction efficiency. It also provides a high-quality data foundation for subsequent image stitching and fusion.

[0026] Preferably, as an improvement, the view interface is divided into: a fisheye real-time acquisition area and a schematic diagram area; the schematic diagram area includes a schematic diagram for displaying a feature point position indication and a distance parameter; The distance parameters include d1, d2 and d3; d1 represents the width of the calibration plate; d2 represents the distance between the front side of the vehicle and the calibration plate; and d3 represents the distance between the left side of the vehicle and the calibration plate.

[0027] Beneficial Effects: The fisheye real-time acquisition area and schematic area enhance the visualization, ease of operation, and accuracy of the calibration process. Its d1, d2, and d3 allow the AVM system to adapt to calibration plates of varying sizes and different vehicle platforms, enhancing the system's versatility and scalability, reducing its reliance on specific sites and facilitating its use in diverse and complex scenarios.

[0028] Preferably, as an improvement, the number of the original calibration images is 9-15.

[0029] Beneficial effect: 9 to 15 original calibration images can provide sufficient information to accurately calculate camera parameters without causing excessive computational complexity, ensuring processing speed and efficiency.

[0030] Preferably, as an improvement, one unit in the shift value represents one pixel point Beneficial effect: Each unit corresponds to a pixel point, which can ensure that the position of the feature point is accurately placed, thereby improving the quality of the final image stitching and fusion.

[0031] The beneficial effects of this scheme are as follows: (1) This scheme breaks through the "black box" calibration thinking in traditional methods through the division of the visual calibration interface and the integrated process of original calibration image acquisition, internal reference calibration, dedistortion to final splicing and fusion, and can implement a "what you see is what you get" calibration interaction method.

[0032] (2) Feature point position adjustment area allows users to adjust key spatial parameters such as d1, d2, and d3 in real time according to the on-site layout, breaking the limitations of traditional reliance on fixed calibration sites.

[0033] (3) It has two display and adjustment modes: "splicing first and then merging" and "direct splicing and fusion". It can be switched according to actual needs to enhance the flexibility of the system.

[0034] (4) This solution uses a combination of feature point extraction, distortion modeling, Zhang Zhengyou calibration, and perspective transformation techniques to enable the system to accurately calibrate and correct the calibration images acquired by the fisheye camera in a short period of time, meeting the requirements for rapid response and real-time dynamic adjustment in intelligent driving. Efficient image processing not only improves calibration efficiency but also enhances the system's ability to perceive the vehicle's surroundings during parking, driving in narrow sections, and complex road conditions, effectively reducing traffic accidents caused by blind spots and significantly enhancing driving safety and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention provides a convenient flow chart of an AVM calibration and correction method based on a fisheye camera.

[0036] Figure 2 A structural schematic diagram of the calibration plate position in a convenient fisheye camera-based AVM calibration and correction method provided by an embodiment of the present invention.

[0037] Figure 3 A schematic structural diagram of the marking interface in a convenient fisheye camera-based AVM calibration and correction method provided by an embodiment of the present invention.

[0038] Figure 4 for Figure 3 A partial schematic diagram of .

[0039] Figure 5 A schematic structural diagram of introducing perspective transformation into a convenient fisheye camera-based AVM calibration and correction method provided by an embodiment of the present invention.

[0040] Figure 6 Schematic diagram of the front view interface in an embodiment of the present invention.

[0041] Figure 7 Schematic diagram of the rear view interface in an embodiment of the present invention.

[0042] Figure 8 It is a structural diagram of the left view interface in an embodiment of the present invention.

[0043] Figure 9 It is a structural diagram of the right view interface in an embodiment of the present invention.

[0044] Figure 10 This is a structural diagram showing that the splicing effect does not meet the requirements in an embodiment of the present invention.

[0045] Figure 11 for Figure 10 Schematic diagram of the stitching correction structure.

[0046] Figure 12for Figure 11 Schematic diagram of the fusion processing structure.

[0047] Figure 13 This is a structural diagram showing that the fusion effect does not meet the requirements in an embodiment of the present invention.

[0048] Figure 14 for Figure 13 Schematic diagram of the fusion correction structure. DETAILED DESCRIPTION

[0049] The following is further described in detail through specific implementation methods: The embodiment is basically as follows Figure 1 Shown: A convenient AVM calibration and correction method based on fisheye camera, including Step S100: obtaining the original calibrated images of each fisheye camera, and performing internal parameter calibration and dedistortion to obtain a corrected image.

[0050] Step S100 includes the following steps: Step S110: Select appropriate calibration plates and place them around the vehicle; Specifically, such as Figure 2 As shown, four calibration plates with checkerboard-like geometric features are selected and placed horizontally on the ground around the vehicle at the upper left, lower left, upper right, and lower right sides. This ensures that the plates are clearly visible within the field of view of each fisheye camera (front, rear, left, and right), and that adjacent fisheye cameras have overlapping fields of view. In this example, the checkerboard grid is 4x4, with each cell measuring 50cm x 50cm. (Note: There is no specific requirement for the number of checkerboard cells; all cells can be the same.)

[0051] The calibration plate is used to calibrate the AVM system, providing a standard reference and a unified, reusable benchmark. This calibration plate helps the panoramic host perform significant parameter corrections during calibration, effectively eliminating objective errors introduced by the environment or the device itself, thereby improving the quality and consistency of panoramic imaging.

[0052] Step S120: Using fisheye cameras around the vehicle to shoot the calibration plate at different angles and positions to obtain original calibration images; Specifically, fisheye cameras are installed on the front, rear, left, and right sides of the vehicle, ensuring that the field of view of each fisheye camera covers the area surrounding the vehicle, and that the fields of view of adjacent fisheye cameras overlap to some extent. In this embodiment, the fisheye cameras used are model Senyun SG2-AR0233C-5200-G2A, with a focal length of 1.51mm, a field of view of 196°, and an image resolution of 1920*1080. The left fisheye camera is located in the center of the left door, at the same height as the rearview mirror. The right fisheye camera is located in the center of the right door, at the same height as the rearview mirror. The front fisheye camera is located above the center of the front bumper, and the rear fisheye camera is located above the center of the rear bumper.

[0053] In addition, in order to meet the algorithm's requirements for calibration precision and accuracy, it is necessary to collect 9 to 15 original calibration images from different angles and positions to obtain sufficient feature point data, thereby improving the accuracy and stability of the calibration results. During the image capture process, the following conditions need to be met: (1) The positions of all photographed checkerboards need to cover the entire shooting screen and need to cover different angles. The positions of all angles and directions should be as uniform as possible to ensure the representativeness of the calibration data.

[0054] (2) A single checkerboard grid occupies about 1 / 4 to 1 / 3 of the shooting screen, which ensures that the details of the checkerboard grid are clearly visible while avoiding the problem of being too small or too large to affect the recognition effect.

[0055] (3) The photographed checkerboard pattern must be displayed completely, and a blank border of at least half a checkerboard should be retained at the edge to prevent feature point recognition errors caused by cropping or occlusion, so that the image processing algorithm can accurately extract feature points, thereby improving the reliability of the calibration results.

[0056] (4) Selecting a checkerboard calibration cloth made of non-reflective material can prevent the increase of image noise due to reflections and avoid the failure of corner recognition caused by overexposure or highlight areas, ensuring clear images with moderate contrast, thereby improving the reliability of the calibration results.

[0057] In this embodiment, calibration efficiency and accuracy are significantly improved through the design of the capture process (e.g., parameters such as capture angle, material, and frame coverage) and the number of captures (reducing the number of captures while ensuring accuracy). This allows for rapid calibration while ensuring accuracy, meeting the real-time and convenience requirements of actual vehicle usage scenarios. It is suitable for situations requiring frequent calibration or rapid on-site calibration.

[0058] Step S130: extracting feature points from each original calibration image; Specifically, for each original calibration image, a corner detection algorithm (such as OpenCV's FindChessboardCorners) is used to detect the pixel coordinates of the chessboard corners. World coordinates are assigned to each corner (usually with the calibration plane set to Z=0), and a one-to-one correspondence between the image coordinates and the world coordinates of the calibration plane is ensured. This establishes a correspondence between the image coordinates of the feature points and the world coordinates of the calibration plane, providing a basis for the subsequent calculation of the camera's intrinsic parameters.

[0059] Step S140: Calculate the intrinsic parameter matrix and distortion coefficient of each fisheye camera using the distortion model and Zhang Zhengyou calibration method.

[0060] Specifically, the distortion coefficients include radial and tangential distortion coefficients. Based on the correspondence between image coordinates and world coordinates in multiple calibration images and the distortion characteristics of fisheye cameras, the Kannala-Brandt model and Zhang Zhengyou calibration method are used to calculate the intrinsic parameter matrix and distortion coefficients of each fisheye camera, thereby minimizing the reprojection error.

[0061] Sources of distortion: Deviations in camera lens manufacturing precision (such as curvature deviation) and assembly processes (such as optical axis offset) can cause varying degrees of radial distortion (barrel / pincushion distortion) and tangential distortion (the lens is not parallel to the imaging plane), resulting in distorted images. Fisheye cameras, in particular, introduce significant distortion during capture due to their ultra-wide-angle capabilities.

[0062] Step S150: Based on the intrinsic parameter matrix and the distortion coefficient, the collected original calibration image is dedistorted to obtain a corrected image.

[0063] Specifically, the original calibration image captured by the fisheye camera is dedistorted using the obtained intrinsic parameter matrix and distortion coefficients. If the dedistortion effect is not good (for example, the edges are still bent or stretched), repeat the above steps to recalculate the intrinsic parameters and dedistortion to avoid poor subsequent AVM image stitching effects.

[0064] S200: Enter the calibration interface, select the fisheye cameras corresponding to the front, back, left, and right directions respectively, and select feature points in the calibration images currently captured by each fisheye camera in real time according to the corresponding schematic diagram; Specifically, the following steps are included: S210: Connect the relevant calibration equipment to the vehicle's OBD port and enter the calibration interface; Specifically, connect the fisheye camera to the industrial computer and enter the calibration interface. Figure 3 、 Figure 4As shown in the figure, the calibration interface is divided into three areas from left to right: the AVM view area for displaying the stitching effect and the fusion effect, the fisheye acquisition area for displaying the current correction image acquisition status of the four-way fisheye camera, and the feature point management area for calibrating and adjusting feature points.

[0065] The AVM view area includes a panoramic BEV view and a fusion button for triggering multi-view fusion, and the fusion button is located in the upper right corner.

[0066] The fisheye acquisition area displays the corrected fisheye images captured in real time by the four fisheye cameras of front view, rear view, left view and right view in a split screen from top to bottom.

[0067] From top to bottom, the feature point management area includes a feature point selection area for selecting the view to be adjusted and a feature point position adjustment area for fine-tuning the feature point coordinates. Specifically, the feature point selection area includes four views to be adjusted: front view, rear view, left view, and right view, as well as a stitching button in the lower left corner. The four views to be adjusted correspond to the acquisition status of the fisheye acquisition area. The feature point position adjustment area includes feature point values for selecting specific feature points, shift values for entering feature point pixel offset values, and operation buttons for controlling the movement direction of the feature points. The operation buttons include four buttons for controlling the movement direction: up, down, left, and right.

[0068] Step S220: introducing perspective transformation to convert the corrected image captured in real time by each fisheye camera into an independent BEV view; Specifically, such as Figure 5 As shown in the figure, the perspective transformation is introduced to convert the corrected fisheye images collected by each fisheye camera in real time into independent BEV views, and to convert the corrected images to the area responsible for the camera in each direction according to the user-selected viewing angle, so that the user can flexibly select the required observation angle and adjust the corresponding surfaces in the front, back, left and right under the viewing angle to be displayed directly in front, which is convenient for intuitive parameter adjustment and achieves better fusion effect.

[0069] Step S230: Select the view to be adjusted in the feature point selection area to enter the corresponding view interface, and click to select the corresponding actual point on the correction image screen captured in real time by the fisheye camera according to the position indication of the feature points listed in the view interface.

[0070] Specifically, such as Figure 6As shown in the figure, taking the front view as an example, select the front view in the feature point selection area to enter the corresponding view interface. The view interface is divided into two areas: the fisheye real-time acquisition area and the schematic area. The schematic area includes a schematic diagram and distance parameters. Next, according to the location of feature points P1, P2, P3, and P4 listed in the schematic diagram, accurately click and select the corresponding actual points on the calibrated fisheye screen in the fisheye acquisition area. By determining the positions of these points in the image, a correspondence between image coordinates and real-world coordinates can be established. Then, fill in the actual distances d1, d2, and d3 in the distance parameter field. These distances serve as important parameters for subsequent calculations and corrections, ensuring accurate image stitching and correction. d1 represents the width of the calibration plate; adjusting d1 changes the plate's size; d2 represents the distance between the front of the vehicle and the calibration plate; and d3 represents the distance between the left side of the vehicle and the calibration plate. By combining the points in the image with the actual distances, the fisheye camera can be calibrated and corrected more accurately, improving the accuracy and reliability of the AVM system.

[0071] Similarly, the selection operations for other view feature points are the same, specifically, Figure 7 (Rear view), Figure 8 (left view), Figure 9 (right view) as shown.

[0072] In this solution, the parameters of d1, d2, and d3 can be set dynamically, making it possible to flexibly adjust the size and angle of the calibration plate, greatly reducing the requirements for the calibration site, and allowing calibration work to be carried out in a wider variety of environments (such as roads where vehicles actually travel, parking lots, and other responsible environments), thereby effectively saving costs.

[0073] Step S300: performing stitching and fusion processing on the selected views; this includes the following steps: Step S310: Splicing first and then fusion processing Step S311: preliminarily stitching the BEV views generated by different fisheye cameras to obtain a pre-stitched panoramic BEV view, and displaying the pre-stitched BEV view in real time on the calibration interface; Specifically, such as Figure 10 As shown, select the left view in the fisheye real-time acquisition area and click the stitching button to perform the stitching operation. The BEV views generated by the front, rear, left, and right fisheye cameras are aligned to the same coordinate system according to the calibrated internal parameters for preliminary stitching, and a pre-stitched panoramic BEV view with the perspective displayed directly in front is obtained in the AVM view area.

[0074] Step S312: Determine whether the pre-stitching result meets the requirements. If not, adjust the feature point parameters in the calibration interface until the stitching effect meets the requirements. If it meets the requirements, perform fusion processing. Specifically, the pre-stitched panoramic BEV view is tested for stitching effect, focusing on whether the stitching seams are continuous and whether the images are aligned. Figure 10 As shown in the figure, the image in the upper left position is obviously offset to the right, that is, the position of the feature point P1 is deviated. In order to correct this problem, the image needs to be adjusted to the left. During the operation, enter the feature point P1 to be corrected in the feature point input box of the calibration interface, and enter 2 in the displacement value to indicate the pixel offset for each adjustment. Then click the left shift button to gradually adjust the position of P1 to the left until the edges and corners of the checkerboard calibration plate can be aligned. Figure 11 As shown, during the adjustment process, the AVM view area synchronously updates the stitching effect, displaying the adjustment status in real time. At this point, it can be clearly seen that the checkerboard edges are completely aligned, indicating that the adjustment is complete. In this embodiment, each click of the left shift button shifts P1 to the left by 2 pixels.

[0075] like Figure 12 As shown in the figure, after all stitching adjustments are completed, click the "Fusion" button and the system will fuse the stitched images. During the fusion process, the BEV view of each fisheye camera accounts for 50% of the total, and the fused image is displayed in the AVM view area, with the view as the front direction, ensuring that users can intuitively view the seamless and naturally transitioned panoramic fused image.

[0076] Step S320: Direct splicing and fusion processing Step S321: Preliminary stitching of the BEV views generated by different fisheye cameras; Step S322: Execute fusion processing to obtain a panoramic fusion image and determine whether the fusion effect meets the requirements. If not, adjust the feature point parameters in the calibration interface until the fusion effect meets the requirements.

[0077] Specifically, such as Figure 13 As shown, select the right view in the fisheye real-time acquisition area and click the stitching button to perform the stitching operation. The BEV views generated by the front, rear, left and right fisheye cameras are aligned to the same coordinate system according to the calibrated internal parameters for preliminary stitching, and a pre-stitched panoramic BEV view with the perspective displayed directly in front is obtained in the AVM view area. Then, click the fusion button to perform fusion, and a panoramic fused view with the perspective displayed directly in front is obtained in the AVM view area.

[0078] Then, the fusion effect of the panoramic fusion view is tested, focusing on whether the fusion seam is continuous and whether the images are aligned. Figure 13 As shown in the figure, the image in the upper right position is obviously offset to the left, that is, the position of the feature point P2 is deviated. In order to correct this problem, the image needs to be adjusted to the right. During the operation, select the feature point P2 to be corrected in the feature point input box of the calibration interface, and enter 4 in the displacement value to indicate the pixel offset for each adjustment. Then click the right shift button to gradually adjust the position of P2 to the right until the edges and corners of the checkerboard calibration plate can be aligned. Figure 14 As shown, during the adjustment process, the AVM view area synchronously updates the fusion effect, displaying the adjustment status in real time. It can be clearly seen that the checkerboard corners are perfectly aligned, and image ghosting is very low, indicating that the adjustment is complete. In this embodiment, each click of the left shift button shifts P2 to the right by 4 pixels.

[0079] In summary, with traditional methods, when stitching or fusion errors occur, operators rely on experience to repeatedly modify the parameters of the feature points in the calibration image according to the schematic diagram, and then re-stitch, display, and verify. This correction process is cumbersome and highly subjective, resulting in a calibration accuracy rate of 60%.

[0080] In contrast, this solution introduces a visual calibration interface to display the fusion and stitching effects in real time, allowing operators to adjust and modify relevant parameters on the calibration interface based on intuitive image feedback, greatly improving the accuracy and efficiency of operations. The calibration accuracy rate is over 90%.

[0081] This solution uses a graphical interactive interface (i.e., the calibration interface and the view interface) and the linkage between the calibration, stitching, and fusion links to allow operators to visually adjust feature points in the feature point management area directly based on the panoramic BEV view displayed in the AVM view area (such as selecting an adjustment feature point and clicking an operation button to move the feature point position) without having to return to the step of "selecting feature points through a schematic diagram." This eliminates multiple intermediate steps in the traditional process and significantly reduces the difficulty of operation. This solution breaks through the traditional "black box" adjustment operation logic and visualizes the originally abstract feature point selection and parameter adjustment process. Users can view the stitching or fusion effect in real time on the calibration interface and directly adjust the feature point position based on image feedback. This "what you see is what you get" approach significantly improves the transparency and controllability of the operation and reduces the cost of trial and error. It not only improves the transparency of the calibration process, but also improves the accuracy and efficiency of the calibration result correction.

[0082] Furthermore, this solution establishes an efficient calibration process by integrating real-time calibration image acquisition, feature point extraction, distortion modeling, Zhang Zhengyou calibration method, perspective transformation, and calibration interface design. The system can accurately calibrate and correct calibration images acquired by the fisheye camera in a short period of time, significantly improving the real-time performance and response speed of the calibration process, meeting the requirements of intelligent driving for rapid response and real-time dynamic adjustment in complex operating conditions.

[0083] At the same time, this solution presents the image acquisition effect, correction process and final presentation structure in real time through the calibration interface, allowing operators to intuitively grasp the current calibration status, facilitating quick confirmation or fine-tuning, greatly improving the convenience and efficiency of operation.

[0084] Our approach not only takes into account the timeliness, accuracy, and visualization of the calibration process, but also achieves the goals of low cost and ease of operation, thereby effectively improving the vehicle's perception of the surrounding environment during parking, driving in narrow sections, and complex road conditions, effectively reducing traffic accidents caused by blind spots, and thus comprehensively enhancing driving safety, stability, and convenience.

[0085] The above is only an embodiment of the present invention, and the common knowledge such as the specific technical solutions and / or characteristics in the solution are not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the description can be used to interpret the content of the claims.

Claims

1. A convenient AVM calibration and correction method based on a fisheye camera, characterized by: Step S100: obtaining the original calibration images of each fisheye camera, and performing internal calibration and dedistortion to obtain the corrected images; Step S200: Enter the calibration interface, select the fisheye cameras corresponding to the front, back, left, and right directions respectively, and select feature points in the calibration images currently captured by each fisheye camera in real time according to the corresponding schematic diagram; Step S300: performing stitching and fusion processing on the selected views; It includes the following steps: Step S310: splicing first and then fusion processing; Step S311: preliminarily stitching the BEV views generated by different fisheye cameras to obtain a pre-stitched panoramic image, and displaying the pre-stitched panoramic image in real time on the calibration interface; Step S312: Determine whether the pre-stitching result meets the requirements. If not, adjust the feature point parameters in the calibration interface until the stitching effect meets the requirements. If it meets the requirements, perform fusion processing. Step S320: direct splicing and fusion processing; Step S321: Preliminary stitching of the BEV views generated by different fisheye cameras; Step S322: Execute fusion processing to obtain a panoramic fusion image and determine whether the fusion effect meets the requirements. If not, adjust the feature point parameters in the calibration interface until the fusion effect meets the requirements.

2. The convenient AVM calibration and correction method based on a fisheye camera according to claim 1, characterized in that: Step S100 includes the following steps: Step S110: Select appropriate calibration plates and place them around the vehicle; Step S120: Using fisheye cameras around the vehicle to shoot the calibration plate at different angles and positions to obtain original calibration images; Step S130: extracting feature points from each original calibration image; Step S140: Calculating the intrinsic parameter matrix and distortion coefficient of each fisheye camera using the distortion model and Zhang Zhengyou calibration method; Step S150: Based on the intrinsic parameter matrix and the distortion coefficient, the collected original calibration image is dedistorted to obtain a corrected image.

3. The convenient fisheye camera-based AVM calibration and correction method according to claim 2, characterized in that: The calibration plate is a checkerboard, and the original calibration image needs to meet the following conditions during the shooting process: All the chessboards in the photos need to cover the entire shooting screen and need to cover different angles; A single checkerboard image occupies 1 / 4 to 1 / 3 of the shooting image; The checkerboard pattern must be displayed completely, with at least half a square of blank border left at the edge; Choose a checkerboard calibration cloth made of non-reflective material.

4. The convenient fisheye camera-based AVM calibration and correction method according to claim 1, characterized in that: The number of the feature points is 4.

5. The convenient fisheye camera-based AVM calibration and correction method according to claim 1, characterized in that: Step S200 includes the following steps: Step S210: Connect the relevant calibration equipment to the vehicle's OBD port to enter the calibration interface; The calibration interface is divided into three areas from left to right: an AVM view area for displaying stitching and fusion effects, a fisheye acquisition area for displaying the current correction image acquisition status of the four-way fisheye camera, and a feature point management area for calibrating and adjusting feature points. The AVM view area includes a panoramic BEV view and a fusion button for triggering multi-view fusion, and the fusion button is located in the upper right corner; The fisheye acquisition area displays the corrected image pictures currently captured in real time by the four fisheye cameras of front view, rear view, left view and right view in sequence from top to bottom; The feature point management area includes, from top to bottom, a feature point selection area for selecting a view to be adjusted and a feature point position adjustment area for fine-tuning feature point coordinates.

6. The convenient fisheye camera-based AVM calibration and correction method according to claim 5, characterized in that: The feature point selection area includes four views to be adjusted, namely the front view, the rear view, the left view and the right view, and a stitching button from top to bottom, wherein the stitching button is located in the lower left corner; The view to be adjusted corresponds to the fisheye real-time acquisition area; The feature point position adjustment area includes operation buttons for selecting a feature point value of a specific feature point, inputting a shift value of a pixel offset, and controlling the moving direction of the feature point; The operation buttons include four buttons: up, down, left and right.

7. The convenient fisheye camera-based AVM calibration and correction method according to claim 5, characterized in that: Step S200 further includes the following steps: Step S220: introducing perspective transformation to convert the corrected image captured in real time by each fisheye camera into an independent BEV view; Step S230: Select the view to be adjusted in the feature point selection area to enter the corresponding view interface, and click to select the corresponding actual point on the correction image screen captured in real time by the fisheye camera according to the position indication of the feature points listed in the view interface.

8. The convenient AVM calibration and correction method based on a fisheye camera according to claim 7, characterized in that: The view interface is divided into: a fisheye real-time acquisition area and a schematic diagram area; the schematic diagram area includes a schematic diagram and distance parameters for displaying feature point position indications; The distance parameters include d1, d2 and d3; wherein d1 represents the width of the calibration plate; d2: the distance between the front side of the vehicle and the calibration plate; d3: The distance between the left side of the vehicle and the calibration plate.

9. The convenient fisheye camera-based AVM calibration and correction method according to claim 1, characterized in that: The number of the original calibration images is 9-15.

10. The convenient AVM calibration and correction method based on a fisheye camera according to claim 6, characterized in that: One unit in the shift value represents one pixel.

Images