Interactive fisheye image distortion correction and splicing system

Abstract

The embodiment of the invention provides an interactive fisheye image distortion correction and splicing system. The system comprises an interactive parameter calibration module, a fisheye image distortion correction module, a region-of-interest intelligent selection module, a perspective transformation module, a parameter fine tuning optimization module and a multi-image intelligent splicing module. According to the system, intelligent distortion correction and seamless splicing processing of fisheye images are accurately realized by using an interactive user interface technology, a computer vision algorithm, a perspective transformation technology and a multi-camera splicing algorithm, a unified and standard technical solution for parameter calibration and image processing is provided, high consistency and repeatability of processing results are achieved, and the system is suitable for large-scale popularization and application. Manual on-site calibration is not needed, and internal and external parameters of the camera do not need to be obtained. And the labor cost can be greatly saved on the premise of ensuring the accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an interactive fisheye image distortion correction and stitching system. Background Technology

[0002] Fisheye cameras offer an ultra-wide field of view and are widely used in security monitoring, panoramic photography, and autonomous driving. However, due to the inherent optical characteristics of fisheye lenses, the captured images suffer from severe radial distortion, resulting in significant deformation at the image edges and straight lines being distorted into curves, affecting the practical application of the image. Currently, extensive research has been conducted both domestically and internationally on fisheye image distortion correction techniques, but some problems still exist in practical applications, and even professional institutions and personnel have significant differences in their understanding of certain technologies.

[0003] Existing fisheye image processing methods consider "complex parameter calibration, poor real-time performance, and low stitching accuracy" as the main technical challenges. However, different personnel have different understandings of "the accuracy of parameter calibration and real-time interactive adjustment," which directly affects the quality, efficiency, and on-site adaptability of fisheye image correction.

[0004] Existing solutions often require manual on-site calibration or acquisition of camera intrinsic and extrinsic parameters, which greatly increases labor costs. Furthermore, current fisheye image correction mainly relies on manual data collection and judgment, which suffers from problems such as large data volume, difficulty in obtaining information for manual recognition, inaccurate data collection and long process, high personnel workload, high long-term costs, and long update cycle (usually 1 year), making it difficult to meet the requirements of timely fisheye image updates as the current equipment and surrounding environment change. Fisheye image recognition rules are also greatly affected by the knowledge level and experience of personnel. Summary of the Invention

[0005] This disclosure provides an interactive fisheye image distortion correction and stitching system to solve the problems that current fisheye image processing mainly relies on manual parameter setting and image processing, which results in complex parameter calibration, high labor costs, and low accuracy.

[0006] To address the aforementioned problems, in a first aspect, an interactive fisheye image distortion correction and stitching system is provided, the system comprising: An interactive parameter calibration module is used to realize the intelligent calibration and real-time adjustment of imaging parameters in fisheye image distortion correction, and to obtain the calibration parameters of the imaging parameters in fisheye image distortion correction. A fisheye image distortion correction module is provided, which is connected to the interactive parameter calibration module. The fisheye image distortion correction module is used to convert a radially distorted fisheye image into a corrected standard perspective image through calibration by the interactive parameter calibration module. The region of interest (ROI) intelligent selection module is connected to the fisheye image distortion correction module. The ROI intelligent selection module is used to obtain the region of interest in the standard perspective image after correction by the fisheye image distortion correction module, and to crop the standard perspective image after correction based on the ROI to obtain the cropped image of the ROI. A perspective transformation module is connected to the region of interest intelligent selection module. The perspective transformation module is used to transform the cropped image of the region of interest obtained by the region of interest intelligent selection module into a bird's-eye view. A parameter fine-tuning and optimization module is connected to the perspective transformation module. The parameter fine-tuning and optimization module is used to optimize the bird's-eye view and transformation parameters generated by the perspective transformation module to obtain an optimized bird's-eye view. A multi-image intelligent stitching module is connected to the parameter fine-tuning and optimization module. The multi-image intelligent stitching module is used to automatically stitch together bird's-eye view images generated by multiple cameras into a panoramic image and perform seamless fusion processing at the stitching boundaries.

[0007] In conjunction with the first aspect, in one possible implementation, the interactive parameter calibration module is used for: Read the original fisheye image and initialize the imaging parameters in fisheye image distortion correction. The imaging parameters include the radius of the fisheye imaging surface and the distance from the imaging plane to the optical center. The adjusted imaging parameters can be obtained in real time through the slider control in the interactive control interface; Generate an image grid coordinate system and calculate the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. Perform a remapping transformation on the corrected mapping coordinates and display the corrected standard perspective image in real time; When the corrected standard perspective image meets the preset conditions, the corresponding imaging parameters are determined as calibration parameters and the calibration parameters are saved.

[0008] In conjunction with the first aspect, in one possible implementation, the fisheye image distortion correction module is used for: Obtain the calibration parameters of the imaging parameters and the original fisheye image in fisheye image distortion correction; Generate an image coordinate grid for the original fisheye image, and establish a coordinate system with the image center of the original fisheye image as the origin; Calculate the radial distance between each pixel in the original fisheye image and the center of the image; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. The corrected coordinates are remapped using bilinear interpolation to generate and output a corrected standard perspective image.

[0009] In conjunction with the first aspect, in one possible implementation, the fisheye correction mathematical model is rf=R*arctan(r / D), where r is the radial distance between a pixel in the original fisheye image and the center of the image, rf is the corrected radial distance, R is the radius of the fisheye imaging surface, and D is the distance from the imaging plane to the optical center.

[0010] In conjunction with the first aspect, in one possible implementation, the region of interest intelligent selection module is used for: Capture the left mouse button press event in the corrected standard perspective image window, and determine the position coordinates of the mouse click in the left mouse button press event as the starting coordinates; Track mouse movement events and update the endpoint coordinates in real time based on the mouse position in the mouse movement events; Based on the starting coordinates and the ending coordinates, draw a colored rectangle in the corrected standard perspective image; When the left mouse button is released, the region of interest in the corrected standard perspective image is determined based on the starting coordinates and the ending coordinates when the left mouse button is released; The corrected standard perspective image is cropped based on the region of interest to obtain a cropped image of the region of interest.

[0011] In conjunction with the first aspect, in one possible implementation, the perspective transformation module is used for: Obtain a cropped image of the region of interest from the corrected standard perspective image; activate a four-point selection interface on the cropped image; and obtain four control points of the cropped image based on user operations on the four-point selection interface. Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the perspective transformation matrix based on the four control points and the four vertices of the preset standard rectangular area; Based on the perspective transformation matrix, the cropped image is subjected to perspective transformation to generate a bird's-eye view; The transformation parameters are saved to a configuration file for subsequent fine-tuning and reuse. The transformation parameters include at least the perspective transformation matrix.

[0012] In conjunction with the first aspect, in one possible implementation, the parameter fine-tuning optimization module is used for: Obtain the corrected standard perspective image and the transformation parameters of the perspective transformation module; Load and display the bird's-eye view generated by the perspective transformation module, and launch the four-point selection interface; The system acquires the four control points that the user has recalibrated on the four-point selection interface, and draws and displays the marks of the four control points in real time to show the position of the control points selected by the user. Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the new perspective transformation matrix corresponding to the four control points recalibrated by the user; Based on the new perspective transformation matrix, an optimized bird's-eye view is generated; The comparison view shows the bird's-eye view before and after the optimization for user confirmation; If the user confirms that they are satisfied with the optimized bird's-eye view, then update the configuration file of the transformation parameters and save the optimized bird's-eye view; If the user confirms that they are not satisfied with the optimized bird's-eye view, then return to the step of obtaining the four control points recalibrated by the user on the four-point selection interface, until the user confirms that they are satisfied with the optimized bird's-eye view.

[0013] In conjunction with the first aspect, in one possible implementation, the multi-image intelligent stitching module is used for: The system acquires bird's-eye view images generated by multiple cameras, performs a consistency check on the image dimensions of the bird's-eye view images generated by the multiple cameras, and performs standardization preprocessing on the bird's-eye view images generated by the multiple cameras if the heights of the bird's-eye view images generated by the multiple cameras are inconsistent. Calculate the total width of the stitched bird's-eye view images generated by the multiple cameras based on the overlapping area of ​​the bird's-eye view images generated by the multiple cameras; Create a blank canvas corresponding to the panoramic image based on the total width of the stitched image; The bird's-eye view images generated by the multiple cameras are stitched together at a designated position on the blank canvas according to the arrangement order of the cameras to obtain a stitched image; Seamless fusion processing is performed at the splicing boundary of the spliced ​​image to obtain a seamlessly fused spliced ​​image; The seamlessly merged stitched image is then resized and its image quality is optimized to obtain a panoramic image.

[0014] In conjunction with the first aspect, in one possible implementation, the step of performing seamless fusion processing at the splicing boundaries of the stitched images to obtain a seamlessly fused stitched image includes: Based on the preset boundary region width, extract the boundary transition region between adjacent images in the stitched image; The boundary transition region is subjected to Gaussian blurring, and the processed boundary transition region is merged back to its original position to obtain a seamlessly merged stitched image.

[0015] In conjunction with the first aspect, in one possible implementation, the step of resizing and optimizing the image quality of the seamlessly fused stitched image to obtain a panoramic image includes: The size of the seamlessly merged stitched image is adjusted to the preset panoramic image size, and the stitched image after size adjustment is resampled using the Lanczos interpolation algorithm to obtain a high-quality panoramic image.

[0016] The beneficial effects of the embodiments disclosed herein include: This disclosure provides an interactive fisheye image distortion correction and stitching system. Utilizing interactive user interface technology, computer vision algorithms, perspective transformation technology, and multi-camera stitching algorithms, it accurately achieves intelligent distortion correction and seamless stitching of fisheye images. It provides a unified and standardized technical solution for parameter calibration and image processing, ensuring high consistency and repeatability of processing results. Furthermore, it eliminates the need for manual on-site calibration and the acquisition of camera intrinsic and extrinsic parameters. This significantly reduces labor costs while maintaining accuracy.

[0017] The interactive real-time parameter adjustment mechanism solves the problems of complex parameter calibration and low adjustment efficiency in traditional methods; the intelligent fisheye distortion correction algorithm improves the accuracy and stability of image processing; the visualized ROI region selection and perspective transformation technology simplifies the operation process and lowers the technical threshold; and the multi-camera intelligent stitching algorithm enables the generation of high-quality panoramic images.

[0018] This invention adopts a processing mode that prioritizes automated intelligent processing and supplements it with manual interactive adjustments. This significantly reduces reliance on professional technicians, improves the efficiency and quality of fisheye image processing, and ultimately meets the technical requirements of intelligent processing and real-time stitching of fisheye images in application fields such as panoramic monitoring, autonomous driving, and virtual reality. It also promotes the development of fisheye image processing technology towards intelligence, standardization, and automation. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of an interactive fisheye camera distortion correction and stitching system provided in an embodiment of the present disclosure; Figure 2 A flowchart of an interactive fisheye image distortion correction and stitching method provided in this disclosure embodiment; Figure 3 A flowchart for an interactive parameter calibration provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating intelligent selection of region of interest and perspective transformation for generating a bird's-eye view, provided as an embodiment of this disclosure; Figure 5 A flowchart for parameter fine-tuning optimization provided in this embodiment of the disclosure; Figure 6 A flowchart illustrating intelligent multi-image stitching provided in this embodiment of the disclosure; Figure 7 Original fisheye images captured by three fisheye cameras provided in embodiments of this disclosure; Figure 8 This is a corrected and stitched panoramic bird's-eye view provided in an embodiment of this disclosure. Detailed Implementation

[0020] This disclosure provides an interactive fisheye camera distortion correction and stitching system. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0021] This disclosure provides an interactive fisheye camera distortion correction and stitching system. Figure 1 This is a schematic diagram of the structure of an interactive fisheye camera distortion correction and stitching system provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the system includes: an interactive parameter calibration module 101, a fisheye image distortion correction module 102, an intelligent region of interest selection module 103, a perspective transformation module 104, a parameter fine-tuning optimization module 105, and a multi-image intelligent stitching module 106.

[0022] The interactive parameter calibration module 101 is used to realize the intelligent calibration and real-time adjustment of imaging parameters in fisheye image distortion correction, and obtain the calibration parameters of imaging parameters in fisheye image distortion correction. The fisheye image distortion correction module 102 is connected to the interactive parameter calibration module 101. The fisheye image distortion correction module 102 is used to convert the radially distorted fisheye image into a corrected standard perspective image through the calibration of the interactive parameter calibration module 101. The region of interest intelligent selection module 103 is connected to the fisheye image distortion correction module 102. The region of interest intelligent selection module 103 is used to obtain the region of interest in the standard perspective image after correction by the fisheye image distortion correction module 102, and to crop the standard perspective image after correction according to the region of interest to obtain the cropped image of the region of interest. The perspective transformation module 104 is connected to the region of interest intelligent selection module 103. The perspective transformation module 104 is used to transform the cropped image of the region of interest obtained by the region of interest intelligent selection module 103 into a bird's-eye view. The parameter fine-tuning and optimization module 105 is connected to the perspective transformation module 104. The parameter fine-tuning and optimization module 105 is used to optimize the bird's-eye view and transformation parameters generated by the perspective transformation module 104. The multi-image intelligent stitching module 106 is connected to the parameter fine-tuning and optimization module 105. The multi-image intelligent stitching module 106 is used to automatically stitch the bird's-eye view images generated by multiple cameras into a panoramic image and perform seamless fusion processing at the stitching boundary.

[0023] This disclosure introduces an interactive user interface and automated image processing technology to address the problems that current fisheye image processing mainly relies on manual parameter setting and image processing, resulting in complex parameter calibration, low efficiency of manual adjustment, unstable processing accuracy and cumbersome process, high technical requirements for operators and high long-term maintenance costs, long parameter update cycle, difficulty in meeting the requirement of timely parameter recalibration when equipment configuration and shooting environment change, and the image processing effect being greatly affected by the operator's technical level and experience.

[0024] Figure 2 A flowchart of an interactive fisheye image distortion correction and stitching method provided in this disclosure embodiment is shown below. Figure 2As shown, this method is implemented through an interactive fisheye image distortion correction and stitching system. After acquiring the fisheye image, the interactive parameter calibration module 101 is responsible for adjusting the radius R of the fisheye imaging surface and the distance D from the imaging plane to the optical center in real time through the slider interface, and previewing the correction effect of the fisheye image based on the radius R of the fisheye imaging surface and the distance D from the imaging plane to the optical center in real time (displaying the corrected standard perspective image), and generating a mapping relationship matrix between the fisheye image and the corrected standard perspective image; the fisheye image distortion correction module 102 is responsible for converting the fisheye image into a standard perspective image using a remapping algorithm according to the calibration parameters; the region of interest intelligent selection module 103 is responsible for accurately selecting the ROI (Region of Interest) through mouse interaction. The image has an area of ​​interest (ROI) and supports rectangular bounding box visualization; the perspective transformation module 104 is responsible for converting the corrected image into a standard bird's-eye view and supports four-point perspective transformation; the parameter fine-tuning optimization module 105 is responsible for secondary optimization and adjustment of the generated bird's-eye view by reselecting transformation points and optimizing the perspective matrix; the multi-image intelligent stitching module 106 is responsible for automatically stitching the bird's-eye views from multiple cameras into a panoramic image (panoramic bird's-eye view) according to preset rules and blurring the stitching boundaries to achieve a seamless stitching effect.

[0025] As one possible implementation, the interactive parameter calibration module 101 is based on computer vision technology and graphical user interface technology, covering multiple scenarios such as parameter initialization, real-time adjustment, and effect preview. It uses slider control and remapping algorithm to realize intelligent calibration and real-time adjustment of imaging parameters in fisheye image distortion correction. Figure 3 A flowchart for interactive parameter calibration provided in this disclosure embodiment, such as Figure 3 As shown, the interactive parameter calibration module 101 uses interactive parameter calibration tools and mathematical model algorithms to adjust, preview, and confirm the calibration parameters, and then generates a calibration parameter data file and stores it for later use.

[0026] Specifically, the interactive parameter calibration module 101 is used for: Read the original fisheye image and initialize the imaging parameters in the fisheye image distortion correction. The imaging parameters include the radius of the fisheye imaging surface and the distance from the imaging plane to the optical center. The adjusted imaging parameters can be obtained in real time through the slider control in the interactive control interface; Generate an image grid coordinate system and calculate the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. Perform a remapping transformation on the corrected mapped coordinates and display the corrected standard perspective image in real time; When the corrected standard perspective image meets the preset conditions, the corresponding imaging parameters are determined as calibration parameters and saved.

[0027] In some embodiments, the preset condition is that the user is satisfied with the corrected standard perspective image. The user adjusts the imaging parameters by sliding the slider, and the system updates the correction effect in real time until the user is satisfied. The system then saves the final calibration parameters and the mapping relationship matrix file between the original fisheye image and the corrected standard perspective image.

[0028] As one possible implementation, fisheye image distortion correction is defined as the process of converting a radially distorted fisheye image into a standard perspective image through mathematical transformation. Specifically, the fisheye image distortion correction module 102 is used for: Obtain the calibration parameters of the imaging parameters and the original fisheye image in fisheye image distortion correction; Generate an image coordinate grid for the original fisheye image, and establish a coordinate system with the image center of the original fisheye image as the origin; Calculate the radial distance between each pixel in the original fisheye image and the image center; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. The corrected coordinates are remapped using bilinear interpolation to generate and output a corrected standard perspective image.

[0029] In this embodiment of the disclosure, the output corrected standard perspective image is used to prepare for subsequent region selection and perspective transformation.

[0030] As one possible implementation method, the mathematical model for fisheye correction is r f =R*arctan(r / D), where r is the radial distance between a pixel in the original fisheye image and the image center. f R is the corrected radial distance, R is the radius of the fisheye imaging surface, and D is the distance from the imaging plane to the optical center.

[0031] The corrected coordinates of each pixel are (x) f ,y f ), x f =c x +r f / r*(xc x ), y f =c y +r f / r*(yc y), where, (c x , c y ) represents the coordinates of the image center.

[0032] As one possible implementation, the region of interest intelligent selection module 103 is based on mouse interaction technology and visual feedback mechanism to achieve accurate selection and real-time preview of the region of interest in the corrected image. Figure 4 A flowchart illustrating intelligent selection of a region of interest and perspective transformation for generating a bird's-eye view, as provided in this disclosure embodiment, is shown below. Figure 4 As shown, the region of interest intelligent selection module 103 is used for: Capture the left mouse button press event in the corrected standard perspective image window, and determine the coordinates of the mouse click position in the left mouse button press event as the starting coordinates; Track mouse movement events and update the endpoint coordinates in real time based on the mouse position in the mouse movement events; Draw a colored rectangle in the corrected standard perspective image based on the starting and ending coordinates; When the left mouse button is released, the region of interest in the corrected standard perspective image is determined based on the starting coordinates and the ending coordinates when the left mouse button is released. The corrected standard perspective image is cropped based on the region of interest to obtain the cropped image of the region of interest.

[0033] In some embodiments, the selected region of interest is saved via keyboard shortcuts, and the image is cropped; the cropped image file is saved to provide input data for the perspective transformation module 104.

[0034] As one possible implementation, such as Figure 4 As shown, the perspective transformation module 104 is used for: Obtain a cropped image of the region of interest from the corrected standard perspective image. Launch a four-point selection interface on the cropped image and obtain four control points of the cropped image based on user operations on the four-point selection interface. Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the perspective transformation matrix based on the four control points and the four vertices of the preset standard rectangular area; Based on the perspective transformation matrix, the cropped image is transformed by perspective to generate a bird's-eye view; Save the transformation parameters to the configuration file for later fine-tuning and reuse. The transformation parameters must include at least the perspective transformation matrix.

[0035] As one possible implementation method, Figure 5 A flowchart for parameter fine-tuning optimization provided in this disclosure embodiment, such as... Figure 5As shown, the parameter fine-tuning optimization module 105 is used for: Obtain the corrected standard perspective image and the transformation parameters of perspective transformation module 104; Load and display the bird's-eye view generated by the perspective transformation module 104, and start the four-point selection interface; Get the four control points that the user recalibrated on the four-point selection interface, and draw the markers of the four control points in real time to display the position of the control points selected by the user; Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the new perspective transformation matrix corresponding to the four control points recalibrated by the user; Based on the new perspective transformation matrix, an optimized bird's-eye view is generated; The comparison view shows the bird's-eye view before and after the optimization for user confirmation; If the user confirms that they are satisfied with the optimized bird's-eye view, then update the configuration file of the transformation parameters and save the optimized bird's-eye view; If the user confirms that they are not satisfied with the optimized bird's-eye view, return to the step of obtaining the four control points recalibrated by the user on the four-point selection interface, until the user confirms that they are satisfied with the optimized bird's-eye view.

[0036] In some embodiments, the order of selecting the four control points is top left → top right → bottom left → bottom right, and the selected points are drawn in real time using green circles as markers.

[0037] As one possible implementation method, Figure 6 A flowchart of a multi-image intelligent stitching method provided in this disclosure embodiment is shown below. Figure 6 As shown, the multi-image intelligent stitching module 106 is used for: Acquire bird's-eye view images generated by multiple cameras, perform a consistency check on the image size of the bird's-eye view images generated by multiple cameras, and if the heights of the bird's-eye view images generated by multiple cameras are inconsistent, perform standardization preprocessing on the bird's-eye view images generated by multiple cameras. Calculate the total width of the stitched bird's-eye view images generated by multiple cameras based on the overlapping area of ​​the images. Create a blank canvas corresponding to the panoramic image based on the total width of the stitched image; The bird's-eye view images generated by multiple cameras are stitched together in the order of the cameras to a specified position on a blank canvas to obtain a stitched image; Seamless fusion processing is performed at the splicing boundary of the spliced ​​images to obtain a seamlessly fused spliced ​​image; The seamlessly merged stitched images are resized and their image quality is optimized to obtain a panoramic image.

[0038] As one possible implementation, seamless fusion processing is performed at the stitching boundaries of the stitched images to obtain a seamlessly fused stitched image, including: Based on the preset boundary region width, extract the boundary transition region between adjacent images in the stitched image; Gaussian blur is applied to the boundary transition area, and the processed boundary transition area is then merged back to its original position to obtain a seamlessly merged stitched image.

[0039] As one possible approach, the seamlessly stitched images are resized and their image quality optimized to obtain a panoramic image, including: The size of the seamlessly merged stitched image is adjusted to the preset panoramic image size. The stitched image after size adjustment is then resampled using the Lanczos interpolation algorithm to obtain a high-quality panoramic image.

[0040] like Figure 6 As shown, the bird's-eye view images cam_1, cam_2, and cam_3 from three cameras are read. An image size consistency check is performed on these three images. If the heights are inconsistent, the images are preprocessed to standardize their dimensions. The total stitching width is calculated as width_total = w1 + w2 + w3, where w1, w2, and w3 are the widths of the bird's-eye view images from the three cameras, respectively. A blank canvas (RGB mode) is created, and the bird's-eye view images from the three cameras are stitched together sequentially. The position of the bird's-eye view image from camera 1 is (0, 0), the position of the bird's-eye view image from camera 2 is (w1, 0), and the position of the bird's-eye view image from camera 3 is (w1 + w2, ..., w3). 0); Define the stitching boundary region according to the preset boundary region width blur_width=10px, extract the boundary transition region through crop_box setting, and apply Gaussian blur to the boundary transition region to soften the boundary; paste the Gaussian blurred boundary transition region back to its original position; adjust the image size to the preset panoramic image size of 2500 pixels * 1000 pixels, apply Lanczos resampling to improve image quality, save the final panoramic image and select real-time preview display.

[0041] For example, Figure 7 These are the original fisheye images captured by the three fisheye cameras provided in this embodiment of the disclosure. Figure 8 This disclosure provides a corrected and stitched panoramic bird's-eye view, achieved through an interactive fisheye image distortion correction and stitching system provided in this disclosure. Figure 7 The original fisheye images captured by the three fisheye cameras shown are processed to obtain the following: Figure 8 The image shown is a panoramic bird's-eye view after correction and stitching.

[0042] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0043] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0044] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0045] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An interactive fisheye image distortion correction and stitching system, characterized in that, The system includes: An interactive parameter calibration module is used to realize the intelligent calibration and real-time adjustment of imaging parameters in fisheye image distortion correction, and to obtain the calibration parameters of the imaging parameters in fisheye image distortion correction. A fisheye image distortion correction module is provided, which is connected to the interactive parameter calibration module. The fisheye image distortion correction module is used to convert a radially distorted fisheye image into a corrected standard perspective image through calibration by the interactive parameter calibration module. The region of interest (ROI) intelligent selection module is connected to the fisheye image distortion correction module. The ROI intelligent selection module is used to obtain the region of interest in the standard perspective image after correction by the fisheye image distortion correction module, and to crop the standard perspective image after correction based on the ROI to obtain the cropped image of the ROI. A perspective transformation module is connected to the region of interest intelligent selection module. The perspective transformation module is used to transform the cropped image of the region of interest obtained by the region of interest intelligent selection module into a bird's-eye view. A parameter fine-tuning and optimization module is connected to the perspective transformation module. The parameter fine-tuning and optimization module is used to optimize the bird's-eye view and transformation parameters generated by the perspective transformation module to obtain an optimized bird's-eye view. A multi-image intelligent stitching module is connected to the parameter fine-tuning and optimization module. The multi-image intelligent stitching module is used to automatically stitch together bird's-eye view images generated by multiple cameras into a panoramic image and perform seamless fusion processing at the stitching boundaries.

2. The system according to claim 1, characterized in that, The interactive parameter calibration module is used for: Read the original fisheye image and initialize the imaging parameters in fisheye image distortion correction. The imaging parameters include the radius of the fisheye imaging surface and the distance from the imaging plane to the optical center. The adjusted imaging parameters can be obtained in real time through the slider control in the interactive control interface; Generate an image grid coordinate system and calculate the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. Perform a remapping transformation on the corrected mapping coordinates and display the corrected standard perspective image in real time; When the corrected standard perspective image meets the preset conditions, the corresponding imaging parameters are determined as calibration parameters and the calibration parameters are saved.

3. The system according to claim 1, characterized in that, The fisheye image distortion correction module is used for: Obtain the calibration parameters of the imaging parameters and the original fisheye image in fisheye image distortion correction; Generate an image coordinate grid for the original fisheye image, and establish a coordinate system with the image center of the original fisheye image as the origin; Calculate the radial distance between each pixel in the original fisheye image and the center of the image; Based on the adjusted imaging parameters, the radial distance from each pixel in the original fisheye image to the image center of the original fisheye image, and the fisheye correction mathematical model, the corrected coordinates of each pixel in the original fisheye image are calculated using the arctangent function. The corrected coordinates are remapped using bilinear interpolation to generate and output a corrected standard perspective image.

4. The system according to claim 2 or 3, characterized in that, The mathematical model for fisheye correction is r. f =R*arctan(r / D), where r is the radial distance between a pixel in the original fisheye image and the image center, r f R is the corrected radial distance, R is the radius of the fisheye imaging surface, and D is the distance from the imaging plane to the optical center.

5. The system according to claim 1, characterized in that, The region of interest intelligent selection module is used for: Capture the left mouse button press event in the corrected standard perspective image window, and determine the position coordinates of the mouse click in the left mouse button press event as the starting coordinates; Track mouse movement events and update the endpoint coordinates in real time based on the mouse position in the mouse movement events; Based on the starting coordinates and the ending coordinates, draw a colored rectangle in the corrected standard perspective image; When the left mouse button is released, the region of interest in the corrected standard perspective image is determined based on the starting coordinates and the ending coordinates when the left mouse button is released; The corrected standard perspective image is cropped based on the region of interest to obtain a cropped image of the region of interest.

6. The system according to claim 1, characterized in that, The perspective transformation module is used for: Obtain a cropped image of the region of interest from the corrected standard perspective image; activate a four-point selection interface on the cropped image; and obtain four control points of the cropped image based on user operations on the four-point selection interface. Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the perspective transformation matrix based on the four control points and the four vertices of the preset standard rectangular area; Based on the perspective transformation matrix, the cropped image is subjected to perspective transformation to generate a bird's-eye view; The transformation parameters are saved to a configuration file for subsequent fine-tuning and reuse. The transformation parameters include at least the perspective transformation matrix.

7. The system according to claim 1, characterized in that, The parameter fine-tuning and optimization module is used for: Obtain the corrected standard perspective image and the transformation parameters of the perspective transformation module; Load and display the bird's-eye view generated by the perspective transformation module, and launch the four-point selection interface; The system acquires the four control points that the user has recalibrated on the four-point selection interface, and draws and displays the marks of the four control points in real time to show the position of the control points selected by the user. Based on the scaling ratio of the cropped image in the display window, the display coordinates of the four control points are converted into the original image coordinates of the cropped image; Calculate the new perspective transformation matrix corresponding to the four control points recalibrated by the user; Based on the new perspective transformation matrix, an optimized bird's-eye view is generated; The comparison view shows the bird's-eye view before and after the optimization for user confirmation; If the user confirms that they are satisfied with the optimized bird's-eye view, then update the configuration file of the transformation parameters and save the optimized bird's-eye view; If the user confirms that they are not satisfied with the optimized bird's-eye view, then return to the step of obtaining the four control points recalibrated by the user on the four-point selection interface, until the user confirms that they are satisfied with the optimized bird's-eye view.

8. The system according to claim 1, characterized in that, The multi-image intelligent stitching module is used for: The system acquires bird's-eye view images generated by multiple cameras, performs a consistency check on the image dimensions of the bird's-eye view images generated by the multiple cameras, and performs standardization preprocessing on the bird's-eye view images generated by the multiple cameras if the heights of the bird's-eye view images generated by the multiple cameras are inconsistent. Calculate the total width of the stitched bird's-eye view images generated by the multiple cameras based on the overlapping area of ​​the bird's-eye view images generated by the multiple cameras; Create a blank canvas corresponding to the panoramic image based on the total width of the stitched image; The bird's-eye view images generated by the multiple cameras are stitched together at a designated position on the blank canvas according to the arrangement order of the cameras to obtain a stitched image; Seamless fusion processing is performed at the splicing boundary of the spliced ​​image to obtain a seamlessly fused spliced ​​image; The seamlessly merged stitched image is then resized and its image quality is optimized to obtain a panoramic image.

9. The system according to claim 8, characterized in that, The step of performing seamless fusion processing at the splicing boundary of the spliced ​​image to obtain a seamlessly fused spliced ​​image includes: Based on the preset boundary region width, extract the boundary transition region between adjacent images in the stitched image; The boundary transition region is subjected to Gaussian blurring, and the processed boundary transition region is merged back to its original position to obtain a seamlessly merged stitched image.

10. The system according to claim 8, characterized in that, The process of adjusting the size and optimizing the image quality of the seamlessly merged stitched image to obtain a panoramic image includes: The size of the seamlessly merged stitched image is adjusted to the preset panoramic image size, and the stitched image after size adjustment is resampled using the Lanczos interpolation algorithm to obtain a high-quality panoramic image.

Images