Image processing method and system for double-fisheye panoramic image fusion
By employing a dual-fisheye panoramic image fusion method, images are aligned and corrected in real time, solving the problem of high latency in panoramic image generation in existing technologies. This enables efficient and real-time panoramic image generation, suitable for virtual reality and augmented reality applications.
Patent Information
- Application Number
- CN202411431571.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-01-10
AI Technical Summary
Existing technologies cannot achieve real-time processing when generating panoramic images, resulting in high latency, which cannot meet the application scenarios with high real-time requirements. Furthermore, they consume a lot of computing resources and cannot run effectively on devices with limited resources.
The dual fisheye panoramic image fusion method includes image preprocessing, alignment, rendering and fusion. It utilizes feature point detection and pixel-level blending technology to perform image alignment and color difference correction in real time. The image fusion and color difference correction modules are integrated to simplify the processing flow.
It enables real-time generation of panoramic images, improves response speed, reduces computing resource consumption, enhances image quality and user experience, is suitable for virtual reality and augmented reality applications, and simplifies system development and maintenance.
Smart Images
Figure CN119295325A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and system for dual fisheye panoramic image fusion. Background Art
[0002] 360° panoramic image capture usually uses two back-to-back fisheye lenses to cover the panoramic field of view. However, due to differences in exposure and metering between the images taken by the two lenses at the same time, the synthesized panoramic image has chromatic aberration and seam problems. Some existing solutions mainly use the method of first capturing images and then processing them to perform chromatic aberration correction and image fusion.
[0003] These methods have major drawbacks in terms of real-time performance. Since they need to perform complex calculations and processing after image acquisition, these methods perform poorly in application scenarios with high real-time requirements, have high processing delays, and cannot be obtained immediately. In addition, these post-processing methods usually consume a lot of computing resources, which is not conducive to running on devices with limited resources. Summary of the invention
[0004] In view of the deficiency in the above-mentioned technology that the collected images cannot be processed in real time to generate a panoramic image, the present invention provides an image processing method and system for dual fisheye panoramic image fusion.
[0005] To achieve the above object, the present invention provides an image processing method for dual fisheye panoramic image fusion, comprising: S1, captures and collects images of the external environment through dual fisheyes; S2, preprocessing the collected images to improve image quality; S3, aligning the pre-processed images so that the overlapping areas can be seamlessly connected; S4, finally rendering the image after image alignment to improve the overall color accuracy of the image. During the image rendering process, the images are fused and the color difference correction is performed on the overlapping areas of the images, and then the processed panoramic image is output.
[0006] In S2, the image is pre-processed by denoising and adjusting the white balance.
[0007] In S3, the feature point detection and matching algorithm of the image is used, and the transformation matrix of the overlapping area of the two images is calculated to align the two images.
[0008] Among them, in S4, the overlapping area of the image is smoothed by using the multi-resolution image fusion technology of pixel-level mixing.
[0009] Among them, the captured images are subjected to histogram matching and grayscale equalization through chromatic aberration correction, the brightness, contrast and hue of the images are adjusted, the seams and chromatic aberration between images are reduced, and the image color and brightness are kept consistent.
[0010] The present invention also provides an image processing system for dual fisheye panoramic image fusion, which is applied to an image processing method for dual fisheye panoramic image fusion, comprising: Image acquisition module, which captures and collects images of the external environment through dual fisheyes; Image preprocessing module, which preprocesses the collected original images to improve the image quality; Image alignment module, which aligns the pre-processed images so that the overlapping areas can be seamlessly connected to form a complete panoramic image; Image fusion and chromatic aberration correction module adjusts the chromatic aberration, brightness, contrast and hue of the overlapping area of the images to keep the image colors in the overlapping area consistent and perform panoramic fusion of the two images; The rendering module performs image rendering on the fused panoramic image, improves the image quality and outputs the panoramic image.
[0011] The image preprocessing module performs denoising and white balance adjustment preprocessing on the image.
[0012] The image alignment module uses image feature point detection and matching algorithms and calculates a transformation matrix of the overlapping area of two images to align the overlapping areas of the two images and obtain a complete panoramic image.
[0013] The image fusion and chromatic aberration correction module smoothes the overlapping areas of the images through a multi-resolution image fusion technique of pixel-level mixing.
[0014] The image fusion and chromatic aberration correction module performs histogram matching and grayscale equalization on the captured images through chromatic aberration correction, adjusts the brightness, contrast and hue of the images, reduces the seams and chromatic aberration between images, and keeps the image color and brightness consistent.
[0015] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention provides an image processing method and system for dual fisheye panoramic image fusion. Compared with the traditional panoramic image generation method, the present invention realizes simultaneous image rendering, chromatic aberration correction and image fusion in the image processing process, improves the real-time performance and response speed of the panoramic image generation system, performs real-time processing and optimizes images in the image rendering process, meets the requirements of fast response and instant rendering, and also reduces the system computing resource occupation and energy consumption, and is particularly suitable for panoramic image applications of virtual reality VR and augmented reality AR that require high real-time performance and high efficiency; and through image fusion algorithm and color correction technology, effectively reduces the seams and chromatic aberration problems of images taken with fisheye lenses in overlapping areas, and improves the visual quality and user experience of panoramic images; the present invention integrates image fusion and chromatic aberration correction into one module, simplifies the image processing process, reduces the complexity of the system program, further improves the development and maintenance efficiency of the system, and can ensure the stability and reliability of the system operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a module interaction diagram of the present invention. DETAILED DESCRIPTION
[0017] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with the accompanying drawings.
[0018] In the following description, the example details are given to provide a deeper understanding of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. It should be understood that the specific embodiments are only used to explain the present invention and are not used to limit the present invention.
[0019] It should be understood that when the terms "include" and / or "comprises" are used in this specification, they indicate the existence of the stated features, integers, steps, operations, elements or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components or combinations thereof.
[0020] In the prior art, the method for generating a panoramic image is to first collect images and then perform image retouching processing on the images. The panoramic images obtained by this method have high quality, but poor real-time performance. The images cannot be processed in real time during the image collection, and the system needs to be waited for. Therefore, this method has the problem of being unable to perform image retouching processing in real time and synchronously. It takes a certain amount of time to obtain a complete panoramic image, which reduces the user experience. See also Figure 1-Figure 2 , an image processing method for dual fisheye panoramic image fusion of the present invention comprises: S1, captures and collects images of the external environment through dual fisheyes; S2, preprocessing the collected images to improve image quality; S3, aligning the pre-processed images so that the overlapping areas can be seamlessly connected; S4, finally rendering the image after image alignment to improve the overall color accuracy of the image. During the image rendering process, the images are fused and the color difference correction is performed on the overlapping areas of the images, and then the processed panoramic image is output.
[0021] First, the image of the external environment is collected through two fisheye lenses. Since the original image will have a large distortion, the image will have significant spherical distortion, which will make the image in some areas show obvious bending and stretching, and there will be uneven distribution problems in color and brightness. Therefore, after the original image is collected, the original image is first preprocessed to improve the overall image quality, and the image is denoised and white balanced. Then, the same feature points of the two images are calculated and aligned, and then the transformation matrix of the overlapping area of the two images is calculated to make the overlapping area seamlessly connected and consistent. After alignment, the image is rendered to improve the image quality. At the same time, image fusion and real-time chromatic aberration correction are also performed. During the rendering process, the overlap will also cause image color mismatch. The chromatic aberration correction is used to adjust the image attributes such as brightness, contrast, and hue in real time to make the colors of the two images closer and closer to the environmental color. At the same time, the overlapping areas of the aligned images are fused to generate a complete panoramic image. After rendering, the panoramic image can be output.
[0022] In this embodiment, image fusion is mainly an advanced image fusion algorithm, which adopts a multi-resolution image fusion technology based on pixel-level mixing to smooth the overlapping areas of the images and optimize the seam processing of the overlapping areas to ensure seamless connection of the panoramic images. The fusion process first sets the maximum radius and minimum radius of the pre-set image to define the overlapping area of the fisheye lens, converts the texture coordinates into image coordinates, and then calculates the center coordinates of the left and right fisheye lenses. For each pixel point, it is determined whether it is in the overlapping area based on its distance from the center. For the pixel points in the overlapping area, the new coordinates are recalculated by rotating the angle and distance, and color sampling and mixing are performed. Finally, a linear interpolation method is used to gradually mix the images of the two lenses in the overlapping area to achieve seamless connection.
[0023] In this embodiment, the real-time chromatic aberration correction mainly adopts histogram matching and grayscale equalization technology to adjust the color and brightness of the images taken by the two fisheye lenses to make them more consistent. The coordinates of the overlapping area of the fisheye lenses are obtained by pre-setting the maximum radius and the minimum radius, and the pixel values of the overlapping area are counted. The histograms of the three primary color channels of red, green and blue are calculated, and the cumulative distribution function (CDF) is used to convert the histogram into a cumulative histogram. The cumulative histogram of one image is matched with the cumulative histogram of another image, and then a mapping relationship is generated. The pixel values of the images are adjusted using the mapping relationship so that the color distribution of the two images is consistent, thereby reducing chromatic aberration.
[0024] In this embodiment, the image is pre-processed by denoising and white balancing adjustment through the image pre-processing module. Since the original image has many defects, such as significant spherical distortion of the image, the image in some areas will show obvious bending and stretching, and there will be uneven distribution in color and brightness, resulting in large image noise and color differences. The image is pre-processed by means of denoising or white balancing to improve the overall image quality, so as to facilitate the subsequent steps of image fusion and alignment.
[0025] In this embodiment, the present invention further discloses an image processing system for dual fisheye panoramic image fusion, which is applied to an image processing method for dual fisheye panoramic image fusion, including: Image acquisition module, which captures and collects images of the external environment through dual fisheyes; Image preprocessing module, which preprocesses the collected original images to improve the image quality; Image alignment module, which aligns the pre-processed images so that the overlapping areas can be seamlessly connected; Image fusion and chromatic aberration correction module adjusts the chromatic aberration, brightness, contrast and hue of the overlapping area of the images to keep the image colors in the overlapping area consistent and perform panoramic fusion of the two images; The rendering module works in real time with the image fusion and chromatic aberration correction module to render the fused panoramic image, and output the panoramic image after improving the image quality.
[0026] After the image acquisition module acquires the image of the external environment, it is sent to the image preprocessing module to perform image processing operations such as denoising and white balancing on the original image, thereby improving the image quality so that the image alignment module can accurately and seamlessly connect the overlapping areas of the images. After the images are aligned, the color difference, brightness, contrast and hue of the overlapping areas of the two images are kept consistent through the image fusion and chromatic aberration correction modules, and panoramic fusion is performed to merge the overlapping areas together, and image rendering is performed at the same time. During the rendering process, the chromatic aberration correction module adjusts parameters in real time until the rendering is completed. The present invention integrates the two functions of image fusion and chromatic aberration correction into the same processing module, simplifies the traditional image processing process, reduces the complexity of system implementation, further improves the development and maintenance efficiency of the system, and can ensure the stability and reliability of the system operation.
[0027] In this embodiment, the image alignment module uses the image feature point detection and matching algorithm, and calculates the transformation matrix of the overlapping area of the two images, so that the overlapping area of the two images is aligned to obtain a complete panoramic image; specifically, the transformation matrix is also called a coordinate mapping matrix. First, the two pictures of image A and image B are recorded correspondingly through the coordinate mapping matrix to form an A1 coordinate mapping matrix and a B1 coordinate mapping matrix, that is, feature point detection is performed on the pictures, and then the two coordinate mapping matrices are used to achieve point-to-point mapping through the coordinate mapping relationship, so that the feature points between the two images are matched, the position alignment is achieved, and the two image pictures remain overlapped.
[0028] The advantages of the present invention are: 1) Image rendering, chromatic aberration correction and image fusion are carried out simultaneously, which improves the real-time performance and response speed of the panoramic image generation system. Real-time processing and image optimization are performed during the image rendering process, meeting the needs of fast response and instant rendering.
[0029] 2) It reduces the system computing resource usage and energy consumption, and is particularly suitable for panoramic image applications of virtual reality (VR) and augmented reality (AR) that require high real-time performance and high efficiency.
[0030] 3) Through image fusion algorithm and color correction technology, the seams and chromatic aberration problems in the overlapping areas of fisheye lens images are effectively reduced, improving the visual quality and user experience of panoramic images.
[0031] 4) Integrating image fusion and chromatic aberration correction into one module simplifies the image processing process, reduces the complexity of the system program, further improves the efficiency of system development and maintenance, and ensures the stability and reliability of system operation.
[0032] The above disclosures are only several specific embodiments of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. An image processing method for dual fisheye panoramic image fusion, characterized in that: include: S1, captures and collects images of the external environment through dual fisheyes; S2, preprocessing the collected images to improve image quality; S3, aligning the pre-processed images so that the overlapping areas in the images can be seamlessly connected; S4, rendering the image after image alignment to improve the overall color accuracy of the image. During the image rendering process, the image is fused and the color difference correction is performed on the overlapping area of the image, and then the processed panoramic image is output.
2. The image processing method for dual fisheye panoramic image fusion according to claim 1, characterized in that: In S2, the image is pre-processed by denoising and white balance adjustment.
3. The image processing method for dual fisheye panoramic image fusion according to claim 1, characterized in that: In S3, the image feature point detection and matching algorithm is used, and the transformation matrix of the overlapping area of the two images is calculated to align the two images.
4. The image processing method for dual fisheye panoramic image fusion according to claim 1, characterized in that: In S4, the overlapping areas of the images are smoothed by using multi-resolution image fusion technology with pixel-level mixing.
5. The image processing method for dual fisheye panoramic image fusion according to claim 4, characterized in that: Through chromatic aberration correction, the captured images are subjected to histogram matching and grayscale equalization, the brightness, contrast and hue of the images are adjusted, the seams and chromatic aberration between images are reduced, and the image color and brightness are kept consistent.
6. An image processing system for dual fisheye panoramic image fusion, applied to the image processing method for dual fisheye panoramic image fusion according to claims 1-5, characterized in that: include: Image acquisition module, which captures and collects images of the external environment through dual fisheyes; Image preprocessing module, which preprocesses the collected original images to improve the image quality; Image alignment module, which aligns the pre-processed images so that the overlapping areas can be seamlessly connected; Image fusion and chromatic aberration correction module adjusts the chromatic aberration, brightness, contrast and hue of the overlapping area of the images to keep the image colors in the overlapping area consistent and perform panoramic fusion of the two images; The rendering module works in real time with the image fusion and chromatic aberration correction module to render the fused panoramic image, and output the panoramic image after improving the image quality.
7. The image processing system for dual fisheye panoramic image fusion according to claim 6, characterized in that: The image preprocessing module performs denoising and white balance adjustment preprocessing on the image.
8. The image processing system for dual fisheye panoramic image fusion according to claim 6, characterized in that: The image alignment module uses image feature point detection and matching algorithms and calculates a transformation matrix of the overlapping area of two images to align the overlapping areas of the two images and obtain a complete panoramic image.
9. The image processing system for dual fisheye panoramic image fusion according to claim 6, characterized in that: The image fusion and chromatic aberration correction module performs smoothing on the overlapping areas of the images through a multi-resolution image fusion technology of pixel-level mixing.
10. The image processing system for dual fisheye panoramic image fusion according to claim 9, characterized in that: The image fusion and chromatic aberration correction module performs histogram matching and grayscale equalization on the captured images through chromatic aberration correction, adjusts the brightness, contrast and hue of the images, reduces the seams and chromatic aberration between images, and keeps the image color and brightness consistent.