Panoramic stitching method, camera and storage medium

Abstract

The invention discloses a panorama stitching method, a camera and a storage medium, the panorama stitching method is realized based on a deep learning algorithm, and the method comprises the following steps: S1, obtaining two fisheye images by using two spherical fisheye lenses, and expanding the two fisheye images into an Equation panorama to obtain p1 and p2; s2, expanding panorama images of the effective visual area masks of the two fisheye images in the S1 into m1 and m2; s3, roughly fusing the panorama, fusing the p2 by using the p1 to obtain p12, and fusing the p1 by using the p2 to obtain p21; s4, performing optical flow information estimation on p12 and p21 in the S3 by using a deep learning algorithm; s5, designing a transition zone region fusion coefficient; s6, weighting each coefficient of the transition zone area by using the optical flow information in the S4 and combining the fusion coefficient in the S5, and then introducing a ghosting coefficient rho to calculate a specific color value of each pixel; s7, finely fusing the panorama; and S8, outputting the panoramic image. The method has the advantages of being high in splicing precision and stability, good in splicing real-time performance, good in dynamic scene picture consistency, good in suture line smoothing effect and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing computer vision algorithms, and in particular relates to a panoramic stitching method, a camera and a storage medium. Background Art

[0002] In recent years, 360-degree sports cameras have gradually become an important device in panoramic image production with their wide field of view and immersive experience. Capturing panoramic videos through multiple lenses, such as two spherical fisheye lenses, and stitching images from multiple fields of view into a complete panoramic picture to provide users with a full range of visual effects is the core function of such cameras. However, the implementation of panoramic stitching usually involves complex image processing technology, and its quality directly affects the user experience. At present, panoramic stitching technology is mainly based on feature point matching stitching and optical flow stitching. The feature point matching method detects feature points in the image, such as SIFT, ORB, etc., calculates the geometric transformation matrix between two images, and completes image stitching. In addition, the optical flow algorithm estimates the transformation relationship of the image by analyzing the motion vector between pixels. Compared with feature point matching, it has higher adaptability to complex scenes, such as areas with less texture. However, the optical flow calculation has high requirements on accuracy and computational complexity, which directly affects the real-time performance and quality of stitching.

[0003] Although the existing optical flow algorithms have shown certain advantages in panorama stitching, they still have the following problems and shortcomings:

[0004] In scenes with changing lighting, fast motion, or less texture, optical flow estimation is prone to inaccuracy, resulting in obvious misalignment or edge discontinuity near the stitching line, affecting image quality;

[0005] Optical flow algorithms require high computing resources, especially in the case of high-resolution or multi-lens input. It is difficult to achieve high-quality, low-latency stitching under limited hardware conditions and cannot meet real-time processing requirements.

[0006] For scenes containing fast-moving objects or complex dynamic content, existing optical flow algorithms have difficulty accurately estimating the object's motion trajectory, resulting in ghosting or distortion of dynamic objects in the stitched image;

[0007] In multi-lens stitching, the optical flow algorithm needs to simultaneously handle the geometric distortion between lenses and the smoothness of the stitching lines at the boundaries of the field of view. Existing methods usually have difficulty in balancing local details and overall image consistency, resulting in blur or artifacts in the transition area. Summary of the invention

[0008] In order to solve the deficiencies in the prior art, the present invention provides a panoramic stitching method, a camera and a storage medium.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] The panorama stitching method, based on the deep learning algorithm, includes the following steps:

[0011] S1, using two spherical fisheye lenses to obtain two fisheye images and expand them into Equirectangular panoramas to obtain p1 and p2;

[0012] S2, the effective visualization area mask of the two fisheye images in S1 is expanded into panorama m1 and m2;

[0013] S3, coarse fusion panorama, using p1 to fuse p2 to get p12, and using p2 to fuse p1 to get p21;

[0014] S4, use deep learning algorithm to estimate the optical flow information of p12 and p21 in S3 respectively;

[0015] S5. Design the transition zone regional fusion coefficient, the calculation formula is as follows:

[0016]

[0017] In the formula, blend ij is the fusion coefficient, which varies between [0,1], Fmin ij With Bmin ij They represent the closest distances between the pixel (i, j) of the two spherical fisheye lenses and m1 and m2 respectively;

[0018] S6, using the optical flow information in S4 combined with the fusion coefficient in S5 to weight each coefficient in the transition zone area, and then introducing the ghost coefficient ρ to calculate the specific color value of each pixel;

[0019] S7, fine fusion panorama;

[0020] S8. Panoramic image output.

[0021] Preferably, the fusion calculation of p12 in S3 is:

[0022] p12=p1(m1)+p2(!m1), where (m1) represents the effective visualization area mask expanded panoramic image m1 of the fisheye image in S2, and (!m1) represents the inversion of the effective visualization area mask expanded panoramic image m1 of the fisheye image in S2.

[0023] Preferably, the fusion calculation of p21 in S3 is:

[0024] p21=p2(m2)+p1(!m2), where (m2) represents the effective visualization area mask expanded panoramic image m2 of the fisheye image in S2, and (!m2) represents the inversion of the effective visualization area mask expanded panoramic image m2 of the fisheye image in S2.

[0025] Preferably, the deep learning algorithm is any one of the fastFlowNet algorithm, the RAFT algorithm or the memFlowNet algorithm.

[0026] Preferably, when the panorama is finely fused in S7, for each pixel (x, y) in the panorama, the optical flow vector fLR(x, y) from the first fisheye image to the second fisheye image in the two fisheye images and the optical flow vector fRL(x, y) from the second fisheye image to the first fisheye image are used to calculate the fusion weights wF(x, y) and wB(x, y) of the two images according to the fusion mask m(x, y), wherein the calculation formula of wF(x, y) is:

[0027] wF(x,y)=b(x,y);

[0028] The calculation formula of wB(x,y) is:

[0029] wB(x,y)=1-b(x,y);

[0030] Then, the color value C(x,y) of the pixel (x,y) in the fused panoramic image is calculated based on the fusion weights wF(x,y) and wB(x,y) of the two images and the ghost coefficient ρ(rho).

[0031] Preferably, the calculation formula for the color value C(x,y) of a pixel (x,y) in the panoramic image is:

[0032] C(x,y)=CF(x,y)*wF(x,y)*rho(x,y)+CB(x,y)*wB(x,y)*rho(x,y).

[0033] The present invention also discloses a camera, which includes a processor and a memory. The processor is used to execute the steps of the above-mentioned panoramic stitching method by calling a program or instruction stored in the memory.

[0034] The present invention also discloses a storage medium, wherein the computer-readable storage medium stores a program or an instruction, and the program or the instruction executes the steps of the above-mentioned panoramic stitching method.

[0035] By adopting the above technical solution, the present invention has the following beneficial effects:

[0036] (1) The present invention can use the optical flow estimation model to achieve a stitching effect with higher accuracy and robustness. In the prior art, in complex scenes with lighting changes, fast motion and less texture, it is impossible to effectively estimate the motion vector, resulting in the problem of dislocation, cracks or edge discontinuity at the stitching picture boundary. The present invention improves the stitching accuracy and stability.

[0037] (2) The existing technology has large computational complexity and high time overhead in high-resolution and multi-lens scenes, which makes it difficult to meet the needs of real-time video stitching. The present invention adopts a lightweight optical flow network model to significantly reduce the computational complexity while ensuring accuracy, improve real-time processing performance, and improve the real-time performance of the stitching algorithm;

[0038] (3) Moving objects in existing dynamic scenes are prone to stitching errors, such as ghosting or blurring. The present invention designs a dedicated optical flow stitching strategy to enhance the processing capabilities of object boundaries and details in dynamic scenes, achieve smooth transitions of images, and optimize the image consistency of dynamic scenes;

[0039] (4) In the stitching area of ​​multi-lens stitching, traditional methods often find it difficult to balance the consistency of the global image and the fidelity of local details, resulting in blurred stitching lines or artifacts. The present invention optimizes stitching line processing and reduces visual artifacts by integrating optical flow estimation and geometric correction modules, thereby improving stitching line smoothing effects and geometric distortion correction. The prior art requires two frames of images at adjacent moments as input, but the present invention inputs two spherical fisheye lenses with common view areas engraved at the same time to obtain an unfolded Equirectangular panoramic image of the two fisheye images, and outputs the displacement of each pixel, which can achieve better technical effects.

[0040] In summary, the present invention has the advantages of high stitching accuracy and stability, good real-time stitching, good consistency of dynamic scene images, and good smoothing effect of stitching lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0043] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0044] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Example 1

[0048] In this embodiment, the present invention provides a panoramic stitching method, a camera, and a storage medium, specifically a panoramic stitching method, a camera, and a storage medium of a 360-degree motion camera based on optical flow estimation.

[0049] like Figure 1 As shown, in one embodiment of the present invention, the panoramic stitching method of the present invention is implemented based on a deep learning algorithm, and the deep learning algorithm is any one of the fastFlowNet algorithm, the RAFT algorithm or the memFlowNet algorithm. The existing deep learning algorithm inputs two frames of images at adjacent moments to perform optical flow estimation. In the present invention, an Equirectangular panoramic image of two fisheye images of two spherical fisheye lenses at the same moment is input to perform optical flow estimation, and finally outputs the displacement of each pixel. In the present invention, two fisheye images are obtained by shooting with a dual fisheye lens, wherein the two lenses of the dual fisheye lens are two lenses set back to back, and the sum of the field of view angles of the two lenses is greater than or equal to 360 degrees, and specifically includes the following steps:

[0050] S1. Two fisheye images are obtained by using two spherical fisheye lenses and expanded into an Equirectangular panorama to obtain p1 and p2. A double fisheye image usually refers to an image obtained by shooting with two fisheye lenses. A fisheye lens is a special ultra-wide-angle lens with a very large viewing angle, which can usually reach 180 degrees or even more than 180 degrees. Its imaging characteristic is that it will cause serious barrel distortion of the picture. The captured image presents a circular or approximately circular shape, which can cover a very wide scene in one picture. In order to achieve more comprehensive scene coverage, especially in some scenes that require 360-degree panoramic shooting, two fisheye lenses are used for shooting. The two fisheye lenses can be installed on the same device to obtain images from different angles or directions. A common layout method is to point the two lenses in different directions so that their viewing angles can be combined to cover the full range of scenes as much as possible. That is, the lenses described in the present invention are two lenses set back to back. More specifically, an Equirectangular panorama, also often referred to as an equidistant cylindrical projection panorama, is a specific presentation method that maps a 360-degree full-range scene to a two-dimensional plane. In this panorama, the horizontal direction fully displays a 360-degree viewing angle, corresponding to the width of the image; the vertical direction displays a 180-degree viewing angle, corresponding to the height of the image. Its aspect ratio is generally 2:1, and the overall appearance is a rectangular image. When using a device that can capture a wide viewing angle such as a dual fisheye lens, the resulting image is often highly distorted and deformed, and may be divided into different parts (such as circular images taken by two fisheye lenses). It is not convenient to use directly for display or some subsequent application scenarios that require a conventional image format. Expanding it into an equirectgular panorama is to convert this original image with a special viewing angle and distorted shape into an equidistant image through a series of mathematical transformations and image stitching, correction and other processing techniques. To obtain a panorama with a cylindrical projection rule, specifically, it is necessary to reposition and adjust the pixels in the original image according to the imaging model of the lens (e.g., different optical models such as equidistant projection and stereoscopic projection of a fisheye lens). For an image taken by a dual fisheye lens, firstly, the circular images taken by the two fisheye lenses should be distorted and corrected, and the pixels therein should be restored according to their actual corresponding spatial positions. Then, the corrected images should be spliced ​​and fused according to the requirements of the equidistant cylindrical projection, and the image parts that were originally independent, possibly overlapping, and irregular in shape should be reasonably combined into a complete equirectgular panorama presented in a rectangular form, so that the distribution of each pixel in the image in the horizontal and vertical directions conforms to the equidistant rule of the panoramic view.

[0051] S2. The effective visualization area mask of the two fisheye images in S1 is expanded into panoramic images m1 and m2. In the field of image processing, mask usually refers to a binary image, which is used to mark the area of ​​interest or the specific part that needs to be paid attention to and processed. The visualization area mask specifically defines which areas in the image are expected to be effective in subsequent visualization operations, and excludes other areas. In images taken with fisheye lenses, due to the characteristics of the lens itself, there may be dark corners, severe distortions, and other areas at the edge of the image that lead to poor imaging effects. Or when the panoramic image is collected, there are some parts that do not need to be displayed due to equipment occlusion and other reasons. By creating a mask, the pixels corresponding to these invalid areas are marked as 0 (indicating no attention and no processing), and the pixels in the effective area are marked as 1 (or other appropriate values, indicating that they will be focused on processing and display later), so as to define the visualization area.

[0052] S3, coarse fusion panorama, using p1 to fuse p2 to get p12, using p2 to fuse p1 to get p21, the fusion calculation of p21 in S3 is:

[0053] p21=p2(m2)+p1(!m2), where (m2) represents the effective visualization area mask expanded panorama m2 of the fisheye image in S2, and (!m2) represents the inversion in the effective visualization area mask expanded panorama m2 of the fisheye image in S2;

[0054] The fusion calculation of p21 in S3 is:

[0055] p21=p2(m2)+p1(!m2), where (m2) represents the effective visualization area mask of the fisheye image in S2 to expand the panorama m2, and (!m2) represents the inverse of the effective visualization area mask of the fisheye image in S2 to expand the panorama m2. When making a panorama, for example, multiple images are obtained from different directions through dual fisheye lenses. The coarse fusion stage is to roughly splice these images together. Taking the images taken by two fisheye lenses as an example, first of all, the circular images taken by them are simply aligned, and the corresponding position relationship is found according to some feature points in the image (such as obvious markers and edges in the picture). Then, the overlapping parts of the two images are preliminarily superimposed and spliced. At this time, the overlapping areas may be simply processed in a relatively simple and direct way such as the average pixel value, so that they initially form an image that looks like a complete panorama, but there may be problems such as inconsistent colors and obvious splicing marks in the overlapping areas. This is the effect of coarse fusion. Subsequent fine fusion operations such as fine color correction, light and shadow adjustment, and elimination of splicing marks are required;

[0056] S4, optical flow estimation, specifically using deep learning algorithms to estimate the optical flow information of p12 and p21 in S3, and more specifically, by introducing a multi-scale feature extraction mechanism and a context-aware module, the accuracy and robustness of optical flow estimation in complex scenes are improved;

[0057] S5. Design the transition zone regional fusion coefficient, the calculation formula is as follows:

[0058]

[0059] In the formula, blend ij is the fusion coefficient, which varies between [0,1], Fmin ij With Bmin ij They represent the closest distances between the pixel (i, j) of the two spherical fisheye lenses and m1 and m2 respectively;

[0060] S6, using the optical flow information in S4 combined with the fusion coefficient in S5 to weight each coefficient in the transition zone area, and then introducing the ghost coefficient ρ to calculate the specific color value of each pixel, thereby significantly reducing the visual artifacts in the stitching area;

[0061] S7, finely fusion of the panorama, when finely fusion of the panorama in S7, for each pixel (x, y) in the panorama, the optical flow vector fLR(x, y) from the first fisheye image to the second fisheye image and the optical flow vector fRL(x, y) from the second fisheye image to the first fisheye image are used to calculate the fusion weights wF(x, y) and wB(x, y) of the two images according to the fusion mask m(x, y), wherein the calculation formula of wF(x, y) is:

[0062] wF(x,y)=b(x,y);

[0063] The calculation formula of wB(x,y) is:

[0064] wB(x,y)=1-b(x,y);

[0065] Then, the color value C(x,y) of the pixel (x,y) in the fused panoramic image is calculated based on the fusion weights wF(x,y) and wB(x,y) of the two images and the ghost coefficient ρ(rho).

[0066] More specifically, the calculation formula for the color value C(x, y) of the pixel (x, y) in the panoramic image is:

[0067] C(x,y)=CF(x,y)*wF(x,y)*rho(x,y)+CB(x,y)*wB(x,y)*rho(x,y);

[0068] S8. Panoramic image output.

[0069] The present invention can be divided into two parts as a whole. The first part is to perform optical flow estimation on the panoramic image expanded by two dual fisheye images, and the second part is to perform local fusion of the overlapping area, that is, geometric distortion correction.

[0070] The present invention has the following advantages:

[0071] Improve stitching accuracy and stability. In complex scenes with changing lighting, fast motion, and less texture, the existing optical flow algorithm cannot effectively estimate the motion vector, resulting in dislocation, cracks, or edge discontinuity at the stitching picture boundary. The present invention can efficiently utilize the optical flow estimation model to achieve a stitching effect with higher accuracy and robustness.

[0072] Improve the real-time performance of the stitching algorithm. The traditional optical flow algorithm has a large amount of calculation and high time overhead in high-resolution and multi-lens scenes, and it is difficult to meet the needs of real-time video stitching. The present invention adopts a lightweight optical flow network model to significantly reduce the calculation complexity while ensuring accuracy, thereby improving real-time processing performance.

[0073] Optimize the picture consistency of dynamic scenes. Moving objects in dynamic scenes are prone to introduce splicing errors (such as ghosting or blurring). The present invention designs a dedicated optical flow splicing strategy to enhance the processing capabilities of object boundaries and details in dynamic scenes and achieve smooth transition of pictures.

[0074] Improve the smoothing effect of seam lines and geometric distortion correction. In the seam area of ​​multi-lens stitching, traditional methods often find it difficult to balance the consistency of the global image and the fidelity of local details, resulting in blurred or artifact stitching lines. The present invention optimizes seam line processing and reduces visual artifacts by integrating optical flow estimation and geometric correction modules.

[0075] The present invention mainly solves various defects existing in traditional panoramic stitching technology. Compared with the existing technology, it has the advantages of high-precision optical flow stitching, dynamic scene optimization and panoramic picture quality improvement. It uses lightweight and high-efficiency optical flow estimation to improve the accuracy and continuity near the stitching line, reduce edge dislocation and cracks, and adopts an adaptive stitching strategy for dynamic objects to significantly reduce ghosting and blurring, enhance picture smoothness, and reduce visual artifacts and blur transitions through coarse-fine fusion strategy and stitching line optimization, so as to achieve seamless panoramic effect and improve user experience.

[0076] The present invention also discloses a camera, which includes a processor and a memory. The processor is used to execute the steps of the above-mentioned panoramic stitching method by calling a program or instruction stored in the memory.

[0077] The present invention also discloses a storage medium, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction executes the steps of the above-mentioned panoramic stitching method.

[0078] This embodiment does not impose any formal limitation on the shape, material, structure, etc. of the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. Panoramic stitching method, based on deep learning algorithm, characterized by: The following steps are involved: S1, using two spherical fisheye lenses to obtain two fisheye images and expand them into Equirectangular panoramas to obtain p1 and p2; S2, the effective visualization area mask of the two fisheye images in S1 is expanded into panorama m1 and m2; S3, coarse fusion panorama, using p1 to fuse p2 to get p12, and using p2 to fuse p1 to get p21; S4, use deep learning algorithm to estimate the optical flow information of p12 and p21 in S3 respectively; S5. Design the transition zone regional fusion coefficient, the calculation formula is as follows: In the formula, blend ij is the fusion coefficient, which varies between [0,1], Fmin ij With Bmin ij They represent the closest distances between the pixel (i, j) of the two spherical fisheye lenses and m1 and m2 respectively; S6, using the optical flow information in S4 combined with the fusion coefficient in S5 to weight each coefficient in the transition zone area, and then introducing the ghost coefficient ρ to calculate the specific color value of each pixel; S7, fine fusion panorama; S8. Panoramic image output.

2. The panoramic stitching method according to claim 1, characterized in that: The fusion calculation of p12 in S3 is: p12=p1(m1)+p2(!m1), where (m1) represents the effective visualization area mask expanded panoramic image m1 of the fisheye image in S2, and (!m1) represents the inversion of the effective visualization area mask expanded panoramic image m1 of the fisheye image in S2.

3. The panoramic stitching method according to claim 1, characterized in that: The fusion calculation of p21 in S3 is: p21=p2(m2)+p1(!m2), where (m2) represents the effective visualization area mask expanded panoramic image m2 of the fisheye image in S2, and (!m2) represents the inversion of the effective visualization area mask expanded panoramic image m2 of the fisheye image in S2.

4. The panoramic stitching method according to claim 1, characterized in that: The deep learning algorithm is any one of the fastFlowNet algorithm, the RAFT algorithm or the memFlowNet algorithm.

5. The panoramic stitching method according to claim 1, characterized in that: When the panorama is finely fused in S7, for each pixel (x, y) in the panorama, the optical flow vector fLR(x, y) from the first fisheye image to the second fisheye image and the optical flow vector fRL(x, y) from the second fisheye image to the first fisheye image are used to calculate the fusion weights wF(x, y) and wB(x, y) of the two images according to the fusion mask m(x, y), wherein the calculation formula of wF(x, y) is: wF(x,y)=b(x,y); The calculation formula of wB(x,y) is: wB(x,y)=1-b(x,y); Then, the color value C(x,y) of the pixel (x,y) in the fused panoramic image is calculated based on the fusion weights wF(x,y) and wB(x,y) of the two images and the ghost coefficient ρ(rho).

6. The panoramic stitching method according to claim 5, characterized in that: The calculation formula of the color value C(x,y) of the pixel (x,y) in the panoramic image is: C(x,y)=CF(x,y)*wF(x,y)*rho(x,y)+CB(x,y)*wB(x,y)*rho(x,y).

7. Camera, characterized in that: The camera includes a processor and a memory, and the processor is used to execute the steps of the panoramic stitching method according to any one of claims 1 to 6 by calling a program or instruction stored in the memory.

8. A storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction executes the steps of the panoramic stitching method according to any one of claims 1 to 6.

Images