Panoramic image stitching method based on hyper-parameter optimization, camera and storage medium

Abstract

The invention discloses a panoramic image stitching method based on hyper-parameter optimization, a camera and a storage medium, the method is realized based on a hyper-parameter optimization algorithm and a stitching fusion algorithm, and the method comprises the following steps: S1, using a 360-degree motion camera to shoot a double-fisheye image; s2, setting types and variation ranges of to-be-optimized hyper-parameters of the two fisheye cameras of the 360-degree motion camera in the S1; s3, using a Gaussian model f (x) as a proxy function model of a hyper-parameter optimization algorithm; s4, performing hyper-parameter optimization on the to-be-optimized hyper-parameters of the two fisheye cameras of the 360-degree motion camera set in the S2 by using the hyper-parameter optimization algorithm optimized and updated in the S3; s5, converting the double-fisheye image in the step S1 and the optimized hyper-parameters fov, pitch, roll and yaw in the step S4 into a local equipartition type panorama by using a splicing fusion algorithm; and S6, carrying out fusion on the transition zone regions of the local equipartition type panorama in the step S5 by using a splicing fusion algorithm. The method has the advantages of automatic optimization, efficiency improvement and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing computer vision algorithms, and in particular relates to a panoramic image stitching method based on hyperparameter optimization, a camera and a storage medium. Background Art

[0002] At present, the application of cameras in daily life is becoming more and more common. In addition to ordinary cameras to meet basic photo and video needs, consumers also need to use sports cameras for shooting. Sports cameras are different from ordinary cameras in the traditional sense. Their design, manufacturing, and imaging parameter specifications are significantly higher than those of traditional ordinary cameras. They are generally composed of two fisheye cameras with an angle of approximately 180 degrees to each other. Each fisheye camera is responsible for shooting a field of view slightly larger than 180 degrees in the corresponding direction, so that a panoramic preview can be performed at 360 degrees without blind spots. Visually, in order to stitch the transition zone area more smoothly, the traditional method generally uses feature points or control points to match images, then calculates the mapping matrix, and finally performs common view area fusion. Although this type of method can achieve good stitching effects, it has the following shortcomings: 1. Because the image in the common view area is severely distorted and the visible range is small, it is not easy to find the corresponding control matching points on the dual fisheye image. Even if the visual operators SIFT, SURF, and ORB are used for detection, it is necessary to manually select the matching control matching points, which generally requires the help of image stitching software, such as PTGui; 2. It requires multiple steps to complete, such as generating a configuration file, and then batch stitching and fusion of dual fisheye images taken by the same camera requires a lot of manual processing in the early stage, which is time-consuming and labor-intensive. The final stitching effect depends largely on the quality and quantity of the selected control points. Once there are erroneous control points / feature points, the stitching effect process will cause irregular distortion of the original inherent image; 3. There are certain requirements for dual fisheye images, especially the scene features in the common view area should be as rich as possible, so that matching points can be easily found intuitively. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the present invention provides a panoramic image stitching method, a camera and a storage medium.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A panoramic image stitching method based on hyperparameter optimization, which is implemented based on a hyperparameter optimization algorithm and a stitching fusion algorithm, includes the following steps:

[0006] S1. Use a 360-degree motion camera to capture dual fisheye images;

[0007] S2. Setting the types and ranges of hyperparameters to be optimized of the two fisheye cameras of the 360-degree motion camera in S1, wherein the types of hyperparameters to be optimized of each of the two fisheye cameras of the 360-degree motion camera in S1 include fov, pitch, roll, and yaw, and the units of fov, pitch, roll, and yaw are degrees and are all real numbers;

[0008] S3, using the Gaussian model f(x) as a proxy function model of the hyperparameter optimization algorithm, and optimizing and updating the hyperparameter optimization algorithm using the existing observation data;

[0009] S4, using the hyperparameter optimization algorithm optimized and updated in S3 to optimize the hyperparameters to be optimized of the two fisheye cameras of the 360-degree motion camera set in S2, and obtain the best data point x, the best data point x includes the hyperparameters fov, pitch, roll, and yaw optimized by the hyperparameter optimization algorithm;

[0010] S5, using a stitching fusion algorithm to convert the dual fisheye images in S1 and the optimized hyper parameters fov, pitch, roll, and yaw in S4 into a local equirectangular type panoramic image;

[0011] S6, using a stitching fusion algorithm to fuse the transition zone area of ​​the local equirectangular type panorama in S5, and output a 360-degree stitching fusion panorama, wherein the formula for fusing the transition zone area by the stitching fusion algorithm is:

[0012] I(x,y)=α(x)I1(x,y)+(1-α(x))I2(x,y),

[0013] Where I(x,y) is the pixel value of the fused image at the pixel point (x,y), α(x) is the weight, and I1(x,y) and I2(x,y) are the local panoramas taken by the two fisheye cameras of the 360-degree motion camera.

[0014] Preferably, the value of each pixel point is obtained by inverse mapping in S5, and S5 includes a 2D equirectangular to 3D vector step and a 3D vector to 2D fisheye step, wherein the formula used in the 2D equirectangular to 3D vector step is:

[0015]

[0016] Where (longitude, latitude) is longitude and latitude, (P x,,P y ,P z ) is the coordinate of a point in space;

[0017] The formula used in the 3D vector to 2D fisheye step is:

[0018]

[0019] Where r is the polar diameter, θ is the polar angle, (P x, ,P y ,P z ) are the coordinates of a point in space.

[0020] The present invention also discloses a camera, which includes a processor and a memory. The processor is used to execute the steps of the above-mentioned panoramic image stitching method based on hyperparameter optimization by calling a program or instruction stored in the memory.

[0021] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction executes the steps of the above-mentioned panoramic image stitching method based on hyperparameter optimization.

[0022] By adopting the above technical solution, the present invention has the following beneficial effects:

[0023] (1) The present invention relies on the hyperparameter optimization algorithm to automatically optimize the hyperparameters. There is no need to manually select control points or feature points, nor is there any need to design visual detection operators to detect feature points. It is only necessary to specify the hyperparameters for the variables that need to be optimized and solved, thereby reducing unnecessary work expenses, saving time, and improving efficiency. Moreover, there is no need to use image stitching software.

[0024] (2) The present invention can reduce the distortion of the stitched image and improve the accuracy by using the stitching and fusion algorithm. The existing image stitching technology generally first dedistorts the fisheye image, finds obvious features in the common view area, then matches, and then calculates the homography matrix. The steps are linked together. Once there is an error in one step, it will have a great impact on the quality of the subsequent stitched images. The present invention directly estimates the hyperparameters based on the original fisheye image, and there will be no error accumulation effect. It only relies on the hyperparameter optimization algorithm and the stitching and fusion algorithm, thereby improving the accuracy.

[0025] In summary, the present invention has the advantages of automatic optimization, improved efficiency, reduced distortion of spliced ​​images, improved accuracy, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of stitching dual fisheye images in the present invention;

[0027] Figure 2is an example of a dual fisheye image captured by the present invention;

[0028] Figure 3 It is a schematic diagram of 2D equirectangular to 3D vector in the present invention;

[0029] Figure 4 It is a schematic diagram of 3D vector to 2D fisheye in the present invention;

[0030] Figure 5 It is an example of a partial panoramic image converted by a stitching and fusion algorithm using an image captured by one of the two fisheye cameras of the 360-degree motion camera in the present invention;

[0031] Figure 6 It is an example of a partial panoramic image obtained by converting an image captured by another fisheye camera of the two fisheye cameras of the 360-degree motion camera of the present invention through a stitching and fusion algorithm;

[0032] Figure 7 The present invention uses a splicing fusion algorithm to Figure 5 and Figure 6 An example of a 360-degree stitched and fused panorama output after stitching and fusion. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0034] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0035] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0036] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] Example 1

[0039] In this embodiment, the present invention proposes a panoramic image stitching method based on hyperparameter optimization, a camera and a storage medium, which can complete hyperparameter optimization and panoramic image stitching by using a hyperparameter optimization algorithm and a stitching melting algorithm with less manual intervention, mainly including dual fisheye image input, specifying optimized hyperparameters, specifically including the type and estimated value range of each hyperparameter, the number of algorithm termination iterations or error limits, and finally outputting a 360-degree panoramic image. This method does not require control point or feature point matching steps, only requires an appropriate amount of common view area, does not require any third-party software or tool assistance, and directly specifies the hyperparameters that need to be optimized, such as field of view FOV, pitch angle pitch, roll angle roll, yaw angle yaw, etc., has few specific processing processes, does not require multiple steps, specifies once, and the algorithm automatically optimizes and solves the hyperparameters. The whole process is fast and convenient, not prone to errors, and the image will not have irregular distortion. The algorithm output parameters are suitable for fast stitching of dual fisheye images of the same camera, and are easy to deploy and produce.

[0040] like Figure 1 As shown, in one embodiment of the present invention, the panoramic image stitching method based on hyperparameter optimization of the present invention is implemented based on a hyperparameter optimization algorithm and a stitching fusion algorithm, and includes the following steps:

[0041] S1. Use a 360-degree motion camera to shoot dual fisheye images. The 360-degree motion camera has two fisheye cameras. The images it shoots can be referred to Figure 2 As shown;

[0042] S2. Set the type and variation range of the hyperparameters to be optimized for the two fisheye cameras of the 360-degree motion camera in S1. The setting here is a hypothetical setting. Because there are two annular fisheye images in S1, that is, they are taken by two fisheye cameras, and the distortion radius and Fov are related variables, so the type and variation range of the relevant hyperparameters are set for the two fisheye cameras. Among them, the type of hyperparameters to be optimized for each of the two fisheye cameras of the 360-degree motion camera in S1 includes fov, pitch, roll, yaw, and the units of fov, pitch, roll, and yaw are degrees and are all real numbers. Fov is the field of view angle, pitch is the pitch angle, roll is the roll angle, and yaw is the yaw angle. There are 8 related hyperparameters for the two fisheye cameras. Specifically, the variation ranges of fov, pitch, roll, and yaw of the first fisheye camera are as follows:

[0043]

[0044] In the same world coordinate system, the yaw angles of the second fisheye camera and the first fisheye camera are approximately 180 degrees apart, so the yaw range of the second fisheye camera is yaw∈[175, 185], and the Fov, pitch, and roll ranges of the second fisheye camera are the same as those of the first fisheye camera.

[0045] S3, using the Gaussian model f(x) as the proxy function model of the hyperparameter optimization algorithm, and optimizing and updating the hyperparameter optimization algorithm using the existing observation data, that is, before using the present invention, using the existing method or means to collect relevant historical observation data, and using these existing observation data to update the proxy function model of the hyperparameter optimization algorithm to make it closer to the real objective function;

[0046] S4, using the hyperparameter optimization algorithm optimized and updated in S3 to optimize the hyperparameters to be optimized of the two fisheye cameras of the 360-degree motion camera set in S2, and obtain the best data point x, the best data point x includes the hyperparameters fov, pitch, roll, yaw optimized by the hyperparameter optimization algorithm, and continuously iterate and update the hyperparameter optimization algorithm to find the relatively best data point, so as to maximize the function a(x). In this device, the termination condition of the hyperparameter optimization algorithm is a fixed number of iterations or a fixed deadline until the best data point x is obtained;

[0047] S5, using the stitching fusion algorithm to convert the dual fisheye images in S1 and the optimized hyper parameters fov, pitch, roll, yaw in S4 into a local equirectangular type panorama, S5 uses the inverse mapping method to obtain the value of each pixel point, S5 includes the 2D equirectangular to 3D vector step and the 3D vector to 2D fisheye step, where the formula used in the 2D equirectangular to 3D vector step is:

[0048]

[0049] Where (longitude, latitude) is longitude and latitude, (P x, ,P y ,P z ) is the coordinate of a point in space, the specific transformation diagram is as follows Figure 3 As shown;

[0050] The formula used in the 3D vector to 2D fisheye step is:

[0051]

[0052] Where r is the polar diameter, θ is the polar angle, (P x, ,P y ,P z ) is the coordinate of a point in space, the specific transformation diagram is as follows Figure 4 As shown, the formula in S5 is applied to each pixel of the image taken by the dual fisheye camera, and the following can be obtained: Figure 5 and Figure 6 The local equirectangular type panorama shown;

[0053] S6, using the stitching fusion algorithm to fuse the transition zone area of ​​the local equirectangular type panorama in S5, so as to eliminate the obvious dividing line, make the image transition more natural, and output a 360-degree stitching fusion panorama, wherein the formula for the stitching fusion algorithm to fuse the transition zone area is:

[0054] I(x,y)=α(x)I1(x,y)+(1-α(x))I2(x,y)

[0055] Where I(x,y) is the pixel value of the fused image at the pixel point (x,y), α(x) is the weight, I1(x,y) and I2(x,y) are the local panoramas taken by the two fisheye cameras of the 360-degree motion camera, respectively. The final stitching and fusion effect image, that is, the output 360-degree stitching and fusion panorama is as follows: Figure 7 As shown, FrontFisheye in the accompanying drawings of the present invention is an example of an image taken by the first fisheye camera, and BackFisheye is an example of an image taken by the second fisheye camera.

[0056] The present invention also discloses a camera, which includes a processor and a memory. The processor is used to execute the steps of the above-mentioned panoramic image stitching method based on hyperparameter optimization by calling a program or instruction stored in the memory.

[0057] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction executes the steps of the above-mentioned panoramic image stitching method based on hyperparameter optimization.

[0058] This embodiment does not impose any formal limitation on the shape, material, structure, etc. of the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A panoramic image stitching method based on hyperparameter optimization, characterized in that: The method is implemented based on a hyperparameter optimization algorithm and a splicing fusion algorithm, and includes the following steps: S1. Use a 360-degree motion camera to capture dual fisheye images; S2. Setting the types and ranges of hyperparameters to be optimized of the two fisheye cameras of the 360-degree motion camera in S1, wherein the types of hyperparameters to be optimized of each of the two fisheye cameras of the 360-degree motion camera in S1 include fov, pitch, roll, and yaw, and the units of fov, pitch, roll, and yaw are degrees and are all real numbers; S3, using the Gaussian model f(x) as a proxy function model of the hyperparameter optimization algorithm, and optimizing and updating the hyperparameter optimization algorithm using the existing observation data; S4, using the hyperparameter optimization algorithm optimized and updated in S3 to optimize the hyperparameters to be optimized of the two fisheye cameras of the 360-degree motion camera set in S2, and obtain the best data point x, the best data point x includes the hyperparameters fov, pitch, roll, and yaw optimized by the hyperparameter optimization algorithm; S5, using a stitching fusion algorithm to convert the dual fisheye images in S1 and the optimized hyper parameters fov, pitch, roll, and yaw in S4 into a local equirectangular type panoramic image; S6, using a stitching fusion algorithm to fuse the transition zone area of ​​the local equirectangular type panorama in S5, and output a 360-degree stitching fusion panorama, wherein the formula for fusing the transition zone area by the stitching fusion algorithm is: I(x,y)=α(x)I1(x,y)+(1-α(x))I2(x,y), Where I(x,y) is the pixel value of the fused image at the pixel point (x,y), α(x) is the weight, and I1(x,y) and I2(x,y) are the local panoramas taken by the two fisheye cameras of the 360-degree motion camera.

2. The panoramic image stitching method based on hyperparameter optimization according to claim 1, characterized in that: In S5, the value of each pixel is obtained by inverse mapping. S5 includes a 2D equirectangular to 3D vector step and a 3D vector to 2D fisheye step. The formula used in the 2D equirectangular to 3D vector step is: Where (longitude, latitude) is longitude and latitude, (P x, ,P y ,P z ) is the coordinate of a point in space; The formula used in the 3D vector to 2D fisheye step is: Where r is the polar diameter, θ is the polar angle, (P x, ,P y ,P z ) are the coordinates of a point in space.

3. Camera, characterized in that: The camera includes a processor and a memory, and the processor is used to execute the steps of the panoramic image stitching method based on hyperparameter optimization as described in any one of claims 1 to 2 by calling a program or instruction stored in the memory.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction executes the steps of the panoramic image stitching method based on hyperparameter optimization as described in any one of claims 1 to 2.

Images