Panoramic image fusion stitching method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for fusion and stitching of panoramic images. The method comprises: obtaining image frames to be fused of each camera in the same shooting cycle in a panoramic stitching and fusion scene; determining the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene; using the image pixel points of each camera to fill the blank imaging sphere to obtain a filled imaging sphere; determining the optical flow value corresponding to the image pixel points of each camera; performing optical flow layering on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer; performing image fusion processing on each image optical flow layer to obtain a fused panoramic image in a panoramic stitching and fusion scene. The embodiment of the present invention improves the stitching and fusion accuracy and fusion efficiency of the panoramic image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image fusion and stitching method, device, equipment and storage medium. Background Art

[0002] With the widespread application of augmented reality and virtual reality in daily life and work, the requirements for visual technology are increasing, and the demand for the effect and quality of panoramic image fusion and stitching is also gradually increasing.

[0003] The current real-time 360-degree panoramic computing based on GPU (Graphics Processing Unit) is mainly used for VR (Virtual Reality) panoramic stitching and fusion with small parallax.

[0004] In the process of panorama stitching and fusion, calibration is first performed in a fixed scene in the calibration room to generate a lookup table that maps one-to-one between the spherical coordinate system and the image coordinate system. GPU real-time calculation only needs to parallel map the generated lookup table. The complexity of each GPU kernel function is low, and there is no correlation between kernel functions. This method is only suitable for panoramic fusion of small parallax scenes. For large parallax scenes, this fusion method will have low accuracy of panoramic fusion. In the fusion process, a large amount of calculations need to be performed on each camera, which will also lead to low efficiency of panoramic fusion. Summary of the invention

[0005] The present invention provides a panoramic image fusion and stitching method, device, equipment and storage medium to improve the stitching and fusion accuracy and fusion efficiency of the panoramic image.

[0006] According to one aspect of the present invention, a panoramic image fusion and stitching method is provided, the method comprising:

[0007] Obtain the image frames to be fused from each camera in the same shooting cycle in the panoramic stitching and fusion scene;

[0008] According to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created, based on the image frames to be fused of each camera, determine the image pixel points of each camera on the blank imaging sphere;

[0009] Filling the blank imaging sphere with image pixels of each camera to obtain a filled imaging sphere;

[0010] Determine the optical flow value corresponding to the image pixel point of each camera;

[0011] According to the optical flow values ​​and image pixels of each camera, optical flow layering is performed on the filled imaging sphere to obtain at least one image optical flow layer;

[0012] Image fusion processing is performed on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene.

[0013] According to another aspect of the present invention, a panoramic image fusion and stitching device is provided, the device comprising:

[0014] An image frame acquisition module is used to acquire image frames to be fused from each camera in the same shooting cycle in a panoramic stitching and fusion scene;

[0015] A pixel point determination module, used for determining the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created;

[0016] A filling spherical surface determination module, used to fill the blank imaging spherical surface with image pixels of each camera to obtain a filled imaging spherical surface;

[0017] An optical flow value determination module, used to determine the optical flow value corresponding to the image pixel point of each camera;

[0018] An optical flow layer generation module, used to perform optical flow layering on the filled imaging sphere according to the optical flow values ​​and image pixels of each camera to obtain at least one image optical flow layer;

[0019] The panoramic image generation module is used to perform image fusion processing on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the panoramic image fusion and stitching method described in any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the panoramic image fusion and stitching method described in any embodiment of the present invention when executed.

[0025] The technical solution of the embodiment of the present invention obtains the image frames to be fused of each camera in the same shooting cycle in a panoramic stitching and fusion scene, determines the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene, fills the blank imaging sphere with the image pixel points of each camera to obtain a filled imaging sphere, determines the optical flow value corresponding to the image pixel points of each camera, performs optical flow stratification on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer, performs image fusion processing on each image optical flow layer to obtain a fused panorama in the panoramic stitching and fusion scene, realizes real-time stitching of the panorama in the panoramic stitching and fusion scene, and improves the stitching and fusion accuracy and fusion efficiency of the panorama.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1A is a flow chart of a panoramic image fusion and stitching method provided according to the first embodiment of the present invention;

[0029] Figure 1B is a schematic diagram of a data storage form of an image frame to be fused provided in Embodiment 1 of the present invention;

[0030] Figure 1C This is a schematic diagram of a method execution of an asynchronous copy double buffer mechanism provided according to Embodiment 1 of the present invention;

[0031] Figure 2 is a flow chart of a panoramic image fusion and stitching method provided according to the second embodiment of the present invention;

[0032] Figure 3 is a flow chart of a panoramic image fusion and stitching method provided according to Embodiment 3 of the present invention;

[0033] Figure 4 is a structural schematic diagram of a panoramic image fusion and stitching device provided according to a fourth embodiment of the present invention;

[0034] Figure 5It is a structural schematic diagram of an electronic device for implementing the panoramic image fusion and stitching method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1A This is a flow chart of a panoramic image fusion and stitching method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of performing panoramic image stitching and fusion in a large parallax or small parallax scene. The method can be executed by a panoramic image fusion and stitching device. The panoramic image fusion and stitching device can be implemented in the form of hardware and / or software. The panoramic image fusion and stitching device can be configured in an electronic device. Figure 1A As shown, the method includes:

[0039] S110, obtaining image frames to be fused from each camera in the same shooting cycle in a panoramic stitching and fusion scene.

[0040] The cameras may be located at different shooting angles and positions in the panoramic stitching fusion scene. The same shooting cycle may be the image frames to be fused obtained by each camera in the same shooting cycle. For example, if there are cameras A, B, and C, then camera A obtains the nth image frame N to be fused in the Tth cycle. AT , the nth frame of image to be fused obtained by camera B in the Tth cycle N BT And the nth frame of image frame N to be fused obtained by camera C in the Tth cycleCT .

[0041] It should be noted that the image frames to be fused of each camera can be obtained in the GPU memory. The camera parameter information of each camera is pre-stored in the constant memory of the GPU during initialization. For example, the camera parameter information may include the image width and image height output by the camera and the internal and external parameters of the camera calibration.

[0042] For example, Figure 1B The schematic diagram of a data storage format of an image frame to be fused is shown. For the image frames to be fused generated by cameras 1 to n under the same shooting cycle, they can be stored row by row in a pre-constructed one-dimensional data group based on RGB (Red, Green, Blue) channels.

[0043] Exemplarily, the image frames to be fused of each camera in the same shooting cycle in the panoramic stitching and fusion scene can be obtained from the CPU (Central Processing Unit) memory, and the image frames to be fused obtained based on asynchronous copying can be executed in the GPU video memory to perform the subsequent panoramic stitching and fusion process. However, it should be noted that the process of copying image frames from the memory to the video memory, and the process of executing the panoramic fusion operation in the GPU video memory, both consume a lot of time. Before the next frame of image is copied to the GPU video memory, the panoramic operation process of the current frame image must be completed before the next frame of image can be copied. In order to improve the efficiency of panoramic stitching and fusion, a parallel double buffer mechanism can be used in the process of copying image frames and panoramic stitching and fusion operations to improve execution efficiency.

[0044] In an alternative embodiment, if Figure 1C A schematic diagram of the execution of a method of an asynchronous copy double buffer mechanism. The GPU video memory is divided into cache A and cache B. At time n, when cache A performs a panoramic stitching and fusion operation based on the n-1th frame of the image frame to be fused of each camera, cache B can asynchronously copy the nth frame of the image frame to be fused of each camera from the memory to its own cache. In addition, at time n+1, cache B performs a panoramic stitching and fusion operation based on the nth frame of the image frame to be fused of each camera. At this time, cache A asynchronously copies the n+1th frame of the image frame to be fused of each camera from the memory to its own cache. At time n+2, cache A performs a panoramic stitching and fusion operation based on the n+1th frame of the image frame to be fused of each camera. At this time, cache B asynchronously copies the n+2th frame of the image frame to be fused of each camera from the memory to its own cache.

[0045] S120 , according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene, based on the image frames to be fused of each camera, determine the image pixel points of each camera on the blank imaging sphere.

[0046] The spherical angle resolution and imaging spherical radius of the blank imaging sphere can be preset by relevant technicians according to actual needs and in combination with the panoramic stitching and fusion scene. A blank imaging sphere of the panoramic stitching and fusion scene is created according to the spherical angle resolution and imaging spherical radius.

[0047] The spherical angle resolution and imaging spherical radius of the blank imaging sphere may also be pre-stored in the constant memory of the GPU during initialization.

[0048] Exemplarily, the number k of pixel points generated by each camera on the blank imaging sphere can be determined according to the spherical angle resolution of the created blank imaging sphere, wherein k can be determined as follows:

[0049]

[0050] Among them, r represents the spherical angle resolution, and the initial values ​​of i and j are 0. It should be noted that the number k of pixels on each camera is the same.

[0051] Exemplarily, the world coordinate point corresponding to each pixel point among the k pixels on each camera is determined; according to the world coordinate point, based on the image frames to be fused of each camera, the image pixel points corresponding to the world coordinate points in the image frames to be fused are determined, thereby obtaining the image pixel points of each camera on the blank imaging sphere.

[0052] S130 , using the image pixels of each camera to fill the blank imaging sphere to obtain a filled imaging sphere.

[0053] Exemplarily, the image pixels of each camera are filled onto a blank imaging sphere, and the imaging sphere obtained after filling is used as a filled imaging sphere.

[0054] S140: Determine the optical flow value corresponding to the image pixel of each camera.

[0055] Exemplarily, the optical flow values ​​corresponding to the image pixels of each camera may be determined based on a preset optical flow algorithm, wherein the optical flow algorithm may be preset by relevant technical personnel, for example, the optical flow algorithm may be a HS (Horn & Schunck) optical flow algorithm.

[0056] S150 , performing optical flow layering on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer.

[0057] Exemplarily, the optical flow values ​​may be grouped according to the optical flow values ​​corresponding to the image pixels of each camera to obtain at least one optical flow value group. Based on each optical flow value group, the image pixels are grouped, and specifically, each pixel mapped to each optical flow value group is an image pixel group. Based on each image pixel group, the filled imaging sphere is optically layered to obtain at least one image optical flow layer.

[0058] S160: Perform image fusion processing on the optical flow layers of each image to obtain a fused panoramic image in a panoramic stitching and fusion scene.

[0059] Exemplarily, according to a preset image fusion algorithm, image fusion processing is performed on each image optical flow layer to obtain a fused panoramic image in a panoramic stitching fusion scene. The image fusion algorithm can be preset by relevant technical personnel. For example, the image fusion algorithm can be a SIFT (Scale Invariant Feature Transform) algorithm.

[0060] The technical solution of the embodiment of the present invention obtains the image frames to be fused of each camera in the same shooting cycle in a panoramic stitching and fusion scene, determines the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene, fills the blank imaging sphere with the image pixel points of each camera to obtain a filled imaging sphere, determines the optical flow value corresponding to the image pixel points of each camera, performs optical flow stratification on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer, performs image fusion processing on each image optical flow layer to obtain a fused panorama in the panoramic stitching and fusion scene, realizes real-time stitching of the panorama in the panoramic stitching and fusion scene, and improves the stitching and fusion accuracy and fusion efficiency of the panorama.

[0061] Embodiment 2

[0062] Figure 2 This is a flow chart of a panoramic image fusion and stitching method provided in the second embodiment of the present invention. This embodiment is optimized and improved on the basis of the above technical solutions.

[0063] Further, the step "determine the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the panoramic stitching fusion scene created" is refined into "determine the world coordinate points of each camera on the blank imaging sphere; create at least one core thread; under each core thread, determine the image pixel points of each camera on the blank imaging sphere based on the world coordinate points of each camera and the image frames to be fused, based on the camera imaging model and camera parameter information corresponding to each camera." to improve the method of determining the image pixel points on the blank imaging sphere. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0064] like Figure 2 As shown, the method comprises the following specific steps:

[0065] S210: Obtain image frames to be fused from each camera in the same shooting cycle in a panoramic stitching and fusion scene.

[0066] S220: Create at least one core thread.

[0067] The computing process involved in the panoramic image fusion process performed by the GPU is performed in a pre-created core thread.

[0068] For example, an application may be made to the GPU to create a preset number of core threads, so that the operation flow in the fusion process is executed in parallel in each core thread to improve the execution efficiency of the operation flow, thereby improving the efficiency of panoramic image fusion and stitching.

[0069] In an optional embodiment, the core thread is created as follows: according to the spherical angle resolution of the blank imaging sphere, the pixels to be fused of each camera on the blank imaging sphere are determined; and the same number of core threads as the number of pixels to be fused is created.

[0070] It should be noted that each camera in the panoramic stitching and fusion scene corresponds to an imaging sphere, and a certain number of pixels to be fused are distributed on each imaging sphere.

[0071] The number k of pixels to be fused may be determined as follows:

[0072]

[0073] Where r is the spherical angle resolution of the blank imaging sphere.

[0074] For example, if the number of pixels to be fused is k, an application can be made to the GPU to create k core threads, each of which corresponds to a pixel to be fused on the imaging sphere of each camera. During the execution of the operation, the operation processes of the pixels to be fused complement each other, and the core threads process the pixels to be fused of each camera in parallel, which can improve the execution efficiency of the operation process. Each core thread has its own thread identification number, which corresponds to each pixel to be fused.

[0075] S230, determining the world coordinate point of each camera on the blank imaging sphere.

[0076] Exemplarily, the world coordinate points of each camera on the blank imaging sphere may be world coordinate points corresponding to each pixel point to be fused, and specifically may be coordinate points in the world coordinate system obtained by converting the spherical coordinate system into the Cartesian coordinate system.

[0077] In an optional embodiment, determining the world coordinate point of each camera on the blank imaging sphere includes: determining the polar angle and azimuth angle of each camera on the blank imaging sphere according to the thread identification number of each core thread and the spherical angle resolution; determining the world coordinate point of each camera on the blank imaging sphere according to the polar angle and the azimuth angle.

[0078] Among them, the coordinate points to be fused by each camera are at the polar angle θ and azimuth angle of the blank imaging sphere. is determined as follows:

[0079]

[0080]

[0081] Where r is the spherical angle resolution of the blank imaging sphere; h ID is the thread identification number of the core thread; mod() is the remainder function; round() is the integer function.

[0082] Exemplarily, the method for determining the world coordinate points of the pixels to be fused of each camera on the blank sphere may be as follows:

[0083]

[0084] Wherein, R is the imaging spherical radius of the blank imaging sphere; θ is the polar angle of the blank imaging sphere; is the azimuth of the blank imaging sphere.

[0085] S240 . Under each core thread, according to the world coordinate points of each camera and the image frames to be fused, based on the camera imaging models and camera parameter information corresponding to each camera, determine the image pixel points of each camera on the blank imaging sphere.

[0086] It can be understood that the generation of image pixel points on the blank imaging sphere for each camera is executed in parallel in the core threads.

[0087] Among them, the camera imaging models corresponding to different cameras can be the same or different, and can be pre-set by relevant technical personnel. Before using the camera imaging model, it is necessary to configure the parameters of the camera imaging model, specifically, using the camera parameter information corresponding to each camera to configure the camera imaging model of the corresponding camera. For example, the camera parameter information may include the image width and image height output by the camera and the internal and external parameters of the camera calibration.

[0088] Exemplarily, according to the world coordinate points corresponding to the pixel points to be fused of each camera, based on the pre-configured camera imaging model of each camera, the image pixel points of the world coordinate points output by the model on the image frame to be fused can be determined, thereby obtaining the image pixel points of the camera on the blank imaging sphere.

[0089] S250, using the image pixels of each camera to fill the blank imaging sphere to obtain a filled imaging sphere.

[0090] S260: Determine the optical flow value corresponding to the image pixel point of each camera.

[0091] S270 , performing optical flow layering on the filled imaging sphere according to the optical flow value and the image pixel points of each camera to obtain at least one image optical flow layer.

[0092] S280: Perform image fusion processing on the optical flow layers of each image to obtain a fused panoramic image in a panoramic stitching and fusion scene.

[0093] The technical solution of this embodiment determines the world coordinate points of each camera on the blank imaging sphere by creating at least one core thread. Under each core thread, the image pixel points of each camera on the blank imaging sphere are determined based on the world coordinate points of each camera and the image frame to be fused, based on the camera imaging model and camera parameter information corresponding to each camera. By applying to create a core thread in the GPU and executing the operation flow in the splicing and fusion process in parallel in the core thread, the GPU has both pipeline parallelism and data parallelism when executing, thereby improving the splicing and fusion efficiency of the panorama.

[0094] Embodiment 3

[0095] Figure 3 This is a flow chart of a panoramic image fusion and stitching method provided in Embodiment 3 of the present invention. This embodiment is optimized and improved on the basis of the above technical solutions.

[0096] Furthermore, the step of "determining the optical flow value corresponding to the image pixel point of each camera" is refined into "obtaining adjacent image frames of each camera in adjacent shooting cycles; determining the adjacent pixel points corresponding to the image pixel points of each camera in the corresponding adjacent image frames; and determining the optical flow value corresponding to the image pixel point of each camera based on the image pixel points of each camera and the adjacent pixel points." to improve the method for determining the optical flow value.

[0097] The step of "performing optical flow stratification on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer" is refined into "generating at least one optical flow group according to the optical flow value of each camera; according to each optical flow group, performing optical flow stratification on the filled imaging sphere based on the image pixel points of each camera on the filled imaging sphere to obtain at least one image optical flow layer." to improve the method for determining the image optical flow layer.

[0098] The step of "performing image fusion processing on each image optical flow layer to obtain a fused panoramic image in a panoramic stitching fusion scene" is refined into "determining at least one feature point on each image optical flow layer; performing image fusion processing on each image optical flow layer based on each feature point to obtain a fused panoramic image in a panoramic stitching fusion scene." to improve the generation method of the fused panoramic image. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.

[0099] like Figure 3 As shown, the method comprises the following specific steps:

[0100] S310: Obtain image frames to be fused from each camera in the same shooting cycle in a panoramic stitching and fusion scene.

[0101] S320, according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene, based on the image frames to be fused of each camera, determine the image pixel points of each camera on the blank imaging sphere.

[0102] S330 , using the image pixels of each camera to fill the blank imaging sphere to obtain a filled imaging sphere.

[0103] It should be noted that in the process of calculating the optical flow value using the optical flow algorithm, the calculation of each optical flow point on the imaging sphere is independent of each other, so it is very easy to achieve parallelization, but the efficiency of calculating the optical flow value based on the existing optical flow algorithm is not high. Therefore, the formula of each optical flow calculation point on the imaging sphere can be used to form a linear equation system, and the coefficient matrix can be iteratively solved after LU decomposition (LU Decomposition, triangular decomposition). This method not only has high parallelism, but also can minimize the cumulative error during the iteration process, so that the optical flow value can be determined more accurately.

[0104] S340: Acquire adjacent image frames of each camera in adjacent shooting cycles.

[0105] S350, determining the adjacent pixel points corresponding to the image pixel points of each camera in the corresponding adjacent image frames.

[0106] S360: Determine the optical flow value corresponding to the image pixel of each camera according to the image pixel of each camera and the adjacent pixel points.

[0107] It should be noted that in the process of determining the optical flow value, there is at least one assumption: the brightness pixel value is constant, and the pixel brightness of the object in the image does not change between consecutive frames; the time between adjacent frames of short-distance motion is short enough, and the object movement is small; spatial consistency, adjacent pixels have similar motion.

[0108] For example, assuming that I(x, y, t) is the pixel value of the image pixel (x, y) at time t, and the image pixel moves by Δx, Δy, and Δt between two image frames, it can be concluded that the pixel values ​​between adjacent frames are the same. That is, the pixel values ​​of all image pixels on the image at time T and the corresponding positions of all image pixels on the image at time T+1 are the same. Therefore, the image pixels of each camera have the following relationship with the corresponding adjacent pixels on the corresponding adjacent image frames:

[0109] I(x,y,t)=I(x+Δx,y+Δy,t+Δt);

[0110] Perform a first-order Taylor expansion of I(x+Δx,y+Δy,t+Δt) at time t:

[0111]

[0112] The constraint formula is:

[0113]

[0114] in, is the partial derivative of the image frame in the x direction; is the partial derivative of the image frame in the y direction; is the partial derivative with respect to time between adjacent frames.

[0115] According to the spatial consistency assumption, for multiple image pixels in space:

[0116]

[0117]

[0118]

[0119] Among them, u and v are the optical flow values ​​to be solved.

[0120]

[0121] Among them, Ax=b may be an incompatible linear equation system. Combined with the LU decomposition method, solve the Chebyshev solution of x. Get each optical flow point They are independent of each other, so they can be solved in parallel using GPU core threads.

[0122] S370: Generate at least one optical flow group according to the optical flow value of each camera.

[0123] For example, the maximum optical flow value and the minimum optical flow value may be selected from each optical flow value, and layered from the maximum optical flow value to the minimum optical flow value to obtain at least one optical flow group, wherein the number of optical flow groups may be preset by relevant technical personnel.

[0124] For example, if the optical flow values ​​are a1, a2, a3, a4, a5, a6, a7, a8 and a9 in ascending order, if the number of preset optical flow groups is 3, then optical flow group 1 includes a1, a2 and a3; optical flow group 2 includes a4, a5 and a6; and optical flow group 3 includes a7, a8 and a9.

[0125] S380 , performing optical flow layering on the filled imaging sphere according to each optical flow group and based on the image pixel points of each camera on the filled imaging sphere to obtain at least one image optical flow layer.

[0126] For example, it is possible to determine which optical flow group the optical flow values ​​corresponding to the image pixels of each camera on the filled imaging sphere are located in, so as to merge the image pixels in the optical flow group into the same image optical flow layer. Accordingly, the number of image optical flow layers is the same as the number of optical flow groups.

[0127] S390: Determine at least one feature point on each image optical flow layer.

[0128] Exemplarily, at least one feature point on the optical flow layer of the corresponding image may be determined based on the image pixel points and optical flow values ​​on the optical flow layer of the corresponding image according to the SIFT algorithm, thereby obtaining at least one feature point on the optical flow layer of each image.

[0129] S3100, each image optical flow layer performs image fusion processing based on each feature point to obtain a fused panoramic image in a panoramic stitching and fusion scene.

[0130] Exemplarily, image fusion processing is performed based on the same feature points on the optical flow layer of each image, different feature points are eliminated, and stitching and fusion are performed based on the same feature points to obtain a fused panoramic image in the scene stitching and fusion scenario.

[0131] The technical solution of this embodiment obtains adjacent image frames of each camera in adjacent shooting cycles, determines the adjacent pixel points corresponding to the image pixel points of each camera on the corresponding adjacent image frames, and determines the optical flow value corresponding to the image pixel points of each camera according to the image pixel points and adjacent pixel points of each camera, thereby realizing the decomposition, iteration and parallel solution in the process of solving the optical flow value, improving the efficiency of the optical flow value calculation process, and at the same time, the cumulative error in the iterative calculation process is minimized, so that the accuracy of the solved optical flow value is higher. At least one optical flow group is generated according to the optical flow value of each camera, and according to each optical flow group, the filled imaging sphere is optically layered based on the image pixel points of each camera on the filled imaging sphere to obtain at least one image optical flow layer, and at least one feature point on each image optical flow layer is determined. Each image optical flow layer performs image fusion processing based on each feature point to obtain a fused panorama in a panoramic stitching and fusion scene, further improving the stitching and fusion accuracy and fusion efficiency of the panorama.

[0132] Embodiment 4

[0133] Figure 4 The present invention provides a panoramic image fusion and stitching device according to the fourth embodiment of the present invention. The panoramic image fusion and stitching device provided in the embodiment of the present invention can be applied to the situation of stitching and fusing panoramic images in large parallax or small parallax scenes. The panoramic image fusion and stitching device can be implemented in the form of hardware and / or software, such as Figure 3 As shown, the device specifically includes: an image frame acquisition module 401, a pixel point determination module 402, a filling sphere determination module 403, an optical flow value determination module 404, an optical flow layer generation module 405 and a panoramic image generation module 406.

[0134] The image frame acquisition module 401 is used to acquire the image frames to be fused from each camera in the same shooting cycle in the panoramic stitching and fusion scene;

[0135] A pixel point determination module 402 is used to determine the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created;

[0136] A filling spherical surface determination module 403 is used to fill the blank imaging spherical surface with image pixels of each camera to obtain a filled imaging spherical surface;

[0137] An optical flow value determination module 404 is used to determine the optical flow value corresponding to the image pixel point of each camera;

[0138] An optical flow layer generation module 405 is used to perform optical flow layering on the filled imaging sphere according to the optical flow values ​​and image pixels of each camera to obtain at least one image optical flow layer;

[0139] The panoramic image generation module 406 is used to perform image fusion processing on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene.

[0140] The technical solution of the embodiment of the present invention obtains the image frames to be fused of each camera in the same shooting cycle in a panoramic stitching and fusion scene, determines the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the created panoramic stitching and fusion scene, fills the blank imaging sphere with the image pixel points of each camera to obtain a filled imaging sphere, determines the optical flow value corresponding to the image pixel points of each camera, performs optical flow stratification on the filled imaging sphere according to the optical flow value and image pixel points of each camera to obtain at least one image optical flow layer, performs image fusion processing on each image optical flow layer to obtain a fused panorama in the panoramic stitching and fusion scene, realizes real-time stitching of the panorama in the panoramic stitching and fusion scene, and improves the stitching and fusion accuracy and fusion efficiency of the panorama.

[0141] Optionally, the pixel point determination module 402 includes:

[0142] A core thread creation unit, used to create at least one core thread;

[0143] A world coordinate point determination unit, used for determining the world coordinate point of each camera on the blank imaging sphere;

[0144] The pixel point determination unit is used to determine the image pixel points of each camera on the blank imaging sphere under each of the core threads according to the world coordinate points of each camera and the image frames to be fused, based on the camera imaging models and camera parameter information corresponding to each camera.

[0145] Optionally, the core thread is created as follows:

[0146] Determining the pixel points to be fused of each camera on the blank imaging sphere according to the spherical angle resolution of the blank imaging sphere;

[0147] Create a number of core threads that is equal to the number of pixels to be fused.

[0148] Optionally, the world coordinate point determination unit includes:

[0149] An angle determination subunit, used for determining the polar angle and azimuth angle of each camera on the blank imaging sphere according to the thread identification number of each core thread and the spherical angle resolution;

[0150] The world coordinate point determination subunit is used to determine the world coordinate point of each camera on the blank imaging sphere according to the polar angle and the azimuth angle.

[0151] Optionally, the optical flow value determination module 404 includes:

[0152] An adjacent image frame acquisition unit, used to acquire adjacent image frames of each camera in adjacent shooting cycles;

[0153] An adjacent pixel point determination unit, used to determine the adjacent pixel points corresponding to the image pixel points of each camera on the corresponding adjacent image frames;

[0154] The optical flow value determining unit is used to determine the optical flow value corresponding to the image pixel point of each camera according to the image pixel point and adjacent pixel points of each camera.

[0155] Optionally, the optical flow layer generation module 405 includes:

[0156] An optical flow group generating unit, configured to generate at least one optical flow group according to the optical flow values ​​of each of the cameras;

[0157] The optical flow layer generating unit is used to perform optical flow layering on the filled imaging sphere according to each of the optical flow groups and based on the image pixel points of each of the cameras on the filled imaging sphere to obtain at least one image optical flow layer.

[0158] Optionally, the panoramic image generation module 406 includes:

[0159] A feature point determination unit, used to determine at least one feature point on each of the image optical flow layers;

[0160] The panoramic image generation unit is used for performing image fusion processing on each of the image optical flow layers based on each of the feature points to obtain a fused panoramic image in the panoramic stitching and fusion scene.

[0161] The panoramic image fusion and stitching device provided in the embodiment of the present invention can execute the panoramic image fusion and stitching method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0162] Embodiment 5

[0163] Figure 5A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0164] like Figure 5 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0165] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0166] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a panoramic image fusion stitching method.

[0167] In some embodiments, the panoramic image fusion stitching method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the panoramic image fusion stitching method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the panoramic image fusion stitching method in any other appropriate manner (e.g., by means of firmware).

[0168] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0169] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0170] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0172] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0173] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0174] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0175] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A panoramic image fusion and stitching method, characterized in that: include: Obtain the image frames to be fused from each camera in the same shooting cycle in the panoramic stitching and fusion scene; According to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created, based on the image frames to be fused of each camera, determine the image pixel points of each camera on the blank imaging sphere; Filling the blank imaging sphere with image pixels of each camera to obtain a filled imaging sphere; Determine the optical flow value corresponding to the image pixel point of each camera; According to the optical flow values ​​and image pixels of each camera, optical flow layering is performed on the filled imaging sphere to obtain at least one image optical flow layer; Image fusion processing is performed on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene.

2. The method according to claim 1, characterized in that The step of determining the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created, comprises: Create at least one core thread; Determine the world coordinate point of each camera on the blank imaging sphere; Under each of the core threads, according to the world coordinate points of each of the cameras and the image frames to be fused, based on the camera imaging models and camera parameter information respectively corresponding to each of the cameras, the image pixel points of each of the cameras on the blank imaging sphere are determined.

3. The method according to claim 2, characterized in that The core thread is created as follows: Determining the pixel points to be fused of each camera on the blank imaging sphere according to the spherical angle resolution of the blank imaging sphere; Create a number of core threads that is equal to the number of pixels to be fused.

4. The method according to claim 2, characterized in that: Determining the world coordinate point of each camera on the blank imaging sphere includes: Determining the polar angle and azimuth angle of each camera on the blank imaging sphere according to the thread identification number of each core thread and the spherical angle resolution; The world coordinate point of each camera on the blank imaging sphere is determined according to the polar angle and the azimuth angle.

5. The method according to claim 1, characterized in that The determining of the optical flow value corresponding to the image pixel point of each camera includes: Acquire adjacent image frames of each camera in adjacent shooting cycles; Determine adjacent pixel points corresponding to the image pixel points of each camera on corresponding adjacent image frames; According to the image pixels and adjacent pixels of each camera, the optical flow value corresponding to the image pixels of each camera is determined.

6. The method according to claim 1, characterized in that The step of performing optical flow layering on the filled imaging sphere according to the optical flow values ​​and image pixels of each camera to obtain at least one image optical flow layer comprises: generating at least one optical flow group according to the optical flow values ​​of each of the cameras; According to each of the optical flow groups, based on the image pixel points of each of the cameras on the filled imaging sphere, the filled imaging sphere is subjected to optical flow layering to obtain at least one image optical flow layer.

7. The method according to claim 1, characterized in that The performing image fusion processing on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene includes: Determine at least one feature point on each of the image optical flow layers; Each of the image optical flow layers performs image fusion processing based on each of the feature points to obtain a fused panoramic image in the panoramic stitching and fusion scene.

8. A panoramic image fusion and stitching device, characterized in that: include: An image frame acquisition module is used to acquire image frames to be fused from each camera in the same shooting cycle in a panoramic stitching and fusion scene; A pixel point determination module, used for determining the image pixel points of each camera on the blank imaging sphere based on the image frames to be fused of each camera according to the spherical angle resolution of the blank imaging sphere of the panoramic stitching and fusion scene created; A filling spherical surface determination module, used to fill the blank imaging spherical surface with image pixels of each camera to obtain a filled imaging spherical surface; An optical flow value determination module, used to determine the optical flow value corresponding to the image pixel point of each camera; An optical flow layer generation module, used to perform optical flow layering on the filled imaging sphere according to the optical flow values ​​and image pixels of each camera to obtain at least one image optical flow layer; The panoramic image generation module is used to perform image fusion processing on each of the image optical flow layers to obtain a fused panoramic image in the panoramic stitching and fusion scene.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the panoramic image fusion and stitching method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the panoramic image fusion and stitching method according to any one of claims 1 to 7 when executed.

Images