Panoramic stitching methods and apparatus, electronic devices and storage media

Abstract

This disclosure provides a panoramic stitching method and apparatus, electronic device, and storage medium. The panoramic stitching method includes acquiring images captured by multiple cameras in the same scene; acquiring a remapping lookup table and a fusion lookup table generated based on a stitching model, the stitching model including a hemisphere and a cylinder, the origin of a virtual camera coordinate system associated with the multiple cameras being located at the center of the base surface where the hemisphere and the cylinder meet; and sequentially mapping and fusing the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image. This disclosure effectively reduces camera parallax and stitching alignment errors, thereby significantly improving panoramic stitching quality. It is applicable to open horizontal scenes and exhibits better visual stitching effects. Furthermore, it requires less computation, eliminating the need for real-time detection and updating of the lookup table, thus improving the efficiency and practicality of panoramic stitching.

Description

Technical Field

[0001] This disclosure belongs to the field of panoramic stitching technology, specifically relating to panoramic stitching methods and apparatus, electronic devices and storage media. Background Technology

[0002] Panoramic stitching technology aims to stitch together images or videos captured by a panoramic camera in the same scene, which have overlapping areas, into a panoramic image or video that meets requirements while also providing a wide field of view and high resolution. Choosing a suitable stitching model is crucial for obtaining a high-quality panoramic view. If the stitching distance set in the stitching model is inconsistent with the actual distance of the scene relative to the camera, alignment errors will inevitably occur in the stitched image, manifesting as misalignment, seams, and ghosting, thus affecting the visual quality of the panoramic stitching.

[0003] Currently, most panoramic stitching technologies focus on the 3D rotational relationship between cameras, neglecting their 3D displacement, thus introducing parallax issues. Some solutions consider the 3D displacement between cameras and introduce stitching distance parameters to eliminate parallax at specified distances, but still cannot handle parallax across all distance ranges simultaneously. Other solutions use scene texture content and computer vision methods to align images, addressing parallax; however, this method is computationally expensive, ineffective for weakly textured scenes, and particularly challenging in scenes with abrupt depth changes. Furthermore, existing technologies can achieve spatial domain image stitching but not temporal domain video stitching, limiting panoramic stitching effects in dynamic scenes. Moreover, foreground parallax correction in overlapping panoramic camera areas can cause background changes, creating visual jarring for viewers. Summary of the Invention

[0004] The purpose of this disclosure is to provide a panoramic stitching method and apparatus, electronic equipment and storage medium that can reduce camera parallax and stitching alignment errors, and significantly improve the quality of panoramic stitching.

[0005] In a first aspect, this disclosure provides a panoramic stitching method. The panoramic stitching method includes: acquiring images captured by multiple cameras in the same scene; acquiring a remapping lookup table and a fusion lookup table generated based on a stitching model, wherein the stitching model includes a hemisphere and a cylinder, and the origin of a virtual camera coordinate system associated with the multiple cameras is located at the center of the bottom surface where the hemisphere and the cylinder meet; and sequentially mapping and fusing the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image.

[0006] In one implementation of the first aspect, the center of the hemisphere is the origin of the virtual camera coordinate system, the radius of the hemisphere is the farthest stitching distance, the center of the upper base of the cylinder is the origin of the virtual camera coordinate system, the center of the lower base of the cylinder is the origin of the world coordinate system, the radius of the cylinder is the farthest stitching distance, and the height of the cylinder is the ground clearance of the plurality of cameras.

[0007] In one implementation of the first aspect, the process of successively mapping and fusing the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image includes: performing texture mapping on each overlapping region of the captured images based on the remapping lookup table to obtain a first mapped image; performing texture mapping on each non-overlapping region of the captured images based on the remapping lookup table to obtain a second mapped image; performing image fusion on the first mapped image based on the fusion lookup table to obtain a fused image; and stitching the second mapped image and the fused image together to obtain the panoramic stitched output image.

[0008] In one implementation of the first aspect, the panoramic stitching method further includes: generating the remapping lookup table and the fusion lookup table based on the stitching model, including: calibrating the plurality of cameras to determine the intrinsic and extrinsic parameters of the plurality of cameras; configuring a first stitching parameter and a second stitching parameter, wherein the first stitching parameter is an imaging parameter of the output image, and the second stitching parameter is related to the stitching model; constructing the remapping lookup table and the fusion lookup table based on the intrinsic parameters, the extrinsic parameters, the first stitching parameter, and the second stitching parameter; and saving the remapping lookup table and the fusion lookup table.

[0009] In one implementation of the first aspect, the first stitching parameters include at least one of the following: the width of the output image, the height of the output image, the horizontal field of view of the output image, the vertical field of view of the output image, the horizontal offset of the center point of the output image, the vertical offset of the center point of the output image, and the projection method of the output image; and the second stitching parameters include the ground height of the plurality of cameras and the farthest stitching distance.

[0010] In one implementation of the first aspect, constructing the remapping lookup table based on the intrinsic parameters, the extrinsic parameters, the first stitching parameter, and the second stitching parameter includes: traversing the coordinates (x, y) in the output image; and back-projecting the coordinates (x, y) onto the origin O of the virtual camera coordinate system. c On the surface of the sphere with center P, let the projection point be P. c , the rays are

[0011] Based on the first splicing parameters, the direction vector of the ray is calculated as follows: Based on the second splicing parameters, the direction vector is determined. The modulus S; according to the formula Calculate the projection point P c The coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T Based on the extrinsic parameters and the coordinates [P] cx P cy P cz ] T Calculate the projection point P c Coordinates in different physical camera coordinate systems [X] c Y c Z c ] T ; and based on the intrinsic parameters, the coordinates [X] c Y c Z c ] T Convert to pixel coordinates (u,v) on the input image.

[0012] In one implementation of the first aspect, the direction vector of the ray is calculated based on the first splicing parameter. This includes using the following formula:

[0013]

[0014]

[0015] Where dst_center_x represents the horizontal offset of the center point of the output image, dst_center_y represents the vertical offset of the center point of the output image, dst_fov_x represents the horizontal field of view of the output image, dst_fov_y represents the vertical field of view of the output image, dst_width represents the width of the output image, and dst_height represents the height of the output image.

[0016] In one implementation of the first aspect, the direction vector is determined based on the second concatenation parameter. The modulus S includes:

[0017] when When S = R

[0018] when and hour,

[0019] when and hour,

[0020] in Represents the direction vector x-coordinate point, Represents the direction vector The y-coordinate point, Represents the direction vector The z-coordinate point, R represents the farthest stitching distance, and h represents the height of the virtual camera above the ground.

[0021] In one implementation of the first aspect, based on the extrinsic parameters and the coordinates [P] cx P cy P cz ] T Calculate the projection point P c Coordinates in different physical camera coordinate systems [X] c Y c Z c ] T This includes using the following formula:

[0022]

[0023] Where P c Represents the projection point P c Coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T T 0c T represents the extrinsic parameter matrix. 01 T represents the extrinsic parameter of camera 1 relative to camera 0. 02 P0 represents the extrinsic parameters of camera 2 relative to camera 0, and P0 represents the projection point P. c In the camera's 0 coordinate system, P1 represents the coordinates of the projection point P. c In the camera 1 coordinate system, P2 represents the projection point P. c Coordinates in the camera 2 coordinate system.

[0024] In one implementation of the first aspect, based on the intrinsic parameters, the coordinates [X] are... c Y c Z c ] T Converting to pixel coordinates (u,v) on the input image involves the following formula:

[0025]

[0026] r 2 =x' 2 +y' 2 ,

[0027]

[0028] Where k1, k2, k3, k4, k5 and k6 represent radial distortion coefficients, p1 and p2 represent tangential distortion coefficients, fx and fy represent the camera's focal length, and cx and cy represent the offset of the image center point.

[0029] In one implementation of the first aspect, mapping and fusing the captured image sequentially based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image includes: dividing the input captured image into overlapping and non-overlapping regions; for the overlapping regions, performing texture mapping based on the remapping lookup table, and then performing image fusion on the texture-mapped overlapping regions based on the fusion lookup table to obtain processed overlapping regions; for the non-overlapping regions, performing texture mapping based on the remapping lookup table to obtain processed non-overlapping regions; and combining the processed overlapping regions and the processed non-overlapping regions to obtain the panoramic stitched output image.

[0030] In one implementation of the first aspect, for the overlapping region, performing texture mapping based on the remapping lookup table includes: traversing the coordinates (x, y) of the panoramic stitching output image; querying the coordinates (mapx(x, y), mapy(x, y)) of the corresponding input image in the remapping lookup table according to the coordinates (x, y); interpolating the image content of the input image and obtaining the interpolation result at the coordinates (mapx(x, y), mapy(x, y)); and filling the interpolation result into the coordinates (x, y) of the panoramic stitching output image.

[0031] In one implementation of the first aspect, image fusion of the texture-mapped overlapping region based on the fusion lookup table includes: performing image fusion on the texture-mapped image based on any one of the Alpha fusion algorithm, multi-band fusion algorithm, and Poisson fusion algorithm.

[0032] In one implementation of the first aspect, image fusion of overlapping regions after texture mapping based on the Alpha fusion algorithm includes: traversing the coordinates (x, y) of the panoramic stitching output image; querying the corresponding alpha(x, y) in the fusion lookup table according to the coordinates (x, y); obtaining the pixel value Image c(x, y) of the input image; calculating the fused pixel value based on the alpha(x, y) and the pixel value Image c(x, y); and filling the fused pixel value into the coordinates (x, y) of the panoramic stitching output image.

[0033] In one implementation of the first aspect, calculating the fused pixel value based on the alpha(x,y) and the pixel value Image c(x,y) includes using the following formula:

[0034] Blend(x,y)=alpha(x,y)*Image1(x,y)+(1-alpha(x,y))*Image2(x,y)

[0035] Where alpha(x,y) represents the alpha value at coordinate (x,y) in the fusion lookup table, Image1(x,y) represents the pixel value at coordinate (x,y) in the first input image, Image2(x,y) represents the pixel value at coordinate (x,y) in the second input image, and Blend(x,y) represents the fused pixel value.

[0036] In one implementation of the first aspect, calibrating the plurality of cameras to determine the intrinsic and extrinsic parameters of the plurality of cameras includes: acquiring an image of a calibration board for intrinsic parameter calibration as a first calibration image; acquiring an image of a calibration board for extrinsic parameter calibration as a second calibration image; processing the first calibration image and the second calibration image based on a preset calibration algorithm to obtain the intrinsic, extrinsic, and reprojection error of the camera; determining whether the calibration result of the camera meets the standard based on the reprojection error; if so, saving the intrinsic and extrinsic parameters; otherwise, adjusting the calibration board and / or changing the calibration algorithm, and repeating the above calibration steps until the calibration result of the camera meets the standard.

[0037] Secondly, this disclosure provides a panoramic stitching device. The panoramic stitching device includes: an image acquisition module configured to acquire images captured by multiple cameras in the same scene; and an image stitching module configured to: acquire a remapping lookup table and a fusion lookup table generated based on a stitching model, the stitching model including a hemisphere and a cylinder, the origin of a virtual camera coordinate system associated with the multiple cameras being located at the center of the bottom surface where the hemisphere and the cylinder meet; and sequentially map and fuse the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image.

[0038] In one implementation of the second aspect, the panoramic stitching device further includes: a lookup table generation module, configured to generate the remapping lookup table and the fusion lookup table based on the stitching model.

[0039] Thirdly, this disclosure provides an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to cause the electronic device to perform the panoramic stitching method described in any of the preceding claims.

[0040] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the panoramic stitching method described in any of the preceding claims.

[0041] The panoramic stitching method, apparatus, electronic device, and storage medium described in this disclosure can effectively reduce camera parallax and stitching alignment errors, thereby significantly improving the quality of panoramic stitching; it is suitable for open horizontal scenes and exhibits better visual stitching effects; it requires less computation and does not require real-time detection and updating of remapping lookup tables and fusion lookup tables, thus improving the efficiency and practicality of panoramic stitching. Attached Figure Description

[0042] Figure 1a The image shown is a three-dimensional schematic diagram of the spliced ​​model A in one embodiment.

[0043] Figure 1b The image shown is an input image from a dual fisheye camera in one embodiment.

[0044] Figure 1c The image shown is a texture map of a dual fisheye camera in one embodiment.

[0045] Figure 2a The diagram shows a parallax illustration of a dual fisheye camera in one embodiment.

[0046] Figure 2b The image shows a comparison of the seams produced by different splicing distances in one embodiment.

[0047] Figure 3 The diagram shown is a schematic representation of splicing model C in one embodiment.

[0048] Figure 4a The diagram shown is a two-dimensional schematic representation of a splicing model D according to an embodiment of the present disclosure in one embodiment.

[0049] Figure 4b The image shown is a three-dimensional schematic diagram of a splicing model D according to an embodiment of the present disclosure from a first perspective in one embodiment.

[0050] Figure 4c The image shown is a three-dimensional schematic diagram of a splicing model D according to an embodiment of the present disclosure from a second perspective in one embodiment.

[0051] Figure 5a The diagram shows a distribution of multiple cameras according to an embodiment of the present disclosure in one example.

[0052] Figure 5b The diagram shows a distribution of a multi-view camera according to an embodiment of the present disclosure in another embodiment.

[0053] Figure 6The flowchart shown is an embodiment of a panoramic stitching method according to an embodiment of the present disclosure.

[0054] Figure 7 The diagram shows a flowchart of the generation of a remapping lookup table and a fusion lookup table in one embodiment according to an embodiment of the present disclosure.

[0055] Figure 8 The diagram shown is a camera calibration flowchart of a panoramic stitching method according to an embodiment of the present disclosure.

[0056] Figure 9 The diagram shown is a back-projection schematic diagram of a panoramic stitching method according to an embodiment of the present disclosure.

[0057] Figure 10 The diagram shows the transformation relationship between a unit sphere and a spliced ​​model D in one embodiment according to an embodiment of the present disclosure.

[0058] Figure 11 The diagram shows a comparison of the stitching model D before and after adjusting the extrinsic parameter matrix in one embodiment of the panoramic stitching method according to an embodiment of the present disclosure.

[0059] Figure 12a The image shown is an output image of a panoramic stitching method according to an embodiment of the present disclosure.

[0060] Figure 12b The image shown is an output image of the panoramic stitching method according to an embodiment of the present disclosure in another embodiment.

[0061] Figure 13a The diagram shown is an image stitching flowchart of a panoramic stitching method according to an embodiment of the present disclosure.

[0062] Figure 13b The diagram shows the division of overlapping and non-overlapping regions in one embodiment according to an embodiment of the present disclosure.

[0063] Figure 13c The diagram shows a combination of overlapping and non-overlapping regions in one embodiment according to an embodiment of the present disclosure.

[0064] Figure 14 The diagram shown is a texture mapping schematic of a panoramic stitching method according to an embodiment of the present disclosure.

[0065] Figure 15a The image shown is a first-view input image to be stitched in one embodiment of the panoramic stitching method according to an embodiment of the present disclosure.

[0066] Figure 15b The image shown is a second-view input image to be stitched in one embodiment of the panoramic stitching method according to an embodiment of the present disclosure.

[0067] Figure 15c The image shown is a panoramic stitching output image in one embodiment of the panoramic stitching method according to an embodiment of the present disclosure.

[0068] Figure 16 The diagram shown is a structural schematic of a panoramic stitching device according to an embodiment of the present disclosure.

[0069] Figure 17 The diagram shown is a structural schematic of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0070] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0071] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0072] Furthermore, the use of terms such as "first" and "second" in this disclosure is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this disclosure.

[0073] The basic principle of panoramic stitching technology is as follows: First, a virtual unit sphere is constructed using a panoramic camera as the center point. Next, images captured by each camera lens are projected onto this virtual sphere according to a predetermined stitching model, and fusion processing is performed on the overlapping areas of the images to generate a complete 3D spherical image. Finally, by using different projection methods, the coordinate points on the spherical image are projected from three-dimensional space to a two-dimensional plane, thus converting it into a planar image and completing the generation of the panoramic image. This process is conceptually similar to the process of unfolding the three-dimensional Earth surface into a two-dimensional world map. The fusion of overlapping areas can include pyramid-based multi-band fusion algorithms, alpha fusion algorithms, etc.

[0074] The basic process of stitching can include offline calibration (obtaining intrinsic and extrinsic parameters between cameras, including rotation and displacement), setting stitching parameters (stitching distance, projection method, output field of view, global Euler angles), performing texture mapping on each camera according to the stitching parameters, and using a certain fusion method to fuse the texture mapping maps of multiple cameras into a single stitched image.

[0075] Even with accurate offline calibration, if the set stitching distance differs from the actual distance between the scene and the camera, misalignment of the stitched images will occur, resulting in noticeable misalignment, seams, or ghosting. The root cause lies in the discrepancy between the assumed stitching model and reality.

[0076] Several splicing models are introduced below.

[0077] like Figure 1a As shown, stitching model A is designed based on an idealized spherical model at infinity, where the radius of the sphere is set to an extremely large value. This model ignores the displacement between multiple cameras, assuming that the optical centers of the multiple cameras coincide, or that the scene being captured is infinitely far from the camera.

[0078] Figure 1b The image shown is an input image captured by a panoramic camera system consisting of two fisheye cameras mounted back-to-back. This panoramic camera system, including camera 0 and camera 1, is capable of capturing a wide field of view, with each fisheye lens offering a field of view of up to 200 degrees, thus achieving comprehensive coverage of the surrounding environment.

[0079] Figure 1c This diagram illustrates the projection of the acquired input image onto a virtual sphere.

[0080] Building upon stitching model A, model B considers the displacement between multiple cameras and introduces the concept of "stitching distance," which sets a distance for the spherical model that aligns with the shooting scene. A suitable stitching distance can eliminate image misalignment caused by parallax.

[0081] Parallax refers to the change or difference in the position of an object in the field of view when observed from two different positions. Because it is almost impossible to perfectly align the optical centers of multiple cameras during assembly, there will be a certain gap between them, resulting in parallax. The closer the target object is to the camera, the greater the parallax.

[0082] like Figure 2a As shown, O1 and O2 are the camera centers, and Ptrue and Pfalse are the 3D coordinates of the object's real position (with the correct stitching distance) and the virtual 3D coordinates obtained by the incorrect stitching distance, respectively, with camera O1 as the reference coordinate system. The intersection points p1 and p2* of the line connecting Ptrue (obtained with the correct stitching distance) to O1 and O2 respectively with the image plane are matched (image content is consistent). The intersection point p2 of the line connecting Pfalse (obtained with the incorrect stitching distance) to O2 with the image plane is p2, indicating a pixel position shift. The image content at point p2 is inconsistent with that at point p1. The greater the difference between the real-world point coordinates Ptrue and the virtual coordinates Pfalse obtained by the incorrect stitching distance, the greater the offset, which manifests as ghosting or seams in the stitched image. Note that the 3D coordinates of Ptrue and Pfalse are defined in the coordinate system of camera O1, and Ptrue and Pfalse are on the same straight line as O1. Therefore, Ptrue and Pfalse are oriented in the same direction as camera O1. The only difference is their distance from O1. However, when the 3D point is projected onto the 2D plane, the distance information is lost. Therefore, Ptrue and Pfalse are both projected onto point p1.

[0083] Therefore, for stitching model B, the suitability of the chosen stitching distance and the image scene has a significant impact on the stitching effect. However, this model can only set a single stitching distance, meaning the entire sphere maintains the same stitching distance. But in the real world, objects are at varying distances from the camera, which will likely lead to significant differences in the distances of Ptrue and Pfalse in certain areas of the image, resulting in noticeable stitching seams.

[0084] Figure 2b The image shows the effect of different stitching distances, specifically 5m, 10m, and 40m. At each stitching distance, the left image shows the overlapping portion of the two cameras in the entire stitched image (already fused), referred to as the overlap band, while the right image is a magnified view of a localized area of ​​the scene with prominent features. It can be seen that as the stitching distance increases, the stitching seams gradually appear on the nearby "warning sign" (the seams exhibit a "transparent" appearance), while the stitching seams on the distant "trees" gradually disappear (the seams exhibit a double image appearance).

[0085] By specifying an optimal stitching distance, the stitching effect for targets at different distances can be adjusted. A smaller optimal stitching distance results in better stitching of nearby scenes; a larger optimal stitching distance results in better stitching of distant scenes. Therefore, the optimal stitching distance can be set according to the scene requirements: if most objects in the scene are relatively far away, a larger optimal stitching distance should be set; if most objects in the scene are relatively close, a smaller optimal stitching distance should be set. However, for open horizontal scenes such as outdoor plazas and indoor / outdoor sports fields, stitching model B has shortcomings because the distance of objects on the ground relative to the camera changes continuously from near to far.

[0086] Based on the existing splicing model B, this disclosure constructs new splicing models C and D. Figure 3 A schematic diagram of the splicing model C in one embodiment is shown.

[0087] like Figure 3 As shown, the stitching model C includes a hemisphere, wherein the center of the hemisphere is the origin Oc of the virtual camera coordinate system, and the radius of the hemisphere is the farthest stitching distance R.

[0088] This disclosure transforms the traditional single-segment spherical structure into a composite structure of a hemisphere and a horizontal plane (or ground plane). The portion above the horizontal plane remains part of a sphere, just like in stitching model B. However, stitching model C is also impractical. Specifically, because the camera center O is located on the horizontal plane, objects on the horizontal plane are rendered in a degraded manner when imaged on the camera. This phenomenon can be illustrated with a visual example: when a person lies on the ground observing their surroundings, even if the height of an object is zero (i.e., the object is flush with the ground), distant objects will be obscured by nearby objects, preventing the observer from seeing the overall view of the ground.

[0089] Figure 4a A schematic diagram of the splicing model D according to an embodiment of the present disclosure is shown. Figure 4b and Figure 4c A three-dimensional schematic diagram of the spliced ​​model D is shown, viewed from different perspectives.

[0090] like Figures 4a to 4cAs shown, the stitching model D provided in this embodiment includes a hemisphere and a cylinder. The center of the hemisphere is the origin Oc of the virtual camera coordinate system, and the radius of the hemisphere is the farthest stitching distance R. The center of the upper surface of the cylinder is the origin Oc of the virtual camera coordinate system, the center of the lower surface of the cylinder is the origin Ow of the world coordinate system, the radius of the cylinder is the farthest stitching distance R, and the height of the cylinder is the ground clearance h of the multiple cameras. The center of the virtual camera (i.e., the origin Oc of the virtual camera coordinate system) is located at the interface between the hemisphere and the cylinder (i.e., the upper surface of the cylinder), and the center of the ground (i.e., the origin Ow of the world coordinate system) is located at the lower surface of the cylinder.

[0091] Compared to stitching models A, B, and C, stitching model D has separate origins for its virtual camera coordinate system and world coordinate system. Stitching model D of this application is suitable for open horizontal scenes such as outdoor plazas and indoor / outdoor sports fields, and exhibits superior stitching results.

[0092] Using the virtual camera coordinate system as a reference, direction vectors for light rays in all directions are derived from the origin Oc. In the stitching model D, there are three types of direction vectors: upward direction vectors, similar to the sky in a real scene, denoted as the sky vector (Psky or Point); downward direction vectors towards nearby objects, similar to the ground in a real scene, denoted as the ground vector (Pground or Point); and downward direction vectors towards distant objects, similar to distant walls, buildings, or the horizon line, denoted as the wall vector (Pwall or Point). The virtual camera's height above the ground is h, which is the distance between Oc and Ow, and needs to be measured based on the actual camera installation. The furthest stitching distance R is the distance between the camera and a distant wall or building, which needs to be set by the user according to the actual application scenario. If there are no obstructions in front of the camera, R can be set to a very large distance to approximate infinity.

[0093] The stitching model D can support scenes with multiple cameras arranged horizontally, and allows the cameras to tilt downwards at a certain angle. For example... Figure 5a As shown, the multi-view cameras are arranged horizontally, without tilting downwards. Figure 5b As shown, the multi-view cameras are arranged horizontally and tilted downwards at a certain angle.

[0094] The following embodiments of this disclosure provide a panoramic stitching method and apparatus, electronic device, and storage medium, which can effectively reduce parallax and alignment errors, and significantly improve the visual effect of panoramic stitching. The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0095] like Figure 6 As shown, this embodiment provides a panoramic stitching method, including the following steps S100 to S300.

[0096] In step S100, images captured by multiple cameras in the same scene are acquired.

[0097] In some embodiments, a panoramic camera system consisting of several fisheye cameras is used to acquire images. These cameras have a sufficiently wide field of view, enabling 360-degree omnidirectional coverage when combined. The cameras are precisely positioned in different directions, with overlapping areas between their fields of view. Furthermore, the panoramic camera system allows for precise synchronization control to ensure that the fields of view captured at different times correspond accurately.

[0098] It should be noted that this disclosure can also use a panoramic camera system to capture video streams. In this case, the video stream needs to be processed frame by frame.

[0099] In step S200, a remapping lookup table and a fusion lookup table generated based on the stitching model are obtained. The stitching model includes a hemisphere and a cylinder, and the origin of the virtual camera coordinate system associated with the multiple cameras is located at the center of the bottom surface where the hemisphere and the cylinder meet.

[0100] In some embodiments, regarding the stitching model, the center of the hemisphere is the origin of the virtual camera coordinate system, the radius of the hemisphere is the farthest stitching distance, the center of the upper base of the cylinder is the origin of the virtual camera coordinate system, the center of the lower base of the cylinder is the origin of the world coordinate system, the radius of the cylinder is the farthest stitching distance, and the height of the cylinder is the ground clearance of the plurality of cameras.

[0101] In some embodiments, a remapping lookup table and a fusion lookup table can be generated first based on the splicing model, and then the remapping lookup table and the fusion lookup table can be received.

[0102] In other embodiments, a remapping lookup table and a fusion lookup table generated based on a splicing model can be received from an external source.

[0103] In step S300, the captured images are successively mapped and fused based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image.

[0104] In some embodiments, texture mapping is performed on each overlapping region of the captured image based on the remapping lookup table to obtain a first mapped image, and texture mapping is performed on each non-overlapping region of the captured image based on the remapping lookup table to obtain a second mapped image. Additionally, image fusion is performed on the first mapped image based on the fusion lookup table to obtain a fused image. Furthermore, the second mapped image and the fused image are stitched together to obtain the panoramic stitched output image.

[0105] According to embodiments of this disclosure, by utilizing the stitching model and the corresponding remapping lookup table and fusion lookup table, the technical solution of this invention can effectively reduce camera parallax and stitching alignment errors, thereby significantly improving the quality of panoramic stitching. Furthermore, the technical solution of this invention is applicable to open horizontal scenes and exhibits superior visual stitching effects. In addition, the technical solution of this invention has low computational requirements, eliminating the need for real-time detection and updating of the remapping lookup table and fusion lookup table, thus improving the efficiency and practicality of panoramic stitching.

[0106] In one embodiment of this disclosure, the panoramic stitching method further includes step S400: generating the remapping lookup table and the fusion lookup table based on the stitching model.

[0107] like Figure 7 As shown, step S400 may include: calibrating the plurality of cameras to determine the intrinsic and extrinsic parameters of the plurality of cameras; configuring a first stitching parameter and a second stitching parameter, wherein the first stitching parameter is an imaging parameter of the output image, and the second stitching parameter is related to the stitching model; constructing the remapping lookup table and the fusion lookup table based on the intrinsic parameters, the extrinsic parameters, the first stitching parameter and the second stitching parameter; and saving the remapping lookup table and the fusion lookup table.

[0108] Specifically, such as Figure 8 As shown, calibrating the plurality of cameras to determine their intrinsic and extrinsic parameters includes: acquiring an image of a calibration board used for intrinsic parameter calibration as a first calibration image; acquiring an image of a calibration board used for extrinsic parameter calibration as a second calibration image; processing the first and second calibration images based on a preset calibration algorithm to obtain the camera's intrinsic, extrinsic, and reprojection error; determining whether the camera's calibration result meets the standard based on the reprojection error; if so, saving the intrinsic and extrinsic parameters; otherwise, adjusting the calibration board and / or changing the calibration algorithm, and repeating the above calibration steps until the camera's calibration result meets the standard.

[0109] In this embodiment of the disclosure, the calibration board can be any one of the following: a common checkerboard calibration board, a checkerboard calibration board with QR code markings, or a random noise calibration board. For the checkerboard calibration board, the feature points are the corner points where the black and white checkerboard meet. For the random noise calibration board, the feature points are obtained through a feature point extraction algorithm (such as ORB, SIFT, SURF).

[0110] The calibration algorithm used can be the Direct Linear Transform (DLT) method, the Nonlinear Least Squares method, etc., to obtain accurate intrinsic and extrinsic parameters of the multi-camera system, minimizing the reprojection error of paired matching feature points on the calibration board. Generally, an average reprojection error of less than 1 pixel is considered sufficient to determine the accuracy of the calibration result.

[0111] The intrinsic parameters include an intrinsic parameter matrix. The distortion coefficients are defined as follows: fx and fy are the camera's focal lengths, and cx and cy are the offsets of the image center point. The distortion coefficients vary depending on the camera model. For example, the pinhole camera model has six radial distortion coefficients k1, k2, k3, k4, k5, and k6, and two tangential distortion coefficients p1 and p2. The fisheye camera model has four radial distortion coefficients k1, k2, k3, and k4. The omnidirectional camera model has three radial distortion coefficients k1, k2, and k3, two tangential distortion coefficients p1 and p2, and one mirror parameter, Cauchy ξ.

[0112] The extrinsic parameters include an extrinsic parameter matrix. The 3×3 matrix in the upper left corner is the rotation matrix, and the 3×1 column vector in the upper right corner is the displacement vector.

[0113] In this embodiment, the calibration of camera intrinsic and extrinsic parameters is performed offline. After calibration, the obtained camera intrinsic and extrinsic parameters are written to a storage device in text or binary data form.

[0114] It should be noted that each device requires calibration of its internal and external camera parameters. If the camera mounting structure is stable and robust, recalibration is not necessary during later use. If the camera mounting structure is unstable, flimsy, has been subjected to impact, or the offline calibration results are unsatisfactory, online calibration can be performed during later use; this part of the process also falls under online processing.

[0115] This disclosure does not limit the specific method of online calibration, nor does it limit the specific method of offline calibration, nor does it limit the camera model used. It only requires providing intrinsic parameters that can accurately describe the imaging process of the camera and extrinsic parameters that describe the relative positions of multiple cameras.

[0116] In one embodiment of this disclosure, the first stitching parameters include at least one of the following: the width (dst_width, in pixels), the height (dst_height, in pixels), the horizontal field of view (dst_fov_x, in degrees), the vertical field of view (dst_fov_y, in degrees), the horizontal offset of the center point (dst_center_x, in pixels), the vertical offset of the center point (dst_center_y, in pixels), and the projection mode (project_mode) of the output image.

[0117] It should be noted that the first splicing parameter can be debugged online. Once it is adjusted to a suitable value in a certain application scenario, it will theoretically no longer need to be changed or adjusted frequently.

[0118] In one embodiment of this disclosure, the second stitching parameters include the ground clearance h of the plurality of cameras and the farthest stitching distance R.

[0119] Taking project_mode as an isometric rectangular projection as an example, the remap lookup table (remap LUT) is constructed based on the intrinsic parameters, the extrinsic parameters, the first stitching parameter, and the second stitching parameter, including: traversing the coordinates (x, y) in the output image; and back-projecting the coordinates (x, y) onto the origin O of the virtual camera coordinate system. c On the surface of the sphere with center P, let the projection point be P. c , the rays are Based on the first splicing parameters, the direction vector of the ray is calculated as follows: Based on the second splicing parameters, the direction vector is determined. The modulus S; according to the formula Calculate the projection point P c The coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T Based on the extrinsic parameters and the coordinates [P] cx P cy P cz ] T Calculate the projection point P c Coordinates in different physical camera coordinate systems [X] c Y c Z c ] T Based on the intrinsic parameters, the coordinates [X] are... c Y c Z c ] T Convert to pixel coordinates (u,v) on the input image.

[0120] like Figure 9 As shown, the back projection process is equivalent to transforming from the plane coordinate system where the output image is located to the unit sphere.

[0121] In this embodiment of the disclosure, the direction vector of the ray is calculated based on the first splicing parameter. This includes using the following formula:

[0122]

[0123] Where dst_center_x represents the horizontal offset of the center point of the output image, dst_center_y represents the vertical offset of the center point of the output image, dst_fov_x represents the horizontal field of view of the output image, dst_fov_y represents the vertical field of view of the output image, dst_width represents the width of the output image, and dst_height represents the height of the output image.

[0124] The sky vector, wall vector, and ground vector each correspond to different vector magnitudes S. In this embodiment, the direction vector is determined based on the second splicing parameter. The modulus S includes:

[0125] when When S = R

[0126] when and hour,

[0127] when and hour,

[0128] in Represents the direction vector x-coordinate point, Represents the direction vector The y-coordinate point, Represents the direction vector The z-coordinate point, R represents the farthest stitching distance, and h represents the height of the virtual camera above the ground.

[0129] It should be noted that the second stitching parameter (i.e., the virtual camera's height above the ground h and the maximum stitching distance R) is a parameter that frequently needs to be modified during the online processing phase and needs to be changed according to scene variations. If the configured h and R do not match the actual scene conditions, obvious seams will be visible in the overlapping area of ​​the stitched output image, appearing as cracks or transparency in the object. For example, if the actual height of the camera above the ground is 3 meters, and h is configured to 4 meters, then according to the formula... It can be seen that objects on the ground will show obvious seams in the overlapping area. If the camera is placed indoors, and the overlapping area of ​​the camera is exactly facing a wall, with the actual distance between the camera and the wall being 20 meters and R set to 15 meters, then according to the formula... It can be seen that the walls and objects on the walls will have obvious seams in the overlapping areas.

[0130] In this embodiment, the direction vector The module length S is equivalent to the stitching distance in stitching model B. The innovation of the stitching model D used in this disclosure lies in its ability to calculate a stitching distance that conforms to the actual situation of an open horizontal scene, thereby improving the stitching effect.

[0131] In this embodiment of the disclosure, the direction vector of the ray is obtained. and the direction vector After determining the modulus S, it can be calculated using the formula. Calculate the projection point P c The coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T This process is equivalent to transforming from a unit sphere to a composite model D consisting of hemispheres and cylinders, as shown below. Figure 10 As shown.

[0132] When P is obtained c Coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T Next, it needs to be transformed to the coordinate system of a real physical camera. Taking three cameras arranged horizontally and tilted downwards at a certain angle as an example, let the origin of the coordinate system of camera 0 be O0, the origin of the coordinate system of camera 1 be O1, and the origin of the coordinate system of camera 2 be O2. We can assume that the origin O0 of the coordinate system of camera 0 is the same as the origin O of the virtual camera coordinate system. c If they overlap and their directions are also the same, then the extrinsic parameter is the identity matrix, i.e. If the camera tilts downwards at a certain angle, such as 30 degrees, then the 3x3 part in the upper left corner of the extrinsic parameter matrix needs to be adjusted, i.e., the rotation matrix part. Unless there are special requirements, it can still be assumed that the origin O0 of the camera 0 coordinate system coincides with the origin O of the virtual camera coordinate system. c This means the displacement vector is 0, and the extrinsic parameter matrix contains only the rotation matrix. This can be achieved by adjusting T. 0c This ensures the final stitched image appears horizontal, such as... Figure 11 As shown. Since the goal is to present the content of the stitched image horizontally, the virtual camera coordinate system is actually adjusted in reverse.

[0133] In this embodiment of the disclosure, based on the extrinsic parameters and the coordinates [P] cx P cy P cz ] T Calculate the projection point P c Coordinates in different physical camera coordinate systems [X] c Yc Z c ] T This includes using the following formula:

[0134]

[0135] Where P c Represents the projection point P c Coordinates [P] in the virtual camera coordinate system cx P cy P cz ] T T 0c T represents the extrinsic parameter matrix. 01 T represents the extrinsic parameter of camera 1 relative to camera 0. 02 P0 represents the extrinsic parameters of camera 2 relative to camera 0, and P0 represents the projection point P. c In the camera's 0 coordinate system, P1 represents the coordinates of the projection point P. c In the camera 1 coordinate system, P2 represents the projection point P. c Coordinates in the camera 2 coordinate system.

[0136] It should be noted that the T used in the embodiments of this disclosure 01 and T 02 It can be obtained through calibration during the offline phase.

[0137] Taking the pinhole imaging model as an example, based on the intrinsic parameters, the coordinates [X] are... c Y c Z c ] T Converting to pixel coordinates (u,v) on the input image involves the following formula:

[0138]

[0139] r 2 =x' 2 +y' 2 ,

[0140]

[0141] Where k1, k2, k3, k4, k5 and k6 represent radial distortion coefficients, p1 and p2 represent tangential distortion coefficients, fx and fy represent the camera's focal length, and cx and cy represent the offset of the image center point.

[0142] At this point, the remap LUT is fully established. By traversing the (x,y) coordinates in the output image, the mapping of pixel coordinates (u,v) under each physical camera can be obtained.

[0143] In one embodiment of this disclosure, the process of generating the blend lookup table (blend LUT) includes: creating a blank blend lookup table; traversing each coordinate in the blend lookup table; and setting the blend weight value at the coordinate based on the splicing model to obtain the blend lookup table.

[0144] It should be noted that the fusion weight values ​​at different coordinates in the fusion lookup table can be the same or different.

[0145] In this implementation, the generation of the remap LUT depends on the stitching model D, which is directly related to whether the panoramic stitched output image will have obvious stitching flaws and whether the image content will be misaligned.

[0146] This disclosure iterates through the results of camera 0 and camera 1, and plots them as shown in the figure. Figure 12a The output image shown is divided into three parts. The first part represents the valid content of camera 0, the second part represents the valid content of camera 1, and the third part represents the valid content of the overlapping area of ​​camera 0 and camera 1. During the mapping process, if the pixel coordinates (u,v) exceed the resolution range of the input image, it is considered an invalid region and is drawn as the fourth part.

[0147] During the mapping process, the edges of camera 0 or camera 1 exhibit a curved shape. Since processing rectangular block data is more efficient than processing data with curved edges, rectangular blocks are used to divide the output stitched image into non-overlapping areas from different cameras and overlapping areas between the two cameras. This process also removes invalid black areas, such as... Figure 12b As shown, box 1 represents the non-overlapping area of ​​camera 0, and its content is the remap LUT for the non-overlapping area of ​​camera 0; box 2 represents the non-overlapping area of ​​camera 1, and its content is the remap LUT for the non-overlapping area of ​​camera 1; box 3 represents the overlapping area of ​​camera 0 and camera 1, and its content is the remap LUT for the overlapping area of ​​camera 0 and camera 1. Once the remap LUTs of the overlapping areas are obtained, the blend LUTs for Alphablend or Multibandblend blending methods can be obtained. Since the range and boundaries of the overlapping areas used for blending are already known, this will not be elaborated further here.

[0148] In this embodiment, the construction of the remap LUT and blend LUT is performed online. The "remap LUT" and "blend LUT" only need to be executed once to obtain data when the device starts up. If the current camera stitching device remains unchanged or the installation location environment does not change, the "remap LUT" and "blend LUT" can be saved to a storage device for reading and use the next time the device is powered on or the program is started. The fact that the "remap LUT" and "blend LUT" are executed "only once" is the core difference between them and existing dynamic stitching, dynamic seam stitching, and dynamic alignment methods that require updating and calculating the "remap LUT" and / or "blend LUT" for each frame, thus saving computational resources.

[0149] like Figure 13a As shown, in one embodiment of this disclosure, step S300, which involves successively mapping and fusing the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image, includes: performing texture mapping on each overlapping region of the captured images based on the remapping lookup table to obtain a first mapped image; performing texture mapping on each non-overlapping region of the captured images based on the remapping lookup table to obtain a second mapped image; performing image fusion on the first mapped image based on the fusion lookup table to obtain a fused image; and stitching the second mapped image and the fused image together to obtain the panoramic stitched output image.

[0150] In another embodiment of this disclosure, step S300, which involves successively mapping and fusing the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image, includes steps S301 to S304.

[0151] In step S301, the input captured image is divided into overlapping regions and non-overlapping regions.

[0152] In step S302, for the overlapping region, texture mapping is first performed based on the remapping lookup table, and then image fusion is performed on the texture-mapped overlapping region based on the fusion lookup table to obtain the processed overlapping region.

[0153] In step S303, for the non-overlapping region, texture mapping is performed based on the remapping lookup table to obtain the processed non-overlapping region.

[0154] In step S304, the processed overlapping area and the processed non-overlapping area are combined to obtain the panoramic stitching output image.

[0155] Figure 13bThe overlapping and non-overlapping areas of cameras 0, 1, and 2 are shown. Figure 13c A schematic diagram showing the combination of overlapping and non-overlapping regions is provided.

[0156] In one embodiment of this disclosure, for the overlapping region, texture mapping is first performed based on the remapping lookup table, including traversing the coordinates (x, y) of the panoramic stitching output image; according to the coordinates (x, y), the coordinates (mapx(x, y), mapy(x, y)) of the corresponding input image are queried in the remapping lookup table; image content interpolation is performed on the input image, and the interpolation result at the coordinates (mapx(x, y), mapy(x, y)) is obtained; the interpolation result is filled into the coordinates (x, y) of the panoramic stitching output image.

[0157] Figure 14 A schematic diagram of texture mapping in one embodiment of the panoramic stitching method of this disclosure is shown.

[0158] In this embodiment, the texture mapping can remove distortions in the input image, while the back-projection transformation of the output image and the alignment of image content from adjacent cameras both rely on the remap LUT to provide correct texture mapping coordinates. Texture mapping is a reverse mapping process, and the texture mapping formula used includes:

[0159] dst(x,y)=src(mapx(x,y),mapy(x,y))

[0160] In this context, mapx and mapy represent remap LUT, src represents the input image, and dst represents the output image.

[0161] Specifically, the coordinates (x, y) on the output image dst are traversed. Here, the coordinates (x, y) are integers. Based on (x, y), the remap LUT is queried to obtain the corresponding coordinates (mapx(x, y), mapy(x, y)) on the input image. These coordinates (mapx(x, y), mapy(x, y)) will most likely contain decimals. Therefore, image content interpolation needs to be performed on the input image src, such as using linear interpolation, cubic interpolation, Lanczos interpolation, etc. The interpolation result is then filled back into the (x, y) position of the output image, thus obtaining the pixel content dst(x, y) at coordinates (x, y).

[0162] It should be noted that the steps for texture mapping in non-overlapping regions are the same as those for texture mapping in overlapping regions, so they will not be repeated here.

[0163] The texture mapping process disclosed herein can be implemented using a CPU, GPU, DSP, or a unit with texture mapping capabilities.

[0164] In one embodiment of this disclosure, image fusion of the texture-mapped overlapping region based on the fusion lookup table includes: performing image fusion on the texture-mapped image based on any one of the Alpha fusion algorithm, multi-band fusion algorithm, and Poisson fusion algorithm.

[0165] Specifically, the Alpha fusion algorithm is simple to implement and has low overhead, but the merged image may appear "transparent." Multi-band fusion algorithms are complex and have high overhead, but they can balance the transition between low-frequency and high-frequency information, resulting in a better fused image. The Poisson fusion algorithm is even more complex to implement, but it also achieves better fusion results. In practical applications, the appropriate fusion algorithm can be selected based on the specific requirements.

[0166] In one embodiment of this disclosure, image fusion of overlapping regions after texture mapping based on the Alpha fusion algorithm includes: traversing the coordinates (x, y) of the panoramic stitching output image; querying the corresponding alpha(x, y) in the fusion lookup table according to the coordinates (x, y); obtaining the pixel value Image c(x, y) of the input image; calculating the fused pixel value based on the alpha(x, y) and the pixel value Image c(x, y); and filling the fused pixel value into the coordinates (x, y) of the panoramic stitching output image.

[0167] Specifically, the calculation of the fused pixel value based on the alpha(x,y) and the pixel value Image c(x,y) includes the following formula:

[0168] Blend(x,y)=alpha(x,y)*Image1(x,y)+(1-alpha(x,y))*Image2(x,y)

[0169] Where alpha(x,y) represents the alpha value at coordinate (x,y) in the fusion lookup table, Image1(x,y) represents the pixel value at coordinate (x,y) in the first input image, Image2(x,y) represents the pixel value at coordinate (x,y) in the second input image, and Blend(x,y) represents the fused pixel value.

[0170] In this embodiment of the disclosure, the image stitching described in steps S301 to S304 is also performed using an online processing method.

[0171] This disclosure evaluates the proposed panoramic stitching method in an indoor basketball court environment. Two sets of images were prepared to comprehensively evaluate the panoramic stitching effect. Figure 15a and Figure 15b The images shown are the input images to be stitched, captured from different angles. Figure 15c The panoramic stitching output image obtained by applying the panoramic stitching method disclosed herein is shown.

[0172] By comparing the input image to be stitched and the panoramic stitched output image, it can be clearly seen that the panoramic stitching method disclosed in this paper exhibits better stitching results. In particular, in the panoramic stitched output image, there are no seams or ghosting phenomena in the basketball court ground area from near to far. This shows that the stitching process not only successfully combines multiple input images seamlessly, but also maintains the continuity of the scene and visual consistency.

[0173] It should be noted that the scope of protection of the panoramic stitching method described in this disclosure is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the scope of protection of this disclosure.

[0174] like Figure 16 As shown in the figure, this embodiment provides a panoramic stitching device. The panoramic stitching device includes an image acquisition module and an image stitching module.

[0175] The image acquisition module is configured to acquire images captured by multiple cameras in the same scene.

[0176] The image stitching module is configured to acquire a remapping lookup table and a fusion lookup table generated based on a stitching model. The stitching model includes a hemisphere and a cylinder, and the origin of the virtual camera coordinate system associated with the multiple cameras is located at the center of the base surface where the hemisphere and the cylinder meet. The image stitching module is configured to successively map and fuse the captured images based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image.

[0177] In one embodiment of this disclosure, the panoramic stitching device provided in this embodiment may further include a lookup table generation module. The lookup table generation module is configured to generate the remapping lookup table and the fusion lookup table based on the stitching model.

[0178] In some embodiments, the lookup table generation module of this disclosure can be integrated into a chip. That is, the panoramic stitching device can be integrated into a single system-on-a-chip (SoC), and this single SoC is contained within an electronic device. In other embodiments, the image acquisition module and the image stitching module can be integrated into a first chip, and the panoramic stitching device can be integrated into a second chip, with the first chip and the second chip contained within an electronic device. In this way, the electronic device can independently execute the panoramic stitching method described herein.

[0179] In other embodiments, the lookup table generation module of this disclosure can also run as a separate module in the cloud. This design allows users to leverage the powerful computing capabilities of cloud computing to generate remapping lookup tables and fusion lookup tables for image stitching. In this case, the image stitching module can perform image stitching operations by downloading these pre-computed lookup tables from the cloud. The advantage is that it reduces the computational burden on local devices, enabling even devices with limited computing power to achieve high-quality image stitching.

[0180] It should be noted that the image acquisition module, lookup table generation module, and image stitching module of this disclosure can be configured to perform the corresponding steps or actions in the panoramic stitching method described above, or correspond one-to-one with the steps in the panoramic stitching method described above, so they will not be described in detail here.

[0181] The panoramic stitching device provided in this embodiment can implement the panoramic stitching method described in this disclosure. However, the devices for implementing the panoramic stitching method described in this disclosure include, but are not limited to, the structures of the panoramic stitching devices listed in this embodiment. Any structural modifications and substitutions of the prior art made based on the principles of this disclosure are included within the protection scope of this disclosure.

[0182] like Figure 17 As shown, this embodiment provides an electronic device including a processor and a memory. The memory is configured to store an executable program. The processor is configured to execute the program to cause the electronic device to perform the panoramic stitching method described above.

[0183] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0184] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0185] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0186] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed, implements the panoramic stitching method described in any of the above embodiments. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0187] This disclosure also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this disclosure are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0188] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0189] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0190] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A panoramic stitching method, characterized in that, include: Acquire images captured by multiple cameras in the same scene; Obtain a remapping lookup table and a fusion lookup table generated based on the stitching model, wherein the stitching model includes a hemisphere and a cylinder, and the origin of the virtual camera coordinate system associated with the multiple cameras is located at the center of the bottom surface where the hemisphere and the cylinder meet; and The captured images are successively mapped and fused based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image. The center of the hemisphere is the origin of the virtual camera coordinate system, the radius of the hemisphere is the farthest stitching distance, the center of the upper surface of the cylinder is the origin of the virtual camera coordinate system, the center of the lower surface of the cylinder is the origin of the world coordinate system, the radius of the cylinder is the farthest stitching distance, and the height of the cylinder is the ground clearance of the multiple cameras.

2. The panoramic stitching method according to claim 1, characterized in that, Based on the remapping lookup table and the fusion lookup table, the captured images are successively mapped and fused to obtain a panoramic stitched output image, including: Based on the remapping lookup table, texture mapping is performed on each overlapping region of the captured image to obtain a first mapped image; Based on the remapping lookup table, texture mapping is performed on each non-overlapping region of the captured image to obtain a second mapped image; Based on the fusion lookup table, the first mapped image is fused to obtain a fused image; and The second mapped image and the fused image are stitched together to obtain the panoramic stitched output image.

3. The panoramic stitching method according to claim 1, characterized in that, Also includes: The remapping lookup table and the fusion lookup table are generated based on the splicing model, including: The plurality of cameras are calibrated to determine their intrinsic and extrinsic parameters; Configure a first stitching parameter and a second stitching parameter, wherein the first stitching parameter is the imaging parameter of the output image, and the second stitching parameter is related to the stitching model; Based on the intrinsic parameters, the extrinsic parameters, the first concatenation parameter, and the second concatenation parameter, the remapping lookup table and the fusion lookup table are constructed; and Save the remapping lookup table and the fusion lookup table.

4. The panoramic stitching method according to claim 3, characterized in that, The first stitching parameters include at least one of the following: the width of the output image, the height of the output image, the horizontal field of view of the output image, the vertical field of view of the output image, the horizontal offset of the center point of the output image, the vertical offset of the center point of the output image, and the projection method of the output image. The second stitching parameters include the ground clearance of the plurality of cameras and the maximum stitching distance.

5. The panoramic stitching method according to claim 3, characterized in that, Based on the intrinsic parameters, the extrinsic parameters, the first concatenation parameter, and the second concatenation parameter, constructing the remapping lookup table includes: Iterate through the coordinates (x, y) in the output image; The coordinates (x, y) are back-projected onto the origin of the virtual camera coordinate system. On the surface of the sphere with center at , let the projection point be . , the rays are ; Based on the first splicing parameters, the direction vector of the ray is calculated as follows: ; Based on the second splicing parameters, the direction vector is determined. The modulus length S; According to the formula Calculate the projection point Coordinates in the virtual camera coordinate system ; Based on the extrinsic parameters and the coordinates Calculate the projection point Coordinates in different physical camera coordinate systems ;as well as Based on the intrinsic parameters, the coordinates Convert to pixel coordinates (u, v) on the input image.

6. The panoramic stitching method according to claim 5, characterized in that, Based on the first splicing parameters, the direction vector of the ray is calculated as follows: Including the use of the following formula: ， ， ， Where dst_center_x represents the horizontal offset of the center point of the output image, dst_center_y represents the vertical offset of the center point of the output image, dst_fov_x represents the horizontal field of view of the output image, dst_fov_y represents the vertical field of view of the output image, dst_width represents the width of the output image, and dst_height represents the height of the output image. Represents the direction vector x-coordinate point, Represents the direction vector The y-coordinate point, Represents the direction vector The z-coordinate point.

7. The panoramic stitching method according to claim 5, characterized in that, Determining the magnitude S of the direction vector based on the second splicing parameters includes: when hour, , when hour, , when hour, , in Represents the direction vector x-coordinate point, Represents the direction vector The y-coordinate point, Represents the direction vector The z-coordinate point, R represents the farthest stitching distance, and h represents the height of the virtual camera above the ground.

8. The panoramic stitching method according to claim 5, characterized in that, Based on the extrinsic parameters and the coordinates Calculate the projection point Coordinates in different physical camera coordinate systems Including the use of the following formula: ， ， ， in Represents the projection point Coordinates in the virtual camera coordinate system , Represents the extrinsic parameter matrix. This represents the extrinsic parameters of camera 1 relative to camera 0. This represents the extrinsic parameters of camera 2 relative to camera 0. Represents the projection point Coordinates in the camera's 0 coordinate system Represents the projection point Coordinates in the camera 1 coordinate system Represents the projection point Coordinates in the camera 2 coordinate system.

9. The panoramic stitching method according to claim 5, characterized in that, Based on the intrinsic parameters, the coordinates Converting to pixel coordinates (u, v) on the input image involves the following formula: , , , ， Where k1, k2, k3, k4, k5 and k6 represent radial distortion coefficients, p1 and p2 represent tangential distortion coefficients, fx and fy represent the camera's focal length, and cx and cy represent the offset of the image center point.

10. The panoramic stitching method according to claim 1, characterized in that, Based on the remapping lookup table and the fusion lookup table, the captured images are successively mapped and fused to obtain a panoramic stitched output image, including: The input captured image is divided into overlapping and non-overlapping regions; For the overlapping region, texture mapping is first performed based on the remapping lookup table, and then image fusion is performed on the texture-mapped overlapping region based on the fusion lookup table to obtain the processed overlapping region. For the non-overlapping regions, texture mapping is performed based on the remapping lookup table to obtain the processed non-overlapping regions; and The processed overlapping area and the processed non-overlapping area are combined to obtain the panoramic stitched output image.

11. The panoramic stitching method according to claim 10, characterized in that, For the overlapping region, the texture mapping based on the remapping lookup table first includes: Iterate through the coordinates (x, y) of the panoramic stitched output image; Based on the coordinates (x, y), look up the corresponding coordinates of the input image (mapx(x, y), mapy(x, y)) in the remapping lookup table; The input image is interpolated to obtain the interpolation result at coordinates (mapx(x,y), mapy(x,y)); and The interpolation result is then filled into the coordinates (x, y) of the panoramic stitched output image.

12. The panoramic stitching method according to claim 10, characterized in that, Image fusion of the texture-mapped overlapping regions based on the fusion lookup table includes: Image fusion is performed on texture-mapped images based on any one of the Alpha fusion algorithm, multi-band fusion algorithm, and Poisson fusion algorithm.

13. The panoramic stitching method according to claim 12, characterized in that, Image fusion based on the Alpha fusion algorithm for overlapping regions after texture mapping includes: Iterate through the coordinates (x, y) of the panoramic stitched output image; Based on the coordinates (x, y), look up the corresponding alpha(x, y) in the fusion lookup table; Get the pixel values ​​of the input image: Image c(x,y); Based on the alpha(x,y) and the pixel value Image c(x,y), calculate the fused pixel value; and The fused pixel values ​​are filled into the coordinates (x, y) of the panoramic stitched output image.

14. The panoramic stitching method according to claim 13, characterized in that, Based on the alpha(x,y) and the pixel value Image c(x,y), the fused pixel value is calculated using the following formula: Blend(x,y)=alpha(x,y) * Image1(x,y) + (1-alpha(x,y)) * Image2(x,y) Where alpha(x,y) represents the alpha value at coordinate (x,y) in the fusion lookup table, Image1(x,y) represents the pixel value at coordinate (x,y) in the first input image, Image2(x,y) represents the pixel value at coordinate (x,y) in the second input image, and Blend(x,y) represents the fused pixel value.

15. The panoramic stitching method according to claim 3, characterized in that, Calibrating the plurality of cameras to determine their intrinsic and extrinsic parameters includes: The image of the calibration plate used for internal parameter calibration is acquired as the first calibration image; The image of the calibration plate used for external parameter calibration is acquired as the second calibration image; The first calibration image and the second calibration image are processed based on a preset calibration algorithm to obtain the camera's intrinsic parameters, extrinsic parameters, and reprojection error; Based on the reprojection error, determine whether the camera calibration results meet the standards; If so, then save the intrinsic parameters and the extrinsic parameters; and Otherwise, adjust the calibration board and / or replace the calibration algorithm, and repeat the above calibration steps until the camera's calibration result meets the standard.

16. A panoramic stitching device, characterized in that, include: The image acquisition module is configured to acquire images captured by multiple cameras in the same scene; as well as The image stitching module is configured as follows: Obtain a remapping lookup table and a fusion lookup table generated based on the stitching model, wherein the stitching model includes a hemisphere and a cylinder, and the origin of the virtual camera coordinate system associated with the multiple cameras is located at the center of the bottom surface where the hemisphere and the cylinder meet; and The captured images are successively mapped and fused based on the remapping lookup table and the fusion lookup table to obtain a panoramic stitched output image. The center of the hemisphere is the origin of the virtual camera coordinate system, the radius of the hemisphere is the farthest stitching distance, the center of the upper surface of the cylinder is the origin of the virtual camera coordinate system, the center of the lower surface of the cylinder is the origin of the world coordinate system, the radius of the cylinder is the farthest stitching distance, and the height of the cylinder is the ground clearance of the multiple cameras.

17. The panoramic stitching device according to claim 16, characterized in that, Also includes: The lookup table generation module is configured to generate the remapping lookup table and the fusion lookup table based on the splicing model.

18. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the panoramic stitching method according to any one of claims 1 to 15.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the panoramic stitching method according to any one of claims 1 to 15.

Images