Image splicing method and device and computer program product

Abstract

The invention discloses an image splicing method and device and a computer program product, and the method comprises the steps: respectively mapping images collected by cameras at different positions to a pre-established imaging spherical surface, and obtaining respective mapping images; layering each mapping image according to the movement speed of the scenery in each mapping image to obtain a multi-layer image; and respectively setting the same scene distance hierarchy for the images with the similar movement speeds, and splicing the images with the same scene distance hierarchy to generate a panoramic image. According to the method, the image is layered based on speed, so that smooth transition and natural fusion from close shot to long shot are ensured, the problems of ghosting and dislocation in traditional panoramic stitching are effectively solved, and the quality and flexibility of image stitching in a large parallax scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to an image stitching method, apparatus, and computer program product. Background Technology

[0002] In the field of virtual reality panoramic technology, current stitching and fusion techniques are primarily applied to scenarios with minimal parallax. In these scenarios, the configuration of multiple pinhole, wide-angle, or fisheye lenses is approximately coaxial, meaning their relative spatial positions allow them to capture images from approximately the same viewpoint. This configuration allows for the use of traditional stitching techniques because the misalignment and distortion generated during stitching of images from different cameras are minimal, resulting in smoother and more natural panoramic images.

[0003] However, when the parallax of a scene is large, meaning the cameras differ significantly in their spatial distribution, the application of traditional stitching techniques faces challenges. Because different cameras have varying perspectives and occlusion conditions, the images of the same physical target they capture will exhibit significant differences. As parallax increases, the effects of these errors and distortions gradually intensify, leading to particularly prominent ghosting and misalignment phenomena during the stitching process, severely impacting the quality and visual appeal of panoramic images. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide an image stitching method, apparatus and computer program product to reduce distortion or deformation during image stitching under large parallax conditions and improve image quality.

[0005] To address the aforementioned technical problems, this invention provides an image stitching method, comprising the following steps:

[0006] Images captured by cameras located at different positions are mapped onto a pre-established imaging sphere to obtain their respective mapped images;

[0007] Each mapped image is layered according to the motion speed of the objects in each mapped image to obtain multi-layered images;

[0008] Images with similar motion speeds are assigned the same field of view level, and images at the same field of view level are stitched together to generate a panoramic image.

[0009] Preferably, the step of layering each mapped image according to the motion speed of objects in each mapped image to obtain multi-layered images specifically includes:

[0010] The motion velocity of each pixel in each mapped image is calculated using an optical flow algorithm;

[0011] Based on the motion velocity calculated by the optical flow algorithm, pixels belonging to the same velocity range in the mapped image are separated to form images with similar motion velocities in each layer.

[0012] Preferably, the scene distance levels are set from low to high according to the distance between scenes, from near to far.

[0013] Preferably, the radius of the imaging sphere is set to 10 to 100 meters.

[0014] The present invention also provides an image stitching device, comprising:

[0015] The mapping module is used to map images captured by cameras located at different positions onto a pre-established imaging sphere to obtain their respective mapped images;

[0016] The layering module is used to layer each mapped image according to the motion speed of the objects in each mapped image, thus obtaining multi-layered images;

[0017] The stitching module is used to set the same scene distance level for images with similar motion speeds, and to stitch together images with the same scene distance level to generate a panoramic image.

[0018] Preferably, the layered module is specifically used for:

[0019] The motion velocity of each pixel in each mapped image is calculated using an optical flow algorithm;

[0020] Based on the motion velocity calculated by the optical flow algorithm, pixels belonging to the same velocity range in the mapped image are separated to form images with similar motion velocities in each layer.

[0021] Preferably, the scene distance levels are set from low to high according to the distance between scenes, from near to far.

[0022] Preferably, the radius of the imaging sphere is set to 10 to 100 meters.

[0023] The present invention also provides an image stitching device, comprising:

[0024] One or more processors;

[0025] Memory;

[0026] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to perform the image stitching method.

[0027] The present invention also provides a computer program product, including computer instructions that instruct a computer device to perform an operation corresponding to the method.

[0028] The present invention offers the following advantages: By performing velocity-based image layering, it ensures a smooth transition and natural blending from foreground to background, effectively solving the ghosting and misalignment problems in traditional panoramic stitching and significantly improving the quality and flexibility of image stitching in high parallax scenes. By decomposing panoramic images into different scene distance levels and processing each level independently, it can optimize for image features at different scene distances, enhancing the adaptability and robustness of the stitching process. This invention has broad application prospects, providing users with a richer and more realistic visual experience. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating an image stitching method according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the principle of an embodiment of the present invention. Detailed Implementation

[0032] The following description of the embodiments is taken with reference to the accompanying drawings, which illustrate specific embodiments in which the invention can be implemented.

[0033] Please refer to Figure 1 As shown, Embodiment 1 of the present invention provides an image stitching method, including the following steps:

[0034] Images captured by cameras located at different positions are mapped onto a pre-established imaging sphere to obtain their respective mapped images;

[0035] Each mapped image is layered according to the motion speed of the objects in each mapped image to obtain multi-layered images;

[0036] Images with similar motion speeds are assigned the same field of view level, and images at the same field of view level are stitched together to generate a panoramic image.

[0037] As demonstrated by the above steps, this invention effectively solves the problems of ghosting and misalignment in image stitching under large parallax scenes by mapping images captured by different cameras onto a unified imaging sphere and intelligently layering them according to the movement speed of objects. Furthermore, by setting the same scene distance level for image layers with similar movement speed trends for precise stitching, the generated panoramic image exhibits higher consistency and visual coherence.

[0038] Specifically, as mentioned above, when using cameras located at different positions to capture images, a large parallax will be introduced if the positional difference between the cameras is large. Therefore, this embodiment of the invention proposes an image stitching method to solve the problem of image stitching under large parallax conditions.

[0039] This invention first establishes a unified imaging sphere. Because different cameras are used, establishing a unified imaging sphere provides a common mapping space for images acquired by all cameras. In this embodiment, the center of the established imaging sphere is used as the origin O of the Cartesian coordinate system, ensuring that all mapping points are referenced to the center of the sphere, forming a unified coordinate system. The radius R of the imaging sphere is set within the range of 10 to 100 meters to ensure that the imaging sphere can cover the field of view of all cameras while avoiding unnecessary space waste.

[0040] After the imaging sphere is established, images captured by cameras at different positions, such as pinhole and wide-angle lenses, are mapped onto this imaging sphere to obtain their respective mapped images. This avoids distortion or deformation, and the image surface appears natural and smooth when viewed from the origin of the imaging sphere.

[0041] Each image point captured by the camera is filled in according to its corresponding position on the imaging sphere, ensuring image integrity and spatial consistency. By establishing a mapping relationship between the imaging sphere coordinates and the camera image coordinates, a precise conversion from camera coordinates to imaging sphere coordinates is achieved, resulting in respective mapped images. The mapping relationship can be established using either on-site calibration, where the correspondence between camera coordinates and imaging sphere coordinates is determined through a calibration process in the actual shooting environment; or using a fitting function method, where a mapping function is fitted based on actual test data using mathematical models and algorithms to achieve coordinate transformation.

[0042] This invention employs an optical flow algorithm to perform layered processing on each mapped image. Optical flow is a computer vision technique used to estimate the motion of feature points in an image sequence. When images acquired by multiple cameras are mapped onto a unified imaging sphere, the velocity of objects in the images is also projected onto the sphere accordingly. In other words, even if objects exhibit different optical flows in the original images from different cameras (due to different viewing angles), their projections onto the imaging sphere will reflect their true three-dimensional velocity.

[0043] Each camera captures an image containing both foreground and background elements of the scene. In a video stream, because objects closer to the camera appear to move faster relative to the camera at the same actual speed of movement, objects further away from the camera typically appear to move more slowly.

[0044] Because the imaging sphere provides a unified reference frame, the measurement of velocity is globally consistent. Even with large parallax, if objects exhibit different optical flows in images from different cameras, objects with the same velocity will show the same optical flow when these images are projected onto the imaging sphere. This allows for the separation of foreground and background based on velocity levels; that is, objects can be categorized into different velocity levels according to their optical flow characteristics on the imaging sphere.

[0045] Specifically, the process of layered processing is as follows:

[0046] First, the optical flow algorithm is used to calculate the motion velocity of each pixel in each mapped image. By calculating the motion of feature points in the image sequence, the optical flow algorithm can capture the velocity information of different scenes.

[0047] Then, based on the motion speed calculated by the optical flow algorithm, the pixels in the image are layered according to their motion speed. Pixels with similar motion speeds are grouped into the same layer, with each layer representing a specific speed range. Pixels belonging to the same speed range are separated to form image layers with similar motion speeds, and the objects within these layers visually exhibit consistent motion characteristics. The motion speed levels of the separated image layers are different.

[0048] It should be noted that in this embodiment, "similar motion speed" refers to the fact that, under optical flow algorithm analysis, the motion speeds of pixels in an image are close to each other within a preset speed range, thus they can be considered to belong to the same dynamic level visually. Specifically, similar motion speed means that the changing trends and speed magnitudes of these pixels in the image sequence are relatively consistent, therefore they can be regarded as a group belonging to the same motion state. By setting a specific speed range, pixels with similar motion speed characteristics can be classified into the same motion speed level.

[0049] Applying the same layered processing to all mapped images ensures that data acquired by different cameras are consistent in terms of velocity level. This velocity-level-based separation allows for more efficient stitching of images from different cameras, with foreground and background scenes being properly processed and blended.

[0050] Next, we set the scene distance levels for images with similar motion speeds at each layer. Specifically, please refer to... Figure 2As shown, this embodiment of the invention analyzes the distribution and changes of feature points in each image layer using the hierarchical scale-invariant feature transform (SIFT) algorithm to accurately determine the scene distance information at each layer. The higher the layer, the farther the scene distance, the smaller the impact of parallax on panoramic stitching, and the better the stitching and fusion; similarly, the lower the layer, the smaller the scene distance, and the greater the impact of parallax on panoramic stitching. Figure 2 Four distance levels are shown, with levels 1-4 representing distances from near to far.

[0051] Images at the same layer level mean that the objects they contain are relatively close in spatial location, i.e., they have similar scene distances. Among these images with similar scene distance layers, common feature points are found using feature matching algorithms (such as nearest neighbor search, distance comparison, etc.). These feature points are unique points in the image that can be identified and matched by the algorithm, such as corners, edges, or other salient visual patterns. After finding the common features, these feature points are used as references to align and stitch the images. Gradient domain fusion, multi-layer fusion, and other image fusion techniques can also be used to smoothly merge image edges, reducing stitching artifacts and ultimately forming a seamless panoramic image.

[0052] It should also be noted that, in this embodiment of the invention, the mapped image is processed in layers and a scene distance level is set. The stitching and fusion are processed at different levels, and the stitching and fusion of each layer are independent of each other, which makes the parallax image stitching more flexible. For example, the stitching parameters can be adjusted according to the specific features of each layer, or different processing algorithms can be applied to different layers.

[0053] Corresponding to the image stitching method described in Embodiment 1 of the present invention, Embodiment 2 of the present invention also provides an image stitching device, comprising:

[0054] The mapping module is used to map images captured by cameras located at different positions onto a pre-established imaging sphere to obtain their respective mapped images;

[0055] The layering module is used to layer each mapped image according to the motion speed of the objects in each mapped image, thus obtaining multi-layered images;

[0056] The stitching module is used to set the same scene distance level for images with similar motion speeds, and to stitch together images with the same scene distance level to generate a panoramic image.

[0057] Preferably, the layered module is specifically used for:

[0058] The motion velocity of each pixel in each mapped image is calculated using an optical flow algorithm;

[0059] Based on the motion velocity calculated by the optical flow algorithm, pixels belonging to the same velocity range in the mapped image are separated to form images with similar motion velocities in each layer.

[0060] Preferably, the scene distance levels are set from low to high according to the distance between scenes, from near to far.

[0061] Preferably, the radius of the imaging sphere is set to 10 to 100 meters.

[0062] Corresponding to the image stitching method described in Embodiment 1 of the present invention, Embodiment 3 of the present invention also provides an image stitching device, comprising:

[0063] One or more processors;

[0064] Memory;

[0065] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to perform the image stitching method.

[0066] Corresponding to the image stitching method described in Embodiment 1 of the present invention, Embodiment 4 of the present invention also provides a computer program product, including computer instructions, which instruct a computer device to perform the operation corresponding to the method.

[0067] Preferably, the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The processor is the control center of the device, connecting various parts of the device through various interfaces and lines.

[0068] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, and a Flash Card, or other volatile solid-state storage devices.

[0069] It should be noted that the above-mentioned devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art.

[0070] For the working principle and process of the above embodiments, please refer to the description of Embodiment 1 of the present invention, which will not be repeated here.

[0071] As described above, compared with existing technologies, the beneficial effects of this invention are as follows: By performing speed-based image layering, this invention ensures a smooth transition and natural fusion from foreground to background, effectively solving the ghosting and misalignment problems in traditional panoramic stitching, and significantly improving the quality and flexibility of image stitching in high parallax scenes. By decomposing panoramic images into different scene distance levels and processing each level independently, optimization can be performed on image features at different scene distances, enhancing the adaptability and robustness of the stitching process. This invention has broad application prospects, providing users with a richer and more realistic visual experience.

[0072] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. An image stitching method, characterized by, The method comprises the following steps: mapping images captured by cameras located at different positions onto a pre-established imaging sphere respectively to obtain respective mapping images; layering the mapping images according to the motion speeds of the scenes in the mapping images respectively to obtain multiple layers of images; setting the same scene distance levels for images with similar motion speeds respectively and splicing the images with the same scene distance levels to generate a panoramic image.

2. The method of claim 1, wherein, The layering the mapping images according to the motion speeds of the scenes in the mapping images respectively to obtain multiple layers of images specifically comprises: calculating the motion speeds of each pixel point in the mapping images by using an optical flow algorithm; separating pixel points in the mapping images belonging to the same speed range according to the motion speeds calculated by the optical flow algorithm to form images with similar motion speeds in each layer.

3. The method of claim 1, wherein, The scene distance levels are set from low to high levels in order from near to far according to the distances of the scenes.

4. The method of claim 1, wherein, The radius of the imaging sphere is set to 10 to 100 meters.

5. An image stitching apparatus characterized by comprising: The method comprises: a mapping module configured to map images captured by cameras located at different positions onto a pre-established imaging sphere respectively to obtain respective mapping images; a layering module configured to layer the mapping images according to the motion speeds of the scenes in the mapping images respectively to obtain multiple layers of images; a splicing module configured to set the same scene distance levels for images with similar motion speeds respectively and splice the images with the same scene distance levels to generate a panoramic image.

6. The apparatus of claim 5, wherein, The layering module is specifically configured to: calculate the motion speeds of each pixel point in the mapping images by using an optical flow algorithm; separate pixel points in the mapping images belonging to the same speed range according to the motion speeds calculated by the optical flow algorithm to form images with similar motion speeds in each layer.

7. The apparatus of claim 5, wherein, The scene distance levels are set from low to high levels in order from near to far according to the distances of the scenes.

8. The apparatus of claim 5, wherein, The radius of the imaging sphere is set to 10 to 100 meters.

9. An image stitching apparatus characterized by comprising: The method comprises: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the image splicing method according to any one of claims 1 to 4.

10. A computer program product, characterised in that, The computer instructions instruct a computer device to perform operations corresponding to the method according to any one of claims 1 to 4.

Images