Image processing method
By distributing anchor points in the image overlap area and calculating feature offsets, seamless alignment and stitching of images are achieved, artifacts, discontinuities and distortion problems during image stitching in the prior art are solved, and high-quality panoramic images are generated.
Patent Information
- Application Number
- CN202410348799.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-12-15
- Filing Date
- 2024-03-26
- Publication Date
- 2025-06-17
AI Technical Summary
When the prior art splicing images with a large number of parallaxes into a panoramic image, visual artifacts, discontinuities and distortions in the overlapping areas are prone to occur.
By obtaining the overlapping areas of the two images and distributing anchor points in these overlapping areas, the anchor points are used to align the two images. The specific steps include identifying anchor points in the overlapping area, calculating feature offsets between anchor points, and generating pixel offsets through interpolation, ultimately achieving seamless alignment and stitching of the image.
It effectively solves the problems of artifacts, discontinuities and distortion in the overlapping area during image stitching, and realizes seamless fusion of images and high-quality panoramic image generation.
Smart Images
Figure CN120163705A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing technology. Background Art
[0002] Algorithms for aligning and stitching images captured from different fields of view with overlapping regions into a panorama image have been widely proposed in computer vision. However, most implementations do not consider images with a large amount of parallax and can result in visible artifacts, discontinuities, and distortions in the overlapping regions. Summary of the Invention
[0003] To solve this prominent problem, the present disclosure proposes an image processing method.
[0004] According to one of the exemplary embodiments, the method includes obtaining a first image and a second image of a scene, identifying a first overlapping region in the first image and a second overlapping region in the second image, distributing anchor points at regular intervals in the first overlapping region and the second overlapping region, and aligning the first image and the second image based on the anchor points in the first overlapping region and the anchor points in the second overlapping region, where the first overlapping region and the second overlapping region are the overlapping regions between the first image and the second image. Brief Description of the Drawings
[0005] Figure 1 A schematic diagram showing an image processing device according to an exemplary embodiment of the present disclosure.
[0006] Figure 2 A flowchart showing an image processing method according to an exemplary embodiment of the present disclosure.
[0007] Figures 3A to 3F A schematic diagram and an implementation showing an image processing method according to one of the exemplary embodiments of the present disclosure.
[0008] To make the above features and advantages of the present application more understandable, several embodiments accompanied by the drawings are elaborated in detail below. Detailed Description of the Embodiments
[0009] Some embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the present application are shown. In fact, the various embodiments of the present disclosure may be implemented in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided so that the present disclosure meets the applicable legal requirements. The same reference numerals refer to the same elements throughout.
[0010] Figure 1A schematic diagram showing an image processing apparatus according to an exemplary embodiment of the present disclosure. All components and configurations of the apparatus are first introduced in Figure 1 and the functions of the components are disclosed in more detail later in Figure 2 .
[0011] Please refer to Figure 1 . The image processing apparatus 100 will at least include a memory 110 and a processor 120. The image processing apparatus 100 may be an electronic system or a computer system. The memory 110 may be various forms of random-access memory (RAM), such as dynamic random-access memory (DRAM). The processor 120 may be one or more of a North Bridge, a South Bridge, a field-programmable gate array (FPGA), a programmable logic device (PLD), an application-specific integrated circuit (ASIC), other similar devices, or a combination thereof. The processor 120 may also be a central processing unit (CPU), a programmable general or special microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), other similar devices, or a combination thereof.
[0012] Figure 2 A flowchart showing an image processing method according to an exemplary embodiment of the present disclosure, where the steps shown in Figure 2 can be implemented by the image processing apparatus 100 shown in Figure 1 .
[0013] Please also refer to Figure 1 and Figure 2, the processor 120 of the image processing device 100 will obtain a first image and a second image of a scene (step S202), and will identify a first overlapping region in the first image and a second overlapping region in the second image (step S204). Herein, the first overlapping region and the second overlapping region will be the overlapping regions between the first image and the second image, where the two overlapping regions capture the same point from different viewpoints. In one embodiment, the two images may be images captured simultaneously by different cameras facing different angles to capture a wider viewing angle. The two images may be transmitted to the image processing device 100 in a wired or wireless manner in real time, or stored in a local database or a cloud server for later processing and analysis. Additionally, the two images may be subjected to camera calibration after being captured to re-project the images, make the brightness and contrast consistent, remove noise, and the present disclosure is not limited thereto.
[0014] Next, the processor 120 will distribute a plurality of anchor points at regular intervals in the first overlapping region and the second overlapping region (step S206). That is to say, the anchor points may be evenly distributed in the overlapping region between the first image and the second image regardless of the complexity of the image content. The density of the anchor points may be set to, for example, 20 pixels, which means the distance between each anchor point is 20 pixels. In other words, each anchor point is equidistantly spaced from each other.
[0015] Next, the processor 120 will align the first image and the second image based on the anchor points in the first overlapping region and the anchor points in the second overlapping region (step S208). The image alignment may be based on each pair of anchor points (i.e., a correspondence) located at the same position in the first overlapping region and the second overlapping region. Since the distribution of the anchor points is independent of the image content, the solution proposed in the present disclosure is effective even for scenes with low contrast, less texture, no corners or edges, or multiple depths compared to existing methods that ultimately result in poor image alignment.
[0016] For better understanding, Figures 3A to 3F a schematic diagram and an implementation of an image processing method according to one of the exemplary embodiments of the present disclosure are shown, and Figures 3A to 3F the method shown may also be implemented by Figure 1 the image processing device 100 shown.
[0017] Please refer to Figure 1 and Figure 3A, the processor 120 will first obtain a first image IMG1 and a second image IMG2 of the scene, and will identify a first overlapping region R1 in the first image IMG1 and a second overlapping region R2 in the second image IMG2. Herein, the first image IMG1 and the second image IMG2 are simultaneously captured by different cameras facing different angles, and the first image IMG1 and the second image IMG2 can be subjected to camera calibration.
[0018] Next, please refer to Figure 1 and Figure 3B , the processor 120 will distribute anchor points at regular intervals in the first overlapping region R1 and the second overlapping region R2. It should be noted that every two anchor points located at the same position in the first overlapping region R1 and the second overlapping region R2 respectively can be regarded as a corresponding group. Next, the processor 120 will calculate the feature offset between the two anchor points in each corresponding group according to the patches centered on each two anchor points in each corresponding group.
[0019] For illustrative purposes, the first anchor point a1 in the first overlapping region R1 and the second anchor point a2 located at the same pixel position in the second overlapping region R2 form a corresponding group, and the first patch P1 and the second patch P2 are two patches corresponding to the first anchor point a1 and the second anchor point a2 respectively. The processor can determine the feature offset between the first anchor point a1 and the second anchor point a2 by shifting the first patch P1 and the second patch P2 to the best matching position within a given range of the first overlapping region R1 and the second overlapping region R2 based on a predetermined statistical criterion (such as the sum of squared differences (SSD) of the pixel values between the two patches P1 and P2). Such a statistical criterion is hereinafter referred to as "matching cost".
[0020] For example, when the offset between the filling blocks P1 and P2 is 0, this means that both the first filling block P1 and the second filling block P2 are located at their original positions, and the processor 120 can calculate the matching cost SUM_0. When the offset between the first filling block P1 and the second filling block P2 is 1, this means that the second filling block P2 is shifted 1 pixel to the right, and the processor 120 can calculate the matching cost SUM_1. Derived in a similar way, when the offset between the first filling block P1 and the second filling block P2 is N, this means that the second filling block P2 is shifted N pixels to the right, and the processor 120 can calculate the matching cost SUM_N. The processor 120 sets the minimum value among the matching costs SUM_0, SUM_1, …, and SUM_N as the feature offset between the first anchor point a1 and the second anchor point a2. From another perspective, when the first filling block P1 and the second filling block P2 are located at the best matching positions (i.e., the positions with the minimum matching cost), the features in P1 and the features in P2 exhibit the highest similarity.
[0021] It should be noted that in many cases, due to the lack of reference features, excessive noise, etc., some feature offsets may be unreliable. In an exemplary embodiment, from a time perspective, if the two overlapping regions are obtained from two video frame sequences respectively, the processor 120 can calculate the feature offsets of each corresponding group of multiple consecutive frames (for example, the first image in the first image sequence and at least one first previous image before the first image, and the second image in the second image sequence and at least one second previous image before the second image), and calculate the new feature offset of each corresponding group, for example, by taking the average of the feature offsets of 5 frames. For each corresponding group, the original feature offset will be replaced by the new feature offset to improve reliability. In an exemplary embodiment, from a spatial perspective, the processor 120 can identify and replace unreliable feature offsets according to a reliability threshold. For example, for two anchor points in a corresponding group with a feature offset K having a matching cost SUM_K, if is less than the reliability threshold, the feature offset K becomes unreliable and will be replaced by a substitute derived from other reliable feature offsets. In an exemplary embodiment, the processor 120 can ensure the reliability of each feature offset from both the time perspective and the spatial perspective. The present disclosure is not limited thereto. More details are provided below for a better understanding from the spatial perspective.
[0022] Please also refer to Figure 1 and Figure 3C, assume that the feature offset between the first anchor point b1 in the first overlapping region R1 and the second anchor point b2 in the second overlapping region R2 is unreliable. The processor 120 can calculate a possible feature offset based on the feature offsets of adjacent anchor points, such as the feature offset between anchor points c1 and c2, the feature offset between anchor points d1 and d2, and the feature offset between anchor points e1 and e2. For example, the possible feature offset between the first anchor point b1 and the second anchor point b2 can be the average of the adjacent feature offsets. The original feature offset will be replaced with the possible feature offset as a more reliable new feature offset for each corresponding group.
[0023] Next, please refer to both Figure 1 and Figure 3D , the processor 120 will generate a feature offset map M1 based on the reliable feature offset and the new (replaced) feature offset, and thus generate two anchor point offset maps M11 and M12 corresponding to the anchor points in the first overlapping region R1 and the second overlapping region R2 respectively. It should be noted that the data at the corresponding positions in the two anchor point offset maps M11 and M12 have equal magnitudes but opposite signs. For example, assume that the feature offset between anchor points g1 and g2 in the feature offset map M1 is 5. The anchor point offsets of both anchor points g1 and g2 located in the anchor point offset maps M11 and M12 are half of the magnitude 5 but have different signs (i.e., anchor point g1 is 2.5 and anchor point g2 is -2.5). On the contrary, assume that the feature offset between anchor points h1 and h2 in the feature offset map M1 is -5. The anchor point offsets of both anchor points h1 and h2 located in the anchor point offset maps M11 and M12 are both 2.5 but have different signs (i.e., anchor point h1 is -2.5 and anchor point h2 is 2.5).
[0024] Next, please refer to both Figure 1 and Figure 3E , the processor 120 can perform interpolation (e.g., pixel-wise linear interpolation) on the anchor point offset map M11 corresponding to the first overlapping region R1 and the anchor point offset map M12 corresponding to the second overlapping region R2 to generate image alignment maps M21 and M22 respectively, such that each pixel in the overlapping regions R1 and R2 has a corresponding pixel offset. In other words, the number of new (interpolated) anchor point offsets and existing (original) anchor point offsets in each of the anchor point offset maps M21 and M22 is the same as the number of pixels in each of the first overlapping region R1 and the second overlapping region R2.
[0025] Next, refer to both Figure 1 , Figure 3E and Figure 3F , the processor 120 can according toFigure 3E The pixel offsets in the image alignment diagrams M21 and M22 are respectively used to perform image alignment on the overlapping regions R1 and R2 to generate a first aligned overlapping region R1' and a second aligned overlapping region R2'. Thereafter, the processor 120 can perform image fusion on the first aligned overlapping region R1' and the second aligned overlapping region R2' based on any one of the existing image blending methods (such as alpha blending) to generate a seamless fused overlapping region R, and perform image stitching according to the first image, the second image, and the fused overlapping region R to generate a combined image IMG12. During this process, the fused overlapping region R provides a smooth transition between the first image and the second image.
[0026] In an exemplary embodiment, image alignment and stitching can be extended to three or more images, where there are overlapping regions between every two images in a manner similar to the above. In addition, the above framework can provide immediate applications in many fields such as video surveillance, panorama rendering, immersive telepresence and conferencing, and autonomous driving.
[0027] It will be apparent to those skilled in the art that various modifications and changes can be made to the structure of the disclosed embodiments without departing from the scope or spirit of the present disclosure. In summary, the present disclosure is intended to cover modifications and variations of the present disclosure that fall within the scope of the following claims and their equivalents.
Claims
1. An image processing method, comprising: Obtaining a first image and a second image of a scene; identifying a first overlapping region in the first image and a second overlapping region in the second image, wherein the first overlapping region and the second overlapping region are overlapping regions between the first image and the second image; distributing a plurality of anchor points at regular intervals in the first overlapping region and the second overlapping region; as well as The first image is aligned with the second image according to the anchor point in the first overlapping region and the anchor point in the second overlapping region.
2. The image processing method according to claim 1, wherein each of the anchor points in the first overlapping region and the anchor points in the second overlapping region are equally spaced from each other, wherein the anchor points in the first overlapping region include a first anchor point, wherein the anchor points in the second overlapping region include a second anchor point, and The first anchor point and the second anchor point are located at the same position in the first overlapping area and the second overlapping area respectively.
3. The image processing method according to claim 2, wherein the step of aligning the first image with the second image according to the anchor point in the first overlapping area and the anchor point in the second overlapping area comprises: Calculating a feature offset between the first anchor point and the second anchor point; generating a feature offset map including the feature offset between the first anchor point and the second anchor point; Generating a first anchor point offset map and a second anchor point offset map according to the feature offset map; interpolating the first anchor point offset map and the second anchor point offset map to generate a first image alignment map and a second image alignment map respectively; and The first overlapping area is aligned with the second overlapping area according to the first image alignment map and the second image alignment map.
4. The image processing method according to claim 3, wherein the step of calculating the feature offset between the first anchor point and the second anchor point comprises: Setting a first patch centered at the first anchor point and a second patch centered at the second anchor point; as well as The characteristic offset between the first anchor point and the second anchor point is determined by shifting the first patch and the second patch to best matching positions in the first overlap region and the second overlap region, respectively.
5. The image processing method according to claim 4, The feature in the first patch located at the best matching position has the highest similarity with the feature in the second patch.
6. The image processing method according to claim 3, wherein after the step of generating the feature offset map, the image processing method further comprises: A new feature offset between the first anchor point and the second anchor point is calculated based on at least one first previous image before the first image in a first image sequence and at least one second previous image before the second image in a second image sequence, and the feature offset between the first anchor point and the second anchor point is replaced by the new feature offset.
7. The image processing method according to claim 3, wherein after the step of generating the feature offset map, the image processing method further comprises: Determining whether the characteristic offset between the first anchor point and the second anchor point is reliable; as well as In response to the feature offset between the first anchor point and the second anchor point being unreliable, a new feature offset between the first anchor point and the second anchor point is calculated based on the feature offset between a first adjacent anchor point and a second adjacent anchor point, and the feature offset between the first anchor point and the second anchor point is replaced by the new feature offset.
8. An image processing method according to claim 3, wherein the first anchor point offset map includes multiple anchor point offsets corresponding to the anchor points in the first overlapping area and the anchor points in the second overlapping area respectively, and wherein the first anchor point offset corresponding to the first anchor point and the second anchor point offset have the same value and different positive and negative signs. 9 . The image processing method according to claim 8 , wherein the magnitude of the first anchor point offset and the magnitude of the second anchor point offset are half of the magnitude of the feature offset between the first anchor point and the second anchor point.
10. The image processing method according to claim 8, wherein the step of interpolating the first anchor point offset map and the second anchor point offset map to generate the first image alignment map and the second image alignment map respectively comprises: A plurality of new anchor point offsets are interpolated between the anchor point offsets corresponding to the anchor points in the first overlap region and the anchor points in the second overlap region, respectively.
11. The image processing method according to claim 10, The number of the new anchor offsets and the anchor offsets in each of the first anchor offset map and the second anchor offset map is respectively the same as the number of pixels in each of the first overlapping region and the second overlapping region.
12. The image processing method according to claim 1, further comprising: Performing image blending according to the aligned first overlapped area and the aligned second overlapped area to generate a blended overlapped area; as well as Image stitching is performed based on the first image, the second image, and the blended overlap region to generate a combined image of the scene.