Image fusion method

Abstract

The application provides an image fusion method, which is based on the imaging principle of an image sensor, judges whether a pixel at the same position in front and back frames is dynamic or static according to the difference in light intensity change of the pixel, retains a moving track while suppressing background noise, finally fuses and presents a complete image of a single frame, so that dynamic objects and static scenes in the image both have clear imaging, the reaction speed is fast, the delay is low, the data amount processed under a high frame rate is small, and the power consumption is low.

Description

Technical Field

[0001] This invention relates to an image fusion method, and more particularly to an imaging principle based on an image sensor. It determines whether a pixel is dynamic or static by the difference in light intensity changes of pixels at the same position in previous and subsequent frames, preserves the movement trajectory while suppressing background noise, and finally fuses to present a complete single-frame image, so that both dynamic objects and static scenes in the image have clear imaging. Background Technology

[0002] Image processing typically records images or videos in frames, using single-frame long exposures or multi-frame short exposures to create stacked images. However, regardless of the method, if a fast-moving object is present during the shooting process, blurring and ghosting can easily occur. Figure 1 .

[0003] Current camera technology produces a clear image by taking a picture and exposing it for a period of time. However, if multiple images are played back consecutively to create a video, a ghosting effect will occur if there are objects moving rapidly during the exposure. This will make it difficult to clearly present moving objects in the scene.

[0004] On the other hand, while short-exposure, high-frame-rate imaging can preserve the trajectory of fast-moving objects, it also significantly reduces the quality of the output image due to flicker noise generated by the short exposure of the CIS components.

[0005] Traditional dynamic vision sensors are event-based image sensors capable of calculating brightness changes between consecutive frames at extremely high frame rates (up to 2000fps). Only pixel information with significant brightness changes is recorded; that is, they only observe dynamic objects in the scene and record their coordinates. However, they cannot simultaneously record complete information about static image content, and their short integration time results in high noise and low resolution. Unless a separate color image sensor is added, not only does this increase the cost and size of the sensing module, but it also necessitates increased computational power to solve the spatial synchronization problem between two sensors in different locations, making the overall system more complex.

[0006] Therefore, after observing the above-mentioned issues, the inventors of this case came up with this invention. Summary of the Invention

[0007] To achieve the above objectives, the present invention provides an image fusion method. Based on the imaging principle of an image sensor, the method determines whether a pixel is dynamic or static by the difference in light intensity changes of pixels at the same position in previous and subsequent frames, preserves the movement trajectory while suppressing background noise, and finally fuses to present a complete single-frame image, so that both dynamic objects and static scenes in the image have clear imaging.

[0008] This includes obtaining image information; calculating the brightness value of each pixel; calculating the absolute value of the brightness difference with the previous frame or the sum of the brightness differences of the previous few frames; determining whether the calculation result is less than or equal to a preset threshold; if so, it is determined to be a static pixel; inheriting it frame by frame; otherwise, it is determined to be a dynamic pixel; and refreshing it frame by frame.

[0009] Preferably, each frame of image information further contains several pixels, or can be said to be composed of several pixels, and there are corresponding pixels in each frame of image.

[0010] Ideally, the process of acquiring image information is a continuous process.

[0011] Preferably, after obtaining the brightness value of each pixel, the brightness value of each pixel in the current frame is compared and calculated with the brightness value of the corresponding pixel in the previous frame to calculate the absolute value of the difference, or it is compared and calculated with the brightness value of each corresponding pixel in the previous few frames to calculate the sum of the absolute values ​​of the differences.

[0012] Preferably, after comparing the absolute value of the brightness difference with a preset threshold, if it is less than or equal to the preset threshold, the pixel is confirmed to be a static pixel and is inherited frame by frame, and the pixels of the two frames are superimposed together. If it is greater than the preset threshold, the pixel is confirmed to be a dynamic pixel and is refreshed frame by frame, and the pixels of the previous frame are replaced with the pixels of the current frame.

[0013] To enable those skilled in the art to understand the purpose, features, and effects of the present invention, the present invention will now be described in detail through the following specific embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the prior art;

[0015] Figure 2 A flowchart of the image fusion method according to the present invention;

[0016] Figure 3 A schematic diagram illustrating an embodiment of the image fusion method according to the present invention;

[0017] Figure 4 A schematic diagram of another embodiment of the image fusion method according to the present invention; and

[0018] Figure 5A schematic diagram of another embodiment of the image fusion method according to the present invention; and

[0019] Figure 6 This is a schematic diagram of another embodiment of the image fusion method according to the present invention.

[0020] Explanation of reference numerals in the attached figures:

[0021] S100, S200, S300, S400, S500, S501, S600, S601: Steps;

[0022] O1, O2, O3, O4: Original image information;

[0023] D2, D3, D4: Determine the image;

[0024] OP1, OP2, OP3, OP4: Resulting images. Detailed Implementation

[0025] The concept of the invention will now be more fully described below with reference to the accompanying drawings, which illustrate exemplary embodiments thereof. The advantages and features of the concept of the invention, as well as methods of achieving it, will become apparent from the exemplary embodiments described in more detail below with reference to the accompanying drawings.

[0026] The terminology used herein is for illustrative purposes only and is not intended to limit the invention. Unless the context clearly indicates otherwise, the singular forms of the terms "a" and "described" as used herein are intended to include multiple forms. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It should be understood that when a component is referred to as "connected" or "coupled" to another component, the component may be directly connected or coupled to the other component or there may be intermediate components.

[0027] Exemplary embodiments are illustrated herein with reference to figures, which are idealized exemplary illustrative diagrams. Therefore, deviations from the illustrated shapes are expected due to factors such as manufacturing techniques and / or tolerances. Consequently, the areas shown in the figures are schematic and their shapes are not intended to illustrate the actual shapes of areas of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0028] Please refer to the prior art of this invention. Figure 1 , Figure 1 This is a diagram illustrating a previous technique, showing that if a high-speed moving object is present during the shooting process, it is easy to produce blurry images and ghosting.

[0029] Please refer to the image fusion method of this invention. Figure 2 , Figure 2 This is a flowchart of the image fusion method according to the present invention. It mainly includes steps S100 to S601, wherein step S400 is a judgment process.

[0030] Specifically, step S100 is to obtain image information; step S200 is to obtain the brightness value of each pixel; step S300 is to calculate the absolute value of the brightness difference with the previous frame or the sum of the brightness differences of the previous few frames; step S400 is to determine whether the calculation result is less than or equal to a preset threshold; if so, proceed to step S500 to determine that it is a static pixel; step S501 is to inherit frame by frame; otherwise, proceed to step S600 to determine that it is a dynamic pixel; step S601 is to refresh frame by frame.

[0031] Specifically, in step S100, the present invention is mainly based on the imaging principle of an image sensor (CMOS Image Sensor, CIS), using a camera lens to capture images frame by frame in a multi-frame short exposure method, thereby obtaining image information. The image information contains images of multiple frames, and the images have a sequential order in time.

[0032] Each obtained frame contains several pixels, or can be said to be composed of several pixels. In each frame, there are corresponding pixels. For example, each frame is composed of 2x2 pixels. Therefore, the pixel in the upper left corner of each frame is the corresponding pixel. That is, the pixels at the same position in each frame are corresponding pixels. Defining what constitutes a corresponding pixel is helpful for subsequent steps.

[0033] Specifically, in step S200, a processor is used to calculate the brightness value of each pixel in the current frame of the image information obtained in step S100. For example, if each frame of the image is composed of 2x2 pixels, the brightness value of each of the four pixels will be calculated in step S200. In addition, the commonly used image specification 1920x1080 is composed of 1920x1080 pixels, and the brightness value of all 1920x1080 pixels will be calculated in step S200.

[0034] Specifically, the process of obtaining image information in step S100 is a continuous process. For example, when shooting a video consisting of 4 frames, step S100 will continuously obtain the 1st to 4th frames. After obtaining the 1st frame, step S200 will be performed to obtain the brightness value of each pixel in the 1st frame. Then, when obtaining the 2nd frame, step S200 will be performed again to obtain the brightness value of each pixel in the 2nd frame, until the 4th frame.

[0035] Specifically, in step S300, following step S200, the brightness value of each pixel is calculated. The processor compares and calculates the brightness value of each pixel in the current frame image with the brightness value of the corresponding pixel in the previous frame image to calculate the absolute value of the difference, or compares and calculates the brightness value of each corresponding pixel in the previous few frames image to calculate the sum of the absolute values ​​of the differences.

[0036] Specifically, taking the example of shooting a video consisting of 4 frames, there is no action in step S300 for the first frame because there is no pixel brightness value of the previous frame. However, starting from the second frame, the absolute value of the brightness difference will be calculated in step S300. When it comes to the third and subsequent frames, step S300 calculates the absolute value of the pixel brightness difference corresponding to the previous frame or the sum of the absolute values ​​of the pixel brightness difference corresponding to the previous frames.

[0037] Similarly, each frame of an image is composed of 2x2 pixels. Therefore, after the calculation in step S300, each frame of an image will generate the absolute value of the brightness difference of 4 pixels. The absolute value of the brightness difference of these 4 pixels is calculated by using the brightness value of the corresponding pixel in the previous frame of the image.

[0038] Specifically, in step S400, the processor compares the value calculated in step S300 with a preset threshold for judgment. The preset threshold does not have a fixed value; it can be the most suitable value after multiple tests. The most suitable value can be further interpreted as having the fewest image ghosting. It should be noted that the preset threshold usually does not exceed 5% of the pixel value. For example, if the pixel value of a pixel is 500, the preset threshold will not exceed 25.

[0039] However, the present invention does not limit the value of the preset threshold, but using the preset threshold as a judgment is still one of the features of the present invention.

[0040] After comparing the result calculated in step S300 with the preset threshold, if the result is less than or equal to the preset threshold, the process proceeds to step S500 to confirm that the pixel is a static pixel. If the result calculated in step S300 is greater than the preset threshold, the process proceeds to step S600 to confirm that the pixel is a dynamic pixel.

[0041] In step S500, since the value calculated in step S300 is less than or equal to the preset threshold, the pixel in the current frame of the image is determined to be a static pixel, and step S501 is performed to inherit the pixel frame by frame.

[0042] Specifically, step S501, which inherits frame by frame, means that in this pixel of the current frame image, since it is determined to be a static pixel, that is, no object has moved or the light and shadow have changed, this pixel of the current frame image and the corresponding pixel of the previous frame image can be regarded as the same. Therefore, the pixels of the two frames are superimposed.

[0043] It should be noted that pixels from at least one previous frame can also be superimposed, but only if it is confirmed that the pixels from at least one previous frame are also static pixels.

[0044] Since the value calculated in step S300 is greater than the preset threshold, step S600 determines that the pixel in the current frame of the image is a dynamic pixel, and therefore performs step S601 to refresh frame by frame.

[0045] Specifically, step S601, which refreshes frame by frame, means that in this pixel of the current frame image, because it is determined to be a dynamic pixel, that is, because an object has moved by or the light and shadow have changed, this pixel of the current frame image is different from the corresponding pixel of the previous frame image. Therefore, the pixel of the previous frame image is replaced with the pixel of the current frame image.

[0046] Repeat steps S100 to S601 until the last frame of the image is processed, and finally output the resulting image.

[0047] Please continue reading. Figure 2 and Figure 3 , Figure 3 This is a schematic diagram of one embodiment of the present invention, which is combined with Figure 2 and 3 More detailed explanation.

[0048] Figure 3 O1 to O4 in the image information obtained in step S100 can also be represented as the original image information obtained. In this embodiment, there are a total of 4 frames of image information obtained, namely O1 to O4. From the original image information O1 to original image information O4 of these 4 frames, it can be clearly seen that an object is moving.

[0049] Specifically, when the original image information O1 is obtained in step S100, it is the first frame image. In step S200, the brightness value of each pixel is calculated. Since it is the first frame image, it will not go through steps S300 to S601, but will directly output the result image OP1, which also means that all the pixels of this frame image have not been modified or replaced.

[0050] When the original image information O2 is obtained, it is the second frame image, that is, it already has at least one previous frame image that can be compared. Therefore, proceed to step S200, calculate the brightness value of each pixel in the original image information O2, and then proceed to step S300, calculate the absolute value of the brightness difference between each pixel in the original image information O2 and each corresponding pixel in the original image information O1.

[0051] Specifically, steps S200 to S300 can be performed in two ways: one is to calculate the brightness value of a pixel in the original image information O2, that is, to calculate the difference between the brightness values ​​of the corresponding pixels in the original image information O1 and take the absolute value; the other is to first calculate the brightness value of each pixel in the original image information O2, and then calculate the difference between the brightness values ​​of each pixel and the corresponding pixels in the original image information O1 and take the absolute value.

[0052] After calculating the absolute value of the brightness difference between all pixels in the original image information O2 and the original image information O1, steps S400, S500, and S600 are performed. The absolute value of the brightness difference is compared with a preset threshold, and it is determined whether each pixel in the original image information O2 belongs to a dynamic or static pixel. The comparison and determination results are as follows: Figure 3 Judging from the image D2, from Figure 3 As you can see, if a pixel is white, it represents a static pixel, meaning the absolute value of the brightness difference is less than or equal to the preset threshold. If a pixel is black, it represents a dynamic pixel, meaning the absolute value of the brightness difference is greater than or equal to the preset threshold.

[0053] Next, steps S501 and S601 are performed. When a pixel is determined to be a moving pixel, the pixel of the original image information O1 is replaced with the pixel of the original image information O2. When a pixel is determined to be a static pixel, the pixels of the two original image information O1 and original image information O2 are superimposed. After steps S501 and S601, the result image OP2 can be obtained. Figure 3 It can be seen that the original image information O2 and the resulting image OP2 are not the same because some pixels have been replaced.

[0054] Specifically, by repeating steps S100 to S601, the original image information O3 and original image information O4 are calculated, compared and judged to obtain the result image OP3 and result image OP4.

[0055] When the original image information O3 is obtained, it is the third frame image, and there is already at least one previous frame image that can be compared. Therefore, the process proceeds to step S200, where the brightness value of each pixel in the original image information O3 is calculated. Then, the process proceeds to step S300, where the absolute value of the brightness difference between each pixel in the original image information O3 and the corresponding pixel in the original image information O2 is calculated.

[0056] When there are two or more comparable images, the calculation in step S300 can be performed in two ways. One is to calculate the absolute value of the brightness difference between each pixel in the original image information O3 and each corresponding pixel in the original image information O2, as described above. The other is to calculate the sum of the absolute values ​​of the brightness differences of the previous few frames. In this embodiment, the absolute value of the brightness difference between each pixel in the original image information O3 and each corresponding pixel in the original image information O2 is added to the absolute value of the brightness difference between each pixel in the original image information O2 and each corresponding pixel in the original image information O1. If expressed by a simple formula, it is Diff3=|diff(2,3)|+|diff(1,2)|.

[0057] Similarly, when the original image information O4 is obtained, the calculation in step S300 can be Diff4 = |diff(3,4)| or Diff4 = |diff(3,4)| + |diff(2,3)| + |diff(1,2)|.

[0058] However, the calculation method selected in step S300 must be consistent. Different methods cannot be used for different frames in the same image processing process. Furthermore, when different calculation methods are selected, the preset threshold will also change accordingly.

[0059] It should be noted that, for ease of explanation and description, the present invention breaks down the process into steps S100 to S601, which may lead to the misunderstanding that the original image information O1 to O4 and the result images OP1 to OP4 are single frames. However, in reality, they are all continuous images. As mentioned in the prior art, the original image information O1 to O4 will exhibit ghosting, while the result images OP1 to OP4 after the image fusion method of the present invention will not exhibit ghosting.

[0060] Furthermore, in steps S501 and S601, if a pixel is determined to be static, it will be superimposed; if a pixel is determined to be dynamic, it will be replaced. This will cause the brightness values ​​of the pixels in the resulting images OP1 to OP4 to be different from those in the original image information O1 to O4. Pixel superposition will increase the brightness. Therefore, if some pixels are always determined to be static pixels, they will be superimposed, making these pixels appear particularly bright in the resulting images OP1 to OP4, resulting in contrast distortion.

[0061] Therefore, in step S601, in addition to replacing the pixels of the previous frame with the pixels of the current frame, the image can be further multiplied by a gain value to make up for the brightness.

[0062] In addition, when determining whether a pixel is a static or dynamic pixel, in addition to calculating the absolute value of the brightness difference between the current frame image pixel and the previous frame image pixel as described above, the present invention may also omit the absolute value, and the comparison rule becomes: if the negative preset threshold ≤ brightness difference ≤ positive default threshold, it is a static pixel; if the brightness difference < negative preset threshold or brightness difference > positive default threshold, it is a dynamic pixel.

[0063] Please continue reading. Figure 4 , Figure 4 This illustrates another method used by the processor in the image fusion method of this invention to determine whether a pixel is a static or dynamic pixel. It employs Block Matching, a motion estimation method. The previous frame (fn-1) is divided into many modules (f'). Each module (f') is then matched against the current frame, and the module (f) with the highest similarity is found. Finally, based on the matching position, the motion vector of that module can be determined. Figure 4 (dx, dy). After finding the movement of all modules, the final estimation result of the movement of all modules is obtained. Applying this method to this invention means replacing the modules with pixels, matching the pixel f' in the previous frame fn-1 with the current frame, and finding the pixel f with the highest similarity.

[0064] Specifically, the most common module matching method is the three-step search, a simple algorithm that effectively reduces the computational cost in motion estimation. When starting the module matching calculation, only nine modules are selected. The algorithm calculates and identifies the modules with the highest similarity among these nine. Then, using this selected module as the center, nine more modules are selected. It's important to note that the distance between these nine modules is only half that of the first selection. After the second calculation, a third calculation is performed in the same manner to obtain the final matching result. Figure 5 As shown.

[0065] Figure 5 Point 1 represents the first 9 selected modules. Using the module at coordinate (0,0) as the reference, we calculate and find the module with the highest similarity. Let's assume the module with coordinate (4,4) is the most similar. In the second round, we select 9 modules centered on the module at coordinate (4,4), which is point 2. We calculate and find the module with coordinate (2,6) that is the most similar. Finally, we find the module with coordinate (2,7) as the module with the highest similarity to the module at coordinate (0,0).

[0066] Specifically, the matching calculation function can use the mean difference or the mean absolute difference. or mean square error Where N is the size of the pixel, and Cij and Rij are the current pixel and the pixel being compared in the previous frame, respectively.

[0067] Please continue to refer to the following. Figure 6 , Figure 6 This illustrates another method used by the processor to determine whether a pixel is static or dynamic in the image fusion method of this invention. In this embodiment, optical flow, a motion estimation method, is used. It is based on the gradient changes of each pixel along the horizontal, vertical, and time axes. Since pixels located near each other will have similar or identical movements, the movement change of a particular pixel can be calculated. Figure 6 At time t, a pixel is located at (x, y), and at time t + dt, the pixel's location moves to (x + dx, y + dy). By repeatedly calculating the movement of all pixels in the image, the movement vector estimation of all pixels can be obtained.

[0068] Specifically, this invention employs the sparse optical flow method, which is a method specifically designed for image matching of points in an image. In other words, given a point, it finds its corresponding point in the current image. Mathematically, given a reference image T and the current image I, it calculates the point Q(xp+u,yp+v) in the current image I corresponding to a point P(xp,yp) in the reference image T, where (u,v) is the offset of the point. If all points within a certain range centered on points P and Q are identical, then these two points are considered a match, thus revealing the movement of a point and achieving the purpose of tracking.

[0069] After determining whether a pixel is a static or dynamic pixel using the aforementioned motion estimation method, if it is a static pixel, it is superimposed; if it is a dynamic pixel, it is replaced. The processor then outputs the resulting image.

[0070] Finally, the technical features of this invention and the technical effects it can achieve are summarized as follows:

[0071] Firstly, through the image fusion method of the present invention, each frame of the image information simultaneously presents clear dynamic objects and static scenes.

[0072] Secondly, the image fusion method of the present invention, based on the imaging principle of the image sensor, determines whether the pixel is dynamic or static by the difference in light intensity change of the pixel at the same position in the previous and subsequent frames, retains the movement trajectory while suppressing background noise, and finally fuses to present a complete single-frame image, so that both dynamic objects and static scenes in the image have clear imaging.

[0073] Third, the image fusion method of the present invention has a fast response speed, low latency, small data processing volume at high frame rates, and low power consumption.

[0074] The above description illustrates the implementation of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention; any equivalent changes or modifications made without departing from the spirit disclosed in the present invention should be included within the scope of the following patent.

Claims

1. An image fusion method, characterized in that, Include: Image information is obtained by capturing images frame by frame using a camera lens. The image information includes multiple frames of images, which are sequential in time, and each image is composed of multiple pixels. A processor is used to calculate the brightness value of each pixel contained in the image of the current frame in the image information; The processor calculates the absolute value of the brightness difference between the pixels corresponding to the previous frame image; The processor compares the absolute value of the brightness difference with a preset threshold. If the pixel is less than or equal to the preset threshold, it is determined to be a static pixel, and the pixel of at least one previous frame image is compared with the pixel of the current frame image; as well as If the value is greater than the preset threshold, the pixel is determined to be a dynamic pixel, and the pixel of the current frame image replaces the pixel of the previous frame image. The processor generates the resulting image output; When the processor determines that a pixel is dynamic and replaces the pixel of the previous frame image with the pixel of the current frame image, the processor further multiplies it by a gain value during the calculation.

2. The image fusion method according to claim 1, characterized in that, When the processor overlays pixels from at least one previous frame, all pixels from at least one previous frame must be determined as static pixels.

3. An image fusion method, characterized in that, Include: Image information is obtained by capturing images frame by frame using a camera lens. The image information includes multiple frames of images, which are sequential in time, and each image is composed of multiple pixels. A processor is used to determine whether the pixels in the current frame of the image are static pixels using a module matching method. If it is a static pixel, superimpose the pixels of at least one previous frame onto the pixels of the current frame; and If it's dynamic pixels, replace the pixels of the previous frame with the pixels of the current frame; and Output the resulting image; When the processor determines that a pixel is a dynamic pixel, it further multiplies the pixel by a gain value during the calculation.

4. An image fusion method, characterized in that, Include: Image information is obtained by capturing images frame by frame using a camera lens. The image information includes multiple frames of images, which are sequential in time, and each image is composed of multiple pixels. A processor is used to determine whether the pixels in the current frame of the image are static pixels using optical flow. If it is a static pixel, superimpose the pixels of at least one previous frame onto the pixels of the current frame; and If it's dynamic pixels, replace the pixels of the previous frame with the pixels of the current frame; and Output the resulting image; When the processor determines that a pixel is a dynamic pixel, it further multiplies the pixel by a gain value during the calculation.

5. The image fusion method according to claim 3 or 4, characterized in that, When the processor overlays pixels from at least one previous frame, all pixels from at least one previous frame must be determined as static pixels.

Images