Image fusion method, electronic device, and storage medium

Abstract

Embodiments of the present application disclose an image fusion method, an electronic device and a storage medium, including: performing Laplacian pyramid decomposition on each image to be fused; determining fusion weights of other decomposition images of a highest layer according to frame differences between a reference decomposition image of the highest layer and other decomposition images of the highest layer; the reference decomposition image being a decomposition image of the image to be fused with the highest definition; determining fusion weights of other decomposition images of a second highest layer according to frame differences between a reference decomposition image of the second highest layer and other decomposition images of the second highest layer and the fusion weights of the other decomposition images of the highest layer; performing the above determining process on remaining layers until fusion weights of other decomposition images of a lowest layer are obtained; fusing the decomposition images of each layer based on the fusion weights of the decomposition images of each layer to obtain intermediate fusion images of each layer; the fusion weights of motion regions in the other decomposition images of each layer being 0, and the fusion weights of motion regions in the reference decomposition images of each layer being 1; and generating a target image based on the intermediate fusion images of each layer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image fusion method, an electronic device and a storage medium. BACKGROUND

[0002] At present, when using an electronic device to take a picture, due to the hardware level of the device, it is usually necessary to take multiple images and fuse the multiple images to synthesize an image with lower noise level, higher dynamic range and higher definition. Whether it is a handheld device or a fixed camera, the photographed object will move. In the related art, when synthesizing an image, the moving area of the image will produce motion blur, ghosting and other problems, and the visual effect of the image fusion result is poor. SUMMARY

[0003] The embodiments of the present application provide an image fusion method, an electronic device and a storage medium to solve the technical problem of poor visual effect of the image fusion result in the related art.

[0004] According to a first aspect of the present application, an image fusion method is disclosed, the method comprising:

[0005] After obtaining multiple to-be-fused images taken continuously, performing Laplacian pyramid decomposition on each to-be-fused image to obtain multiple layers of decomposition images with different resolutions;

[0006] determining the fusion weight of the other decomposition images of the highest layer according to the frame difference between the reference decomposition image of the highest layer and the other decomposition images of the highest layer; the reference decomposition image is the decomposition image of the to-be-fused image with the highest definition, and the other decomposition images are the decomposition images other than the reference decomposition image;

[0007] determining the fusion weight of the other decomposition images of the second highest layer according to the frame difference between the reference decomposition image of the second highest layer and the other decomposition images of the second highest layer and the fusion weight of the other decomposition images of the highest layer; performing the above determination process on the remaining layers until the fusion weight of the other decomposition images of the lowest layer is obtained;

[0008] fusing the decomposition images of each layer based on the fusion weight of the decomposition images of each layer to obtain intermediate fusion images of each layer; the fusion weight of the motion area in the other decomposition images of each layer is 0, and the fusion weight of the motion area in the reference decomposition image of each layer is 1;

[0009] generating a target image based on the intermediate fusion images of each layer.

[0010] According to a second aspect of the present application, an electronic device is disclosed, comprising a memory, a processor and a computer program stored in the memory, the processor executes the computer program to realize the image fusion method as in the first aspect.

[0011] According to a third aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the image fusion method as described in the first aspect.

[0012] According to a fourth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the image fusion method as described in the first aspect.

[0013] In this embodiment, after acquiring multiple continuously captured images to be fused, each image to be fused is decomposed using a Laplacian pyramid to obtain multi-layer decomposed images of different resolutions. The fusion weights of the other decomposed images at the highest level are determined based on the frame difference between the highest-level reference decomposed image and other decomposed images at the highest level. The reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are decomposed images other than the reference decomposed image. The fusion weights of the other decomposed images at the second-highest level are determined based on the frame difference between the second-highest reference decomposed image and other decomposed images at the second-highest level, as well as the fusion weights of the other decomposed images at the highest level. The above determination process is performed on the remaining layers until the fusion weights of the other decomposed images at the lowest level are obtained. Based on the fusion weights of each layer's decomposed images, each layer's decomposed images are fused to obtain intermediate fused images for each layer. The fusion weight of the moving regions in each layer's other decomposed images is 0, and the fusion weight of the moving regions in each layer's reference decomposed images is 1. The target image is generated based on the intermediate fused images of each layer.

[0014] As can be seen, in this embodiment, after decomposing multiple consecutively captured images into a Laplacian pyramid, starting from the highest level of the Laplacian pyramid, the fusion weight of the highest-level decomposed image is calculated based on the frame difference between the decomposed images. This highest-level fusion weight is then passed down to the next level to influence the calculation of the fusion weight of the next level's decomposed image. Since motion detection in noisy images (i.e., higher-level images) within the Laplacian pyramid can often largely avoid the influence of noise on motion detection, motion detection in even larger-scale images may miss some smaller moving objects. Therefore, within the Laplacian pyramid, the fusion weight is calculated starting from the largest-scale image. The system begins detecting moving regions and calculates fusion weights based on frame differences that characterize motion intensity. These fusion weights are then passed down level by level to influence the fusion weight calculations of smaller-scale decomposed images. This ensures accurate detection of large moving objects while also preventing the omission of small moving objects, thus guaranteeing the accuracy of moving region detection. Furthermore, when fusing each layer of the Laplacian pyramid decomposed images based on the fusion weights, the moving regions are directly replaced with pixels from the decomposed image with the highest resolution. This avoids motion blur and ghosting in the moving regions of the fused results, generating a fused image that better matches human vision and improving the visual effect of the image fusion result. Attached Figure Description

[0015] Figure 1 This is a flowchart of an image fusion method provided in an embodiment of this application;

[0016] Figure 2 This is one of the example diagrams of an image fusion method provided in the embodiments of this application;

[0017] Figure 3 This is a second example diagram of an image fusion method provided in an embodiment of this application;

[0018] Figure 4 This is a schematic diagram of the structure of an image fusion device provided in an embodiment of this application;

[0019] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0022] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0023] Taking image fusion in the field of image processing as an example, current machine vision technology is heavily limited by hardware, often requiring the continuous capture of multiple images to synthesize higher-quality images with lower noise levels, higher dynamic range, and higher clarity. Whether using handheld devices or fixed cameras, situations arise where the subject is moving. Related technologies employ full-resolution fusion techniques; however, because full-resolution multi-frame image fusion is easily affected by noise, it is difficult to accurately detect moving areas in the image, leading to problems such as motion blur and ghosting during image fusion, resulting in poor visual quality of the fused image.

[0024] To address the aforementioned technical problems, embodiments of this application provide an image fusion method, an electronic device, and a storage medium, which can improve the robustness of multi-frame fusion algorithms in motion scenes. For ease of understanding, the application scenarios and some related concepts of the embodiments of this application are first introduced below.

[0025] Image pyramids, a type of multi-scale representation of images, are primarily used for image segmentation. They are an efficient yet conceptually simple structure for interpreting images at multiple resolutions. Originally used in machine vision and image compression, an image pyramid is a set of images derived from the same original image, arranged in a pyramid shape with progressively decreasing resolution. Sampling is achieved through stepwise downsampling until a certain termination condition is met. The base of the pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation.

[0026] Levels, scales, sizes, and resolutions: For images at each level in the pyramid, the higher the level, the larger the image scale, the smaller the image size, and the lower the resolution; the lower the level, the smaller the image scale, the larger the image size, and the higher the resolution.

[0027] The Gaussian pyramid is a primary image pyramid used for downsampling.

[0028] The Laplacian pyramid is used to reconstruct an upper-level unsampled image from a lower-level image. In digital image processing, this is equivalent to predicting residuals. It can restore an image to the greatest extent possible and is used in conjunction with the Gaussian pyramid.

[0029] The following section describes an image fusion method provided by an embodiment of this application.

[0030] Figure 1 This is a flowchart of an image fusion method provided in an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps: step 101, step 102, step 103, step 104 and step 105;

[0031] In step 101, after acquiring multiple images to be fused in succession, each image to be fused is decomposed using the Laplacian pyramid to obtain multi-layer decomposed images of different resolutions.

[0032] In this embodiment, the multiple images to be fused can be a series of images captured by mobile devices such as smartphones and drones, or a series of images captured by network cameras.

[0033] In this embodiment of the application, the image to be fused can be a grayscale image or a color image.

[0034] In this embodiment, the same Laplacian pyramid decomposition is performed on each image to be fused to obtain multi-layer decomposed images of different resolutions (i.e. different scales) for each image to be fused.

[0035] In this embodiment of the application, the number of levels of the Laplacian pyramid decomposition can be selected according to the actual needs of the user. For example, the image can be decomposed into 4 levels. The higher the level of the image, the larger the scale of the image, the smaller the size of the image, and the lower the resolution.

[0036] In some embodiments, when the images to be fused are color images, the multi-color channel color images can be converted into grayscale images, and the number of channels of the image is reduced from multiple to one. This can reduce the amount of data processed during image processing. Accordingly, step 101 above may include the following steps: performing Laplacian pyramid decomposition on the Y channel images corresponding to each image to be fused to obtain multi-layer decomposed images of different resolutions, wherein the decomposed images are Y channel decomposed images.

[0037] In this embodiment of the application, when the image to be fused is a color image, the image to be fused can first be converted to a color space, converted to Ycbcr format or YUV format, to obtain the Y channel image (i.e. grayscale image) corresponding to the image to be fused. Then, based on the Y channel decomposition of each layer in the Laplacian pyramid, steps 102 to 105 are executed.

[0038] In step 102, the fusion weights of the other decomposed images at the highest level are determined based on the frame difference between the highest level reference decomposed image and the other decomposed images at the highest level; wherein, the reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are the decomposed images other than the reference decomposed image.

[0039] In this embodiment of the application, the fusion weight of the decomposed image includes: the fusion weight of each pixel position in the decomposed image, wherein the fusion weight of different pixel positions may be the same or different.

[0040] In this embodiment, considering that motion blur and ghosting are likely to occur in the fusion results, mainly in motion regions, and that motion features of moving objects of different sizes differ in images at different scales (e.g., large moving objects show more obvious motion features in large-scale / high-level images, while small moving objects show more obvious motion features in small-scale / low-level images), the motion regions in each decomposed image within the layer are first determined starting from the highest layer of the Laplacian pyramid. When determining motion regions, one decomposed image is used as a reference decomposed image, and the frame difference between other decomposed images and the reference decomposed image is calculated. The motion regions in other decomposed images are determined based on the frame difference. Since the reference decomposed image is the decomposed image of the image to be fused with the highest clarity (i.e., the reference decomposed image is the decomposed image with the highest clarity in this layer), and the frame difference can reflect the similarity of the same pixel positions in two images (the larger the frame difference, the lower the similarity; the smaller the frame difference, the higher the similarity), and the similarity is strongly correlated with the region type (the lower the similarity, the higher the probability of belonging to a motion region; the higher the similarity, the higher the probability of belonging to a stationary region), the accuracy of the motion region determination results of other decomposed images in this layer can be ensured.

[0041] After determining the motion regions of other decomposed images at the highest level, the fusion weight of the motion regions of other decomposed images at the highest level is set to 0, while the fusion weight of the motion region of the reference decomposed image at the highest level is set to 1. This ensures that when fusing the decomposed images at the highest level, the pixels in the motion regions of the fused results are the same as the pixels in the motion regions of the reference decomposed image at the highest level, thus avoiding motion blur and ghosting in the motion regions of the fused results. For stationary regions outside the motion regions, the fusion weight of the stationary regions is determined based on the frame difference between the reference decomposed image at the highest level and other decomposed images at the highest level. This ensures that when fusing the decomposed images at the highest level, the pixels in the stationary regions of the fused results are the sum of the pixels in all decomposed images at the highest level, thus ensuring comprehensive pixel information in the stationary regions of the fused results and improving the visual effect of image fusion.

[0042] In this embodiment, the fusion weight of the moving regions in the other decomposed images of the highest layer is 0, while the fusion weight of the moving regions in the highest-layer reference decomposed image is 1. Furthermore, considering that the highest-layer reference decomposed image has the highest sharpness at the highest level of the Laplacian pyramid, to improve the visual effect of the fusion result, the fusion weight of the stationary regions outside the moving regions in the highest-layer reference decomposed image can also be set to 1. That is, the fusion weight of the highest-layer reference decomposed image is 1. By assigning a higher fusion weight to the highest-layer reference decomposed image, the visual effect of the fusion result can be improved.

[0043] In step 103, the fusion weights of the other decomposed images in the second-highest layer are determined based on the frame difference between the second-highest layer reference decomposed image and the other decomposed images in the second-highest layer, as well as the fusion weights of the other decomposed images in the highest layer. The above determination process is performed on the remaining layers until the fusion weights of the other decomposed images in the lowest layer are obtained.

[0044] In this embodiment of the application, for the Laplacian pyramid, since the highest layer contains large-scale features of the image, its frame difference can determine the motion information of objects at a larger scale in the image. If there is a motion region at this scale, it is likely that there is also a motion region at a smaller scale. Therefore, the fusion weights of other decomposed images at the highest layer can be passed to the second-highest layer to help other decomposed images at the second-highest layer determine the motion region, and then determine the fusion weights of other decomposed images at the second-highest layer.

[0045] In this embodiment of the application, for each layer below the second-highest level of the Laplacian pyramid, the above-described process of fusion weight transfer and calculation is used to obtain the fusion weights of other decomposed images in each layer below the second-highest level.

[0046] In this embodiment, the fusion weight of the motion region in other decomposed images of the second-highest level is 0, and the fusion weight of the motion region in the reference decomposed image of the second-highest level is 1.

[0047] In this embodiment, the fusion weight of the motion region in other decomposed images below the second-highest level is 0, and the fusion weight of the motion region in the reference decomposed images below the second-highest level is 1.

[0048] In other words, the fusion weight of the moving regions in other decomposed images within each layer of the Laplacian pyramid is 0, while the fusion weight of the moving regions in the reference decomposed images of each layer is 1. This ensures that the pixels of the moving regions in the fusion result of each layer are the pixels of the moving regions in the reference decomposed images of that layer, thereby avoiding motion blur and ghosting in the moving regions of the fusion result. Furthermore, it ensures that the pixels of the stationary regions in the fusion result of each layer are the sum of all decomposed images of that layer, thereby ensuring the comprehensiveness of pixel information in the stationary regions of the fusion result and improving the visual effect of image fusion.

[0049] In this embodiment of the application, considering that the clarity of the reference decomposition image is the highest in each layer of the Laplacian pyramid, in order to improve the visual effect of the fusion result, the fusion weight of the static region outside the moving region in the reference decomposition image can also be set to 1. That is, the fusion weight of each layer of reference decomposition image is 1. By assigning a higher fusion weight to each layer of reference decomposition image, the visual effect of the fusion result can be improved.

[0050] In step 104, based on the fusion weights of each layer of decomposed images, the decomposed images of each layer are fused to obtain intermediate fused images of each layer.

[0051] In step 105, the target image is generated based on the intermediate fused images of each layer.

[0052] In this embodiment, Laplacian pyramid reconstruction is performed on the intermediate fused images of all layers to obtain the final target image.

[0053] In one example, there are three images to be fused, taken consecutively: image A, image B, and image C. Among them, image B has the highest resolution.

[0054] like Figure 2 As shown, Laplacian pyramid decomposition is performed on image A, resulting in three layers: decomposed image A2, decomposed image A1, and decomposed image A0. Laplacian pyramid decomposition is also performed on image B, resulting in three layers: decomposed image B2, decomposed image B1, and decomposed image B0. Laplacian pyramid decomposition is also performed on image C, resulting in three layers: decomposed image C2, decomposed image C1, and decomposed image C0. Since image B has the highest sharpness, decomposed images B2, B1, and B0 of image B are all reference decomposed images.

[0055] Starting from the highest level of the Laplacian pyramid, which includes three decomposed images: decomposed image A2, reference decomposed image B2, and decomposed image C2, the fusion weight w of decomposed image A2 is determined based on the frame difference between decomposed image A2 and reference decomposed image B2. A2 Based on the frame difference between the decomposed image C2 and the reference decomposed image B2, the fusion weight w of the decomposed image C2 is determined. C2 Based on the calculated fusion weight w A2 and w C2 The decomposed image A2, the reference decomposed image B2, and the decomposed image C2 are fused to obtain the highest-level intermediate fused image D2, where D2 = A2 * w A2 +B2+C2*w C2 .

[0056] After completing the calculation of the highest-level fusion weights, the highest-level fusion weights w A2 and w C2 The data is passed to the next higher layer for calculation of its fusion weights. The next higher layer consists of three decomposed images: decomposed image A1, reference decomposed image B1, and decomposed image C1. The fusion weights are calculated based on the frame difference between decomposed image A1 and reference decomposed image B1, and the fusion weights w from the previous layer. A2 Determine the fusion weight w of the decomposed image A1. A1 Based on the frame difference between the decomposed image C1 and the reference decomposed image B1, and the fusion weights w from the previous layer... C2 Determine the fusion weights w of the decomposed image C1. C1 Based on the calculated fusion weight wA1 and w C1 The decomposed image A1, the reference decomposed image B1, and the decomposed image C1 are fused to obtain the intermediate fused image D1 of the second-highest level, where D1 = A1 * w A1 +B1+C1*w C1 Similarly, the lowest-level intermediate fused image D0 can be calculated, where D0 = A0 * w A0 +B0+C0*w C0 .

[0057] Finally, Laplacian pyramid reconstruction is performed based on the intermediate fused images D2, D1, and D0 to generate the target image.

[0058] As can be seen from the above embodiments, in this embodiment, after acquiring multiple images to be fused in succession, each image to be fused is decomposed using a Laplacian pyramid to obtain multi-layer decomposed images of different resolutions; the fusion weights of the other decomposed images at the highest level are determined based on the frame difference between the highest-level reference decomposed image and other decomposed images at the highest level; the reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are decomposed images other than the reference decomposed image; the fusion weights of the other decomposed images at the second-highest level are determined based on the frame difference between the second-highest reference decomposed image and other decomposed images at the second-highest level, as well as the fusion weights of the other decomposed images at the highest level; the above determination process is performed on the remaining layers until the fusion weights of the other decomposed images at the lowest level are obtained; based on the fusion weights of each layer of decomposed images, each layer of decomposed images is fused to obtain the intermediate fused images of each layer; the fusion weight of the moving regions in each layer of other decomposed images is 0, and the fusion weight of the moving regions in each layer of reference decomposed images is 1; the target image is generated based on the intermediate fused images of each layer.

[0059] As can be seen, in this embodiment, after decomposing multiple consecutively captured images into a Laplacian pyramid, starting from the highest level of the Laplacian pyramid, the fusion weight of the highest-level decomposed image is calculated based on the frame difference between the decomposed images. This highest-level fusion weight is then passed down to the next level to influence the calculation of the fusion weight of the next level's decomposed image. Since motion detection in noisy images (i.e., higher-level images) within the Laplacian pyramid can often largely avoid the influence of noise on motion detection, motion detection in even larger-scale images may miss some smaller moving objects. Therefore, within the Laplacian pyramid, the fusion weight is calculated starting from the largest-scale image. The system begins detecting moving regions and calculates fusion weights based on frame differences that characterize motion intensity. These fusion weights are then passed down level by level to influence the fusion weight calculations of smaller-scale decomposed images. This ensures accurate detection of large moving objects while also preventing the omission of small moving objects, thus guaranteeing the accuracy of moving region detection. Furthermore, when fusing each layer of the Laplacian pyramid decomposed images based on the fusion weights, the moving regions are directly replaced with pixels from the decomposed image with the highest resolution. This avoids motion blur and ghosting in the moving regions of the fused results, generating a fused image that better matches human vision and improving the visual effect of the image fusion result.

[0060] In another embodiment provided in this application, as a frame difference parameter, the absolute difference of pixel values ​​of two pixels at the same position in two images can accurately measure the similarity between the two pixels at the same position. Therefore, the fusion weight can be determined based on the absolute difference of pixel values. Accordingly, the above step 102 may include the following steps: step 1021 and step 1022.

[0061] In step 1021, the absolute difference between the pixel value at each pixel position in the highest-level reference decomposed image and the pixel value at the corresponding position in other highest-level decomposed images is calculated.

[0062] In this embodiment, the absolute difference between the pixel value of each pixel position in the highest-level reference decomposed image and the corresponding pixel value in other highest-level decomposed images can be calculated pixel by pixel to obtain the absolute difference corresponding to each pixel position in other highest-level decomposed images. The larger the absolute difference of a pixel position, the lower the similarity between other decomposed images and the reference decomposed image at that pixel position, and the higher the probability that the pixel position in other decomposed images belongs to a moving region. The smaller the absolute difference of a pixel position, the higher the similarity between other decomposed images and the reference decomposed image at that pixel position, and the higher the probability that the pixel position in other decomposed images belongs to a stationary region.

[0063] In step 1022, the fusion weight of each pixel position in the other decomposed images of the highest layer is determined based on the absolute difference corresponding to each pixel position. Pixel positions with an absolute difference greater than or equal to a first threshold constitute the motion region.

[0064] In some embodiments, for pixel locations in other decomposed images of the highest layer where the absolute difference is greater than or equal to a first threshold, the pixel location is considered a moving region, and the fusion weight for that pixel location is determined to be 0; for pixel locations in other decomposed images of the highest layer where the absolute difference is less than the first threshold, the pixel location is considered a stationary region, and the fusion weight for that pixel location is determined to be... Where Δ1 is the absolute difference and δ1 is the first parameter.

[0065] As can be seen, in this embodiment, the fusion weight of each pixel position of the highest-level reference decomposed image can be determined based on the absolute difference between the pixel value of each pixel position of the highest-level reference decomposed image and the pixel value of the corresponding position of the other decomposed images of the highest level. Since the calculation of the absolute difference is relatively small and the calculation result is relatively stable, the process of determining the fusion weight is relatively fast and the determination result is relatively accurate.

[0066] In another embodiment provided in this application, step 103 may include the following steps: step 1031, step 1032 and step 1033;

[0067] In step 1031, the weight coefficients of each pixel position in the other decomposed images of the second-highest level are determined based on the frame difference between the second-highest level reference decomposed image and the other decomposed images of the second-highest level.

[0068] In this embodiment of the application, the process of determining the weight coefficients of each pixel position in the other decomposed images of the second-highest level is similar to the process of determining the fusion weights of each pixel position in the other decomposed images of the highest level in step 102, and will not be described again here.

[0069] In step 1032, the fusion weights of the other decomposed images at the highest level are upsampled to the same size as the decomposed images at the second-highest level.

[0070] In this embodiment, since the fusion weights of the other decomposed images at the highest level are the same size as the highest level decomposed image and smaller than the size of the second-highest level decomposed image, the fusion weights of the other decomposed images at the highest level are upsampled to the same size as the second-highest level decomposed image for easier processing.

[0071] In step 1033, the fusion weights of each pixel position in the other decomposed images of the second-highest level are determined based on the weight coefficients of each pixel position in the other decomposed images of the second-highest level and the fusion weights of the corresponding positions obtained by upsampling.

[0072] In some embodiments, the weight coefficients of each pixel position in the other decomposed images of the second-highest level can be multiplied with the fusion weights of the corresponding positions obtained by upsampling to obtain the fusion weights of each pixel position in the other decomposed images of the second-highest level.

[0073] As can be seen, in the embodiments of this application, for the Laplacian pyramid, since the highest layer is a feature of the image at a large scale, its frame difference can determine the motion information of objects at a larger scale in the image. If there is a motion region at this scale, it is likely that there is also a motion region at a smaller scale. Therefore, the fusion weights of other decomposed images at the highest level are passed to the second-highest level to help other decomposed images at the second-highest level determine the motion region, and the fusion weights of other decomposed images at the second-highest level can be accurately determined.

[0074] To facilitate understanding of the technical solutions in the embodiments of this application, taking an RGB image as an example, and combining... Figure 3 The example shown is used for description.

[0075] Figure 3 This is a second example diagram of an image fusion method provided in the embodiments of this application, which includes the following steps: step 301, step 302, step 303, step 304, step 305, step 306, step 307, step 308 and step 309;

[0076] In step 301, multiple RGB images to be fused are acquired in succession.

[0077] In step 302, each RGB image to be fused is converted to the YCC color space to obtain the corresponding Y channel image.

[0078] In step 303, Laplacian pyramid decomposition is performed on each Y-channel image to obtain multi-layer Y-channel decomposed images.

[0079] In step 304, starting from the highest level of the Laplacian pyramid, the frame difference between the Y-channel decomposed images is calculated, and the fusion weight of the highest level Y-channel decomposed image is calculated based on the frame difference.

[0080] In step 305, the fusion weight of the highest layer is passed to the next highest layer.

[0081] In step 306, the frame difference between the Y-channel decomposed images of the second-highest layer is calculated, and the fusion weight of the Y-channel decomposed images of the second-highest layer is calculated based on the frame difference. The fusion weight passed down from the highest layer is amplified to the image size corresponding to the second-highest layer and multiplied with the fusion weight of the second-highest layer to obtain the final fusion weight of the second-highest layer.

[0082] In step 307, the above process is repeated until all levels of the pyramid have been traversed.

[0083] In step 308, the fusion result of each level is calculated based on the Y-channel decomposed image of each level and the corresponding fusion weight.

[0084] In step 309, Laplacian pyramid reconstruction is performed based on the fusion results of all levels to obtain the target image.

[0085] As can be seen, in the embodiments of this application, motion blur and ghosting in the motion region of the fusion result can be avoided, thereby generating a fused image that is more in line with human visual perception and improving the visual effect of the image fusion result.

[0086] Figure 4 This is a schematic diagram of the structure of an image fusion device provided in an embodiment of this application, as shown below. Figure 4 As shown, the image fusion device 400 may include: a decomposition module 401, a first determination module 402, a second determination module 403, a fusion module 404, and a generation module 405;

[0087] The decomposition module 401 is used to decompose each of the images to be fused into a Laplacian pyramid after acquiring multiple images to be fused in succession, so as to obtain multi-layer decomposed images of different resolutions.

[0088] The first determining module 402 is used to determine the fusion weights of other decomposed images at the highest level based on the frame difference between the highest level reference decomposed image and other decomposed images at the highest level; the reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are decomposed images other than the reference decomposed image;

[0089] The second determining module 403 is used to determine the fusion weight of other decomposed images in the second-highest layer based on the frame difference between the second-highest layer reference decomposed image and other decomposed images in the second-highest layer and the fusion weight of other decomposed images in the highest layer; and to perform the above determining process on the remaining layers until the fusion weight of other decomposed images in the lowest layer is obtained.

[0090] The fusion module 404 is used to fuse the decomposed images of each layer based on the fusion weight of each layer to obtain the intermediate fused image of each layer; the fusion weight of the moving region in other decomposed images of each layer is 0, and the fusion weight of the moving region in the reference decomposed image of each layer is 1.

[0091] The generation module 405 is used to generate a target image based on the intermediate fused images of each layer.

[0092] As can be seen from the above embodiments, in this embodiment, after acquiring multiple images to be fused in succession, each image to be fused is decomposed using a Laplacian pyramid to obtain multi-layer decomposed images of different resolutions; the fusion weights of the other decomposed images at the highest level are determined based on the frame difference between the highest-level reference decomposed image and other decomposed images at the highest level; the reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are decomposed images other than the reference decomposed image; the fusion weights of the other decomposed images at the second-highest level are determined based on the frame difference between the second-highest reference decomposed image and other decomposed images at the second-highest level, as well as the fusion weights of the other decomposed images at the highest level; the above determination process is performed on the remaining layers until the fusion weights of the other decomposed images at the lowest level are obtained; based on the fusion weights of each layer of decomposed images, each layer of decomposed images is fused to obtain the intermediate fused images of each layer; the fusion weight of the moving regions in each layer of other decomposed images is 0, and the fusion weight of the moving regions in each layer of reference decomposed images is 1; the target image is generated based on the intermediate fused images of each layer.

[0093] As can be seen, in this embodiment, after decomposing multiple consecutively captured images into a Laplacian pyramid, starting from the highest level of the Laplacian pyramid, the fusion weight of the highest-level decomposed image is calculated based on the frame difference between the decomposed images. This highest-level fusion weight is then passed down to the next level to influence the calculation of the fusion weight of the next level's decomposed image. Since motion detection in noisy images (i.e., higher-level images) within the Laplacian pyramid can often largely avoid the influence of noise on motion detection, motion detection in even larger-scale images may miss some smaller moving objects. Therefore, within the Laplacian pyramid, the fusion weight is calculated starting from the largest-scale image. The system begins detecting moving regions and calculates fusion weights based on frame differences that characterize motion intensity. These fusion weights are then passed down level by level to influence the fusion weight calculations of smaller-scale decomposed images. This ensures accurate detection of large moving objects while also preventing the omission of small moving objects, thus guaranteeing the accuracy of moving region detection. Furthermore, when fusing each layer of the Laplacian pyramid decomposed images based on the fusion weights, the moving regions are directly replaced with pixels from the decomposed image with the highest resolution. This avoids motion blur and ghosting in the moving regions of the fused results, generating a fused image that better matches human vision and improving the visual effect of the image fusion result.

[0094] Optionally, as an embodiment, the first determining module 402 may include:

[0095] The first calculation submodule is used to calculate the absolute difference between the pixel value at each pixel position in the highest-level reference decomposed image and the pixel value at the corresponding position in other highest-level decomposed images;

[0096] The first determining submodule is used to determine the fusion weight of each pixel position of the other decomposed images of the highest layer based on the absolute difference corresponding to each pixel position of the other decomposed images of the highest layer, wherein the pixel positions with an absolute difference greater than or equal to a first threshold constitute the motion region.

[0097] Optionally, as an embodiment, the first determining submodule may include:

[0098] The first determining unit is used to determine the fusion weight of a pixel position as 0 for the pixel position in other decomposed images of the highest layer where the absolute difference is greater than or equal to a first threshold.

[0099] The second determining unit is used to determine the fusion weight of pixel positions where the absolute difference in other decomposed images of the highest layer is less than a first threshold. Where Δ1 is the absolute difference and δ1 is the first parameter.

[0100] Optionally, as an embodiment, the second determining module 403 may include:

[0101] The second determining submodule is used to determine the weight coefficients of each pixel position in the other decomposed images of the second-highest level based on the frame difference between the second-highest level reference decomposed image and the other decomposed images of the second-highest level.

[0102] The sampling submodule is used to upsample the fusion weights of the other decomposed images at the highest level to the same size as the decomposed images at the second highest level.

[0103] The third determining submodule is used to determine the fusion weight of each pixel position in the other decomposed images of the second-highest level based on the weight coefficients of each pixel position in the other decomposed images of the second-highest level and the fusion weight of the corresponding position obtained by upsampling.

[0104] Optionally, as an embodiment, the third determining submodule may include:

[0105] The computation unit is used to perform a product operation on the weight coefficients of each pixel position of the other decomposed images of the second-highest level and the fusion weights of the corresponding positions obtained by upsampling, so as to obtain the fusion weights of each pixel position of the other decomposed images of the second-highest level.

[0106] Optionally, as an embodiment, the image to be fused is a color image;

[0107] The decomposition module 401 may include:

[0108] The decomposition submodule is used to perform Laplacian pyramid decomposition on the Y-channel images corresponding to each image to be fused, so as to obtain multi-layer decomposition images of different resolutions, wherein the decomposition images are Y-channel decomposition images.

[0109] Optionally, as an embodiment, the fusion weight of each pixel position in each layer of the reference decomposed image is 1.

[0110] Any step and specific operation within any step in the embodiments of the image fusion method provided in this application can be performed by a corresponding module in the image fusion apparatus. The procedures for the corresponding operations performed by each module in the image fusion apparatus are described in the embodiments of the image fusion method.

[0111] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0112] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device includes a processing component 522, which further includes one or more processors, and memory resources represented by a memory 532 for storing instructions executable by the processing component 522, such as application programs. The application programs stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 522 is configured to execute instructions to perform the methods described above.

[0113] The electronic device may also include a power supply component 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input / output (I / O) interface 558. The electronic device may operate on an operating system stored in memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0114] According to yet another embodiment of this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the image fusion method as described in any of the above embodiments.

[0115] According to yet another embodiment of this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the image fusion method as described in any of the above embodiments.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0121] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0122] The above provides a detailed description of the image fusion method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image fusion method, characterized in that, The method includes: After acquiring multiple images to be fused in succession, each image to be fused is decomposed using the Laplacian pyramid to obtain multi-layer decomposed images of different resolutions. The fusion weights of the other decomposed images at the highest level are determined based on the frame difference between the highest-level reference decomposed image and the other decomposed images at the highest level; the reference decomposed image is the decomposed image of the image to be fused with the highest clarity, and the other decomposed images are the decomposed images other than the reference decomposed image; Based on the frame difference between the second-highest layer reference decomposed image and other second-highest layer decomposed images, and the fusion weights of the other highest layer decomposed images, the fusion weights of the other second-highest layer decomposed images are determined; the above determination process is performed on the remaining layers until the fusion weights of the other lowest layer decomposed images are obtained. Based on the fusion weights of the decomposed images at each layer, the decomposed images at each layer are fused to obtain the intermediate fused images of each layer; the fusion weights of the moving regions in other decomposed images at each layer are 0, and the fusion weights of the moving regions in the reference decomposed images at each layer are 1; Generate the target image based on the intermediate fused images from each layer; The step of determining the fusion weights of other decomposed images at the highest level based on the frame difference between the highest-level reference decomposed image and other decomposed images at the highest level includes: Calculate the absolute difference between the pixel value at each pixel position in the highest-level reference decomposed image and the pixel value at the corresponding position in other highest-level decomposed images; The fusion weights of each pixel position in the other decomposed images of the highest layer are determined based on the absolute differences between each pixel position. Pixel positions with an absolute difference greater than or equal to a first threshold constitute the motion region.

2. The method according to claim 1, characterized in that, The step of determining the fusion weights for each pixel position in the other decomposed images at the highest level based on the absolute differences between each pixel position in the other decomposed images at the highest level includes: For pixel locations in other decomposed images at the highest level where the absolute difference is greater than or equal to the first threshold, the fusion weight for that pixel location is determined to be 0. For pixel locations in the highest-level decomposed images where the absolute difference is less than the first threshold, the fusion weight for that pixel location is determined as follows: , where Δ1 is the absolute difference and δ1 is the first parameter.

3. The method according to any one of claims 1-2, characterized in that, The step of determining the fusion weights of other decomposed images in the second-highest layer based on the frame difference between the second-highest layer reference decomposed image and other second-highest layer decomposed images, and the fusion weights of other decomposed images in the highest layer, includes: Based on the frame difference between the second-highest level reference decomposed image and other second-highest level decomposed images, determine the weight coefficients of each pixel position in the other second-highest level decomposed images; The fusion weights of the other decomposed images at the highest level are upsampled to the same size as the decomposed images at the second highest level; The fusion weights of each pixel position in the other decomposed images of the second-highest level are determined based on the weight coefficients of each pixel position in the other decomposed images of the second-highest level and the fusion weights of the corresponding positions obtained by upsampling.

4. The method according to claim 3, characterized in that, The step of determining the fusion weights of each pixel position in the other decomposed images of the second-highest level based on the weight coefficients of each pixel position in the other decomposed images of the second-highest level and the fusion weights of the corresponding positions obtained by upsampling includes: The fusion weights of each pixel position in the other decomposed images of the second-highest level are obtained by multiplying the weight coefficients of each pixel position in the other decomposed images of the second-highest level with the fusion weights of the corresponding positions obtained by upsampling.

5. The method according to claim 1, characterized in that, The image to be fused is a color image; The step of performing Laplacian pyramid decomposition on each image to be fused to obtain multi-layered decomposed images of different resolutions includes: The Y-channel images corresponding to each image to be fused are decomposed using Laplacian pyramids to obtain multi-layer decomposed images of different resolutions, wherein the decomposed images are Y-channel decomposed images.

6. The method according to claim 1, characterized in that, The fusion weight of each pixel position in the reference decomposed image of each layer is 1.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-6.

Images