Image fusion methods, computer program products, storage media and electronic devices

Abstract

This application relates to the field of image processing technology, providing an image fusion method, a computer program product, a storage medium, and an electronic device. The image fusion method includes: acquiring n frames of images to be fused with varying exposure levels; fusing channel images belonging to the same channel, and determining the fused result image based on at least one fused channel image; for channel images belonging to the same target channel among the n frames to be fused, calculating the fused channel image through the following steps: extracting the basic feature maps of the n frame channel images using the backbone network of a pre-trained neural network model, and splitting them into n sub-feature maps; calculating the n-frame fusion mask corresponding to the n frame channel images using the n branch networks of the neural network model; and performing weighted fusion of the n frame channel images using the n-frame fusion mask. This method helps improve the quality of the fused result image, avoiding phenomena such as graying and lack of detail.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image fusion method, a computer program product, a storage medium, and an electronic device. Background Technology

[0002] When bright areas under strong light sources and relatively low-brightness areas such as shadows and backlighting coexist in a shooting scene, the image captured by the camera often exhibits a phenomenon where bright areas appear white due to overexposure, while dark areas appear black due to underexposure. This phenomenon severely affects image quality. A camera's ability to represent the brightest and darkest areas in the same scene is limited; this limitation is commonly referred to as "dynamic range."

[0003] High Dynamic Range (HDR) is a technique that aims to preserve the details of a scene as much as possible when there are strong brightness contrasts. A typical application of this technique is to use a specific fusion algorithm to combine multiple frames of images with different exposure levels taken from the same scene into a single HDR image through weighted fusion. This results in an HDR image with good detail in both highlights and shadows.

[0004] However, the weights selected by existing fusion algorithms often lack robustness, which leads to problems such as graying and lack of detail in the synthesized image when the exposure range of the images to be fused is large, making it impossible to achieve a true HDR effect. Summary of the Invention

[0005] The purpose of this application is to provide an image fusion method, a computer program product, a storage medium, and an electronic device to improve the above-mentioned technical problems.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] In a first aspect, this application provides an image fusion method, comprising: acquiring n frames of images to be fused with different exposure levels; wherein n is an integer greater than 1, each frame of the images to be fused includes at least one channel, the at least one channel including a target channel; fusing n frames of channel images belonging to the same channel in the n frames of the images to be fused to obtain at least one frame of channel fused image, and determining a fusion result image based on the at least one frame of channel fused image; wherein, for the n frames of channel images belonging to the same target channel in the n frames of the images to be fused, the corresponding channel fused image is calculated through the following steps: extracting the basic feature map of the n frames of channel images using the backbone network in a pre-trained neural network model, and splitting the basic feature map into n frames of sub-feature maps; calculating the n frames of fusion mask corresponding to the n frames of channel images using n branch networks in the neural network model; wherein, each branch network is used to calculate a frame of fusion mask corresponding to a frame of channel image based on a frame of sub-feature map, and each frame of fusion mask includes weights for fusion of the corresponding frame of channel image; and performing weighted fusion of the n frames of channel images using the n frames of fusion mask to obtain the channel fused image.

[0008] The above method uses a neural network model to calculate the weights (fusion mask) for fusion when fusing channel images of the target channel. Since the neural network model is trained with training data and is not a preset rule determined by experience, the calculated weights have good robustness. This is beneficial for improving the quality of the fused image (HDR image) when the exposure range of the images to be fused is large, and avoids phenomena such as graying and lack of detail.

[0009] Furthermore, the neural network model used in this method consists of two parts: a backbone network and branch networks. The backbone network is shared by n frames of channel images, which facilitates inter-frame information exchange and improves the accuracy of the fusion mask. Each branch network is used to calculate the fusion mask corresponding to a single frame of channel image, which facilitates parameter optimization for that frame of channel image and also improves the accuracy of the fusion mask. The higher the accuracy of the calculated fusion mask, the better the fusion effect will be.

[0010] In one implementation of the first aspect, the backbone network is a network with an attention mechanism, the attention mechanism of which includes at least one of channel attention, frame attention, and spatial attention; and / or, the branch network is a network with an attention mechanism, the attention mechanism of which includes at least one of channel attention and spatial attention; wherein, channel attention refers to assigning corresponding weights to data in different channels of the feature map, frame attention refers to assigning corresponding weights to data in different frames of the feature map, and spatial attention refers to assigning corresponding weights to data at different spatial locations of the feature map.

[0011] In the above implementation, adding different attention mechanisms to the neural network helps to improve the accuracy of the calculated fusion mask.

[0012] In one implementation of the first aspect, the backbone network includes a multi-attention module, which includes channel attention units and frame attention units connected in sequence. The channel attention units are used to calculate the weights of the input feature map of the multi-attention module in the channel dimension, and multiply the calculated weights by data in different channels of the input feature map to obtain the output feature map of the channel attention unit. The frame attention units are used to transpose the channel dimension and frame dimension of the output feature map of the channel attention unit, calculate the weights of the transposed feature map in the channel dimension, multiply the calculated weights by data in different channels of the transposed feature map to obtain a weighted feature map, and transpose the channel dimension and frame dimension of the weighted feature map to obtain the output feature map of the multi-attention module.

[0013] The multi-attention module in the above implementation includes channel attention and frame attention, which helps to enhance information exchange and sharing between channels and frames, strengthen the connection between frames, and thus improve the accuracy of the calculated fusion mask.

[0014] In one implementation of the first aspect, the branch network includes a multi-receptive-field module, which includes m feature extraction branches and a feature fusion unit; wherein m is an integer greater than 1, each feature extraction branch is used to extract features from the input feature map of the multi-receptive-field module to obtain its own output feature map, and the receptive fields corresponding to each feature extraction branch are different, and the feature fusion unit is used to fuse the output feature maps of the m feature extraction branches to obtain the output feature map of the multi-receptive-field module.

[0015] The multi-receptive field module in the above implementation method contains multiple feature extraction branches corresponding to different receptive fields, which helps to improve the consistency of spatial information and prevent the phenomenon of fusion layering (i.e., discontinuous image or layering) in channel fusion images.

[0016] In one implementation of the first aspect, at least m-1 of the m feature extraction branches include dilated convolutional layers; wherein, when m is greater than 2, the dilation rates of the dilated convolutional layers in the at least m-1 feature extraction branches are different; when there are m-1 feature extraction branches including dilated convolutional layers, the remaining feature extraction branch among the m feature extraction branches includes only a regular convolutional layer or a direct connection branch.

[0017] In the above implementation, different receptive fields are achieved by using dilated convolutions with different dilation rates. Alternatively, a feature extraction branch containing only ordinary convolutional layers can be added (ordinary convolution can be regarded as dilated convolution with a dilation rate of 1), or a direct connection branch can be added (to form a residual structure with other branches).

[0018] In one implementation of the first aspect, each of the at least m-1 feature extraction branches is used to calculate the weight of its own dilated convolutional layer's output feature map in the spatial dimension, and multiply the calculated weight with the data at different spatial locations of its own dilated convolutional layer's output feature map to obtain its own output feature map.

[0019] In the multi-receptive field module, spatial attention mechanisms can be combined to further improve the accuracy of the calculated fusion mask.

[0020] In one implementation of the first aspect, the branch network includes a multi-receptive-field module, which includes m feature extraction units connected in sequence; where m is an integer greater than 1, the first feature extraction unit of the m feature extraction units is used to extract features from the input feature map of the multi-receptive-field module to obtain its own output feature map, each subsequent feature extraction unit is used to extract features from the output feature map of the previous feature extraction unit to obtain its own output feature map, and the output feature map of the last feature extraction unit is the output feature map of the multi-receptive-field module, and each feature extraction unit corresponds to a different receptive field.

[0021] In the previous implementation, multiple receptive fields were achieved by setting up multiple feature extraction branches, which were connected in parallel. In the above implementation, these feature extraction branches can also be changed to a series connection (correspondingly, their names are changed to feature extraction units), which can also achieve multiple receptive fields.

[0022] In one implementation of the first aspect, the backbone network is further used to downsample the n-frame channel images; the step of calculating the n-frame fusion mask corresponding to the n-frame channel images using the n branch networks in the neural network model includes: calculating the n-frame low-resolution fusion mask corresponding to the n-frame channel images using the n branch networks in the neural network model; and upsampling the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel images.

[0023] In the above implementation, the target channel image is first downsampled to reduce its resolution, and then further processing is performed until a low-resolution fusion mask is obtained. Only then is the original resolution fusion mask obtained through upsampling. This allows most of the computation in the entire mask calculation process to be performed on the low-resolution image, thereby significantly improving the efficiency of mask calculation and, consequently, the efficiency of image fusion.

[0024] In one implementation of the first aspect, the step of upsampling the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel image includes: performing edge-preserving smoothing filtering and upsampling on the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel image.

[0025] The above implementation method, when calculating the original resolution fusion mask using the low-resolution fusion mask, not only performs upsampling, but also performs edge-preserving smoothing filtering (e.g., guided filtering), which helps to suppress the mask quality degradation caused by upsampling and improve the accuracy of the fusion mask.

[0026] In one implementation of the first aspect, if each frame of the image to be fused includes multiple channels, and not all of the multiple channels are target channels, then the fusion mask calculated when fusing the channel images of the target channels is used to perform weighted fusion on the n frame channel images of the n frames of the image to be fused that belong to the same non-target channel.

[0027] In the above implementation, for channel images that are not the target channels, the fusion mask used by the target channel image can be directly utilized during fusion, thereby simplifying the fusion process and improving fusion efficiency. For example, the target channel can be a relatively important channel in the image to be fused (such as the Y channel of a YUV image), and the non-target channels can be relatively minor channels in the image to be fused (such as the U and V channels of a YUV image). In this way, even if the fusion mask used for the non-target channels is not calculated based on its own channel image, it will not have a significant impact on the image quality of the fused result image.

[0028] Secondly, this application discloses an image fusion apparatus, comprising: an image acquisition component for acquiring n frames of images to be fused with different exposure levels; wherein n is an integer greater than 1, each frame of the image to be fused includes at least one channel, and the at least one channel includes a target channel; and an image fusion component for fusing n frame channel images belonging to the same channel in the n frame images to be fused to obtain at least one frame channel fused image, and determining a fusion result image based on the at least one frame channel fused image; wherein, for the n frame channel images belonging to the same target channel in the n frame images to be fused, the following steps are performed... The corresponding channel fusion image is calculated as follows: The backbone network of the pre-trained neural network model is used to extract the basic feature maps of the n-frame channel images, and the basic feature maps are split into n-frame sub-feature maps; the n branch networks in the neural network model are used to calculate the n-frame fusion mask corresponding to the n-frame channel images; wherein, each branch network is used to calculate a one-frame fusion mask corresponding to a one-frame channel image based on a one-frame sub-feature map, and each fusion mask includes the weights used for fusion of the corresponding one-frame channel images; the n-frame fusion masks are used to perform weighted fusion of the n-frame channel images to obtain the channel fusion image.

[0029] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0031] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1The steps of the image fusion method provided in the embodiments of this application are illustrated;

[0034] Figure 2 It shows Figure 1 The sub-steps that may be included in step S120;

[0035] Figure 3 This paper illustrates a specific workflow of the image fusion method provided in an embodiment of this application;

[0036] Figure 4 This application illustrates a structure of a multi-attention module provided in an embodiment;

[0037] Figure 5 A structure for computing weights in a multi-attention module is shown;

[0038] Figure 6 This invention illustrates a structure of a multi-receptive field module provided in an embodiment of this application;

[0039] Figure 7 A structure for calculating weights in a multi-receptive-field module is shown;

[0040] Figure 8 The functional components included in the image fusion apparatus provided in the embodiments of this application are shown;

[0041] Figure 9 This application illustrates the structure of an electronic device provided in an embodiment. Detailed Implementation

[0042] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement. The image fusion method in this application embodiment also utilizes technologies such as artificial intelligence.

[0043] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0044] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0045] Figure 1 The steps of the image fusion method provided in the embodiments of this application are illustrated. This method can be, but is not limited to, by... Figure 9The illustrated electronic device performs the operation; for details regarding the structure of this electronic device, please refer to the following section. Figure 9 The explanation. (Refer to...) Figure 1 Image fusion methods include:

[0046] Step S110: Obtain n frames of images to be fused, each with slightly different exposure levels.

[0047] Here, n is an integer greater than 1, and the images to be merged refer to the images that need to be merged together. The goal of merging is to obtain an image with HDR effect. The exposure level of the images to be merged can be quantified in a certain way, such as exposure compensation value (EV). Note that step S110 only requires that the exposure levels of the n frames to be merged are not completely the same, but does not require that the exposure levels of the n frames to be merged be completely different: for example, when n=3, the three frames of EV-1, EV0, and EV+1 are a set of images to be merged that meet the requirements; when n=4, the four frames of EV-1, EV0, EV0, and EV+1 are also a set of images to be merged that meet the requirements. Among them, the image of EV0 appears twice, indicating that the image of EV0 is more important and should have a higher proportion in the fusion result.

[0048] This application does not limit how to obtain the n frames of images to be fused that meet the above requirements: for example, it can receive real-time images from the camera of an electronic device as the images to be fused (e.g., setting the EV value of the camera to -1, 0, and 1 and continuously capturing 3 frames); or it can directly read the pre-saved images to be fused from the memory of the electronic device, and so on. It should also be understood that the n frames of images to be fused must be related to the same scene in terms of content; otherwise, fusing them would be meaningless.

[0049] Step S120: Fuse the n-frame channel images belonging to the same channel in the n-frame images to be fused to obtain at least one channel fused image, and determine the fused result image based on the at least one channel fused image.

[0050] Each frame of the images to be merged contains at least one channel, and the number of channels in each frame is the same. For example, n frames of images to be merged can all be grayscale images, i.e., each containing 1 channel; or n frames of images to be merged can all be YUV images, i.e., each containing 3 channels; or n frames of images to be merged can all be RGB images, i.e., each containing 3 channels, and so on.

[0051] If each frame of the image to be fused contains only one channel, then the channel image of that channel is the image to be fused itself. In step S120, fusing the n-frame channel images is the same as fusing the n-frame images to be fused, and the resulting channel fused image is the fused result image.

[0052] If each frame of the image to be fused contains two or more channels, then the images should be fused channel by channel to obtain a channel-fused image with the same number of channels. These channel-fused images are then stitched together according to their respective channels to obtain the fused result image. For example, if n frames of the image to be fused are all YUV images, then the channel images of the Y channels of the n frames should be fused to obtain a Y-channel fused image; the channel images of the U channels of the n frames should be fused to obtain a U-channel fused image; and the channel images of the V channels of the n frames should be fused to obtain a V-channel fused image. Finally, the three channel-fused images are stitched together as new Y, U, and V channel images to obtain the fused result image. Obviously, the fused result image is still a YUV image.

[0053] Each frame of the fused image contains target channels, which can be one or more. Besides the target channels, any remaining channels in the image to be fused are called non-target channels. Which channels are target channels can be predetermined. For example, if the image to be fused is a grayscale image, its only channel is necessarily the target channel; if the image to be fused is a YUV image, the Y channel can be selected as the target channel, in which case U and V are non-target channels. Alternatively, the Y, U, and V channels can all be selected as target channels, in which case there are no non-target channels.

[0054] The difference between target channels and non-target channels is that, for each target channel's n-frame channel image, the following should be performed: Figure 2 Steps S121 to S123 are fused into a corresponding one-frame channel fused image. For each of the n-frame channel images of a non-target channel (if there is a non-target channel), the following steps can be performed: Figure 2 Step S124 in the process is fused into a corresponding one-frame channel fused image. Obviously, steps S121 to S124 are all sub-steps of step S120.

[0055] Steps S121 to S123 constitute the main fusion scheme proposed in this application, which can significantly improve the image fusion effect (mainly referring to HDR effect). Some implementations of these steps can also significantly improve the efficiency of image fusion. Therefore, as an optional strategy, to improve the image fusion effect as much as possible, all channels contained in the image to be fused can be used as target channels. Alternatively, if considering efficiency or other factors, only one or a few channels in the image to be fused that have the most significant impact on the fusion result can be used as target channels (e.g., the Y channel in a YUV image), and the remaining channels can be used as non-target channels (e.g., the U and V channels in a YUV image).

[0056] The following is about Figure 2The steps in the process will be explained in detail, mainly combining the following: Figure 3 The process is shown.

[0057] Step S121: Extract the basic feature map of n-frame channel images using the backbone network in the pre-trained neural network model, and split the basic feature map into n-frame sub-feature maps.

[0058] Step S122: Calculate the n-frame fusion mask corresponding to the n-frame channel images using the n-branch network in the neural network model.

[0059] Steps S121 and S122 will be explained together. The neural network model used in these steps is pre-trained (before step S120 is executed). The training process will be explained later. If there are multiple target channels, a separate neural network model can be trained for each target channel, or multiple target channels can share a single neural network model to save computational resources.

[0060] The specific type of neural network used in the neural network model is not limited; for example, it can be a convolutional neural network (CNN), a recurrent neural network (RNN), an artificial neural network (ANN), or a combination of these neural networks. The neural network model includes at least two parts: a backbone network and n branch networks. The tail of the backbone network is connected to the head of each branch network, and both the backbone network and the branch networks are themselves neural networks with trainable parameters.

[0061] The backbone network can be used to extract features from n frames of target channel images to obtain the corresponding basic feature map. Then, the basic feature map is split into n sub-feature maps, and each sub-feature map is used as the input of a branch network.

[0062] Reference Figure 3 The top white rectangle represents the n-frame channel image of the target channel. Figure 3 In the diagram, n = 3, denoted as x_h, n frames of channel images can be input as a whole into the backbone network. The shape of this n-frame image is n×1×W×H, where n is the number of frames, 1 is the number of channels per frame, and W and H are the width and height of the channel image, respectively. The cuboid below the multi-attention module (described later) represents the calculated basic feature map, with a shape of n×c×w×h, where n is the number of frames, c is the number of channels per frame (c > 1), and w and h are the width and height of the basic feature map, respectively. The n cuboids with a shape of 1×c×w×h below the splitting module represent n frames of sub-feature maps. It can be seen that the splitting module splits the basic feature map along the frame dimension. Each sub-feature map can be considered to correspond to one frame of channel image, but this "correspondence" should not be interpreted as each sub-feature map being calculated based solely on one frame of channel image.

[0063] Each branch network can calculate a corresponding fusion mask based on a sub-feature map of the input frame. For a total of n branch networks, there are n frames of calculated fusion masks. As mentioned earlier, the sub-feature map corresponds to the channel image, so each frame of fusion mask calculated based on the sub-feature map also corresponds to a frame of channel image.

[0064] The fusion mask can be viewed as a weighted image, where each pixel value represents the weight used in fusion of a corresponding frame of channel images. The resolution of the fusion mask is the same as that of the channel images. How to use the fusion mask to fuse channel images will be explained further in step S123.

[0065] Optionally, to facilitate the use of the fusion mask, the weights in the fusion mask can be normalized. This means that after normalization, for any pixel position x in the fusion mask, all n weights at x in the n-frame fusion mask are 1, and the value of each weight is within the interval [0,1]. In the following description of step S123, for simplicity, we can assume that the weights in the fusion mask are all normalized.

[0066] Reference Figure 3 The n gray rectangles below the filtering / upsampling module (described later) represent n frames of fusion masks, denoted as m_h. The shape of each frame of fusion mask is 1×1×W×H, meaning that the shape of a frame of fusion mask is exactly the same as the shape of a frame of channel image.

[0067] Furthermore, it should be noted that although step S122 uses branch networks to calculate the fusion mask, it does not limit the output of the branch networks to necessarily being the fusion mask. Each branch network may directly output the corresponding fusion mask, or it may require further calculations on the output of the branch networks to obtain the corresponding fusion mask. For example, in Figure 3 In the middle, the n smaller gray rectangles below the multi-receptor field module (described later) are the outputs of the branch network, while the n-frame fusion mask is located below the filtering / upsampling module, which is no longer part of the branch network.

[0068] Step S123: Use an n-frame fusion mask to perform weighted fusion on the n-frame channel images to obtain a channel fused image.

[0069] Let's take linear fusion as an example. Linear fusion can be expressed by the formula:

[0070]

[0071] Among them, Y * Represents a channel-fused image, Y i M represents the image of the i-th frame channel. iIndicates the fusion mask of the i-th frame (and Y) i The corresponding fusion mask), the symbol "⊙" indicates the multiplication of pixel values ​​at the corresponding positions. As mentioned earlier, the fusion mask and the channel image have the same resolution, so they can be operated on using the ⊙ symbol. It can be seen that linear fusion is actually the process of using the n-frame fusion mask as the weight image and performing a weighted summation on the n-frame channel images to obtain the channel fusion image. If the weights in the fusion mask are normalized, it can be guaranteed that the pixels in the channel fusion image and the pixels in the channel image have the same value range, for example, both are [0, 255].

[0072] Of course, linear fusion is not the only weighted fusion method. For example, pyramid fusion can also be used, the steps of which are briefly described below:

[0073] Step A: Construct a Gaussian pyramid corresponding to each frame's fusion mask, resulting in a total of n Gaussian pyramids; and construct a Laplacian pyramid corresponding to each frame's channel image, resulting in a total of n Laplacian pyramids.

[0074] The Gaussian Pyramid and the Laplace Pyramid have the same number of layers, and the images of pyramids on the same layer in both types of pyramids have the same resolution. For the construction methods of the Gaussian Pyramid and the Laplace Pyramid, please refer to existing technologies, which will not be explained in detail here. In addition, there is no restriction on the order of constructing the Gaussian Pyramid and the Laplace Pyramid, or they can be carried out simultaneously.

[0075] Using the notation above, if M i Let represent the fusion mask of the i-th frame. Then, the n Gaussian pyramids can be represented as... in Represents the i-th Gaussian pyramid (based on M). i Construct the j-th layer pyramid image. If Y i Let i represent the i-th frame channel image. Then, the n Laplacian pyramids can be represented as follows: in Represents the i-th Laplace pyramid (based on Y). i Construct the j-th layer pyramid image. The values ​​of i and j above should be all values ​​within their respective ranges, i.e., i takes all integers from 1 to n. Assuming that both the Gaussian pyramid and the Laplace pyramid are L layers (L is an integer greater than 1), then j takes all integers from 1 to L.

[0076] Step B: Use n Gaussian pyramids to perform a weighted fusion of n Laplace pyramids to obtain the fused Laplace pyramid.

[0077] In this fused Laplacian pyramid, each pyramid image is obtained by weighted fusion of n frames of pyramid images located at that level from n Gaussian pyramids. For example, when using linear fusion, the fusion process can be expressed as the following formula:

[0078]

[0079] Among them, Y j This represents the image of the pyramid at level j in the merged Laplace's Pyramid. The entire merged Laplace's Pyramid can be denoted as... j takes all integers from 1 to L.

[0080] Step C: Reconstruct the image using the fused Laplacian pyramid to obtain a channel-fused image.

[0081] For how to perform image reconstruction based on the Laplacian pyramid, you can refer to existing technologies, which will not be explained in detail here. The result of the reconstruction is the channel fusion image to be calculated in step S123.

[0082] As can be seen from the comparison, pyramid fusion does not directly use a fusion mask to linearly fuse the channel images themselves. Instead, it uses the Gaussian pyramid of the fusion mask to perform layered linear fusion of the Laplacian pyramid of the channel images. According to the construction process of the Laplacian pyramid, each layer of the pyramid image can be considered as a feature map extracted from the channel image, and these feature maps represent image details at different frequencies in the channel image. Therefore, weighted fusion of pyramid images at the same layer is equivalent to performing image fusion separately in the subspace corresponding to each frequency. Thus, its fusion difficulty is lower, and the fusion effect is better than fusion directly on the original channel images, thereby improving the quality of the channel fused image. Of course, pyramid fusion has higher algorithmic complexity than linear fusion, so different fusion methods can be used depending on the requirements in specific implementations.

[0083] Reference Figure 3 , Figure 3 The diagram illustrates a linear fusion approach. At position m_h, in addition to the n gray rectangles, n white rectangles are drawn. These white rectangles represent the n frames of images to be fused. The "×" and "+" symbols represent the multipliers and adders used for linear fusion. Figure 3 The white rectangle at the bottom represents the channel fusion image of the target channel, which has a shape of 1×1×W×H, the same as any single frame of the channel image.

[0084] about Figure 3 Two more points need to be explained:

[0085] First, the filtering / upsampling module, the multiplier and adder for linear fusion are not included in the neural network model because they do not contain trainable model parameters (but may contain some hyperparameters). Of course, it is also possible to consider them as part of the neural network model without having a substantial impact on the scheme.

[0086] Secondly, Figure 3 The diagram only shows a basic structure of a neural network model. However, a neural network model may contain many additional structures on top of this basic structure, such as pooling layers and convolutional layers. To avoid making the model appear too complex, these additional structures are not shown, but they can be added to the model as needed during implementation. Figures 4-7 Similar situations exist, which will not be explained one by one.

[0087] Step S124: Use other methods to fuse the n-frame channel images to obtain a channel-fused image.

[0088] For channel images belonging to the same non-target channel, other methods can be used for fusion. The "other methods" in step S124 refer to methods other than those in steps S121 to S123, such as any existing image fusion method for obtaining HDR images.

[0089] In addition, the fusion mask calculated during the fusion of the target channel's channel images can be directly used to perform weighted fusion on n frames of channel images belonging to the same non-target channel. For example, if the image to be fused is a YUV image and the target channel is the Y channel, and n frames of fusion masks for the Y channel are calculated by executing steps S121 to S123, then n frames of channel images for the U or V channels can be directly fused using these n frames of fusion masks. The fusion method can refer to step S123, such as linear fusion or pyramid fusion. A similar approach can be taken when there are multiple target channels. For example, if there are two target channels, n frames of fusion masks can be calculated using the channel images of these two target channels, resulting in a total of 2n frames of fusion masks. Then, the fusion masks corresponding to every two frames are averaged to obtain n frames of fusion masks, and these n frames of fusion masks are then used to fuse the channel images of non-target channels.

[0090] The above fusion method simplifies the image fusion process for non-target channels and improves fusion efficiency. For example, the target channel can be a relatively important channel in the image to be fused (such as the Y channel of a YUV image), and the non-target channel can be a relatively minor channel in the image to be fused (such as the U and V channels of a YUV image). In this way, even if the fusion mask used for the non-target channel is not calculated based on its own channel image, it will not have a significant impact on the image quality of the fused image.

[0091] Here is a brief summary Figure 1 and Figure 2 The image fusion method in this paper: When fusing the channel images of the target channel, this method uses a neural network model to calculate the weights (i.e., the fusion mask) for fusion. Since the neural network model is trained with training data and is not a preset rule determined by experience, the calculated weights have good robustness. This helps to improve the quality of the fused image (HDR image) when the exposure range of the images to be fused is large, and avoids phenomena such as graying and lack of detail.

[0092] In this context, if EV values ​​are used to represent exposure levels, a large span of exposure levels can be understood as follows: after arranging n frames of images to be fused in ascending or descending order of EV values, the difference in EV values ​​between adjacent frames is significant, or the difference in EV values ​​between the first and last frames is significant. It should be understood that even if the span of exposure levels between the images to be fused is not large, the image fusion method in this embodiment still improves the quality of the fused image.

[0093] Furthermore, the neural network model used in this method consists of a backbone network and branch networks. The backbone network is shared by n frames of channel images, which facilitates inter-frame information exchange between the images to be fused and improves the accuracy of the fusion mask. Each branch network is used to calculate the fusion mask corresponding to a single frame of channel image, which facilitates parameter optimization for that frame of channel image and also improves the accuracy of the fusion mask. In short, the higher the accuracy of the calculated fusion mask, the better the image fusion effect will be.

[0094] Below, based on the above embodiments, we will continue to introduce the downsampling and upsampling mechanisms in the image fusion method:

[0095] In one implementation, the backbone network also includes a downsampling module, which can be located at the beginning of the backbone network and is used to downsample n-frame channel images, such as... Figure 3 As shown. Downsampling will result in a low-resolution channel image. Note that "low resolution" here is relative, not absolute; it means the resolution is lower than the original resolution of the channel image. This interpretation applies to all subsequent uses of "low resolution." Continue to refer to... Figure 3 The white rectangle below the downsampling module represents the low-resolution channel image, denoted as x_l. Its shape is n×1×w×h, and w<W and h<H, meaning its resolution is lower than the original channel image.

[0096] Subsequent calculations of the neural network model are all based on low-resolution channel images, and the outputs of the n branch networks (i.e., the outputs of the neural network model) are also n low-resolution fusion masks. These masks correspond one-to-one with the n channel images, and the pixel values ​​in them represent the weights used for channel image fusion, but their resolution is lower than the fusion mask mentioned in step S122. (Refer to...) Figure 3 The basic feature map has a shape of n×c×w×h, the sub-feature map has a shape of 1×c×w×h, and the output m_l of the neural network model has a shape of n×1×w×h (in the figure, there are n 1×1×w×h masks located below the multi-receptive field module). Without exception, their resolutions are all lower than the original resolution of the channel image.

[0097] Since the low-resolution fusion mask of n frames is not suitable for weighted fusion, it needs to be upsampled to obtain a fusion mask with the same resolution as the channel image (i.e., the fusion mask mentioned in step S122), before proceeding to the subsequent fusion step S123. Figure 3 In this process, the upsampling operation is performed by the filtering / upsampling module (the role of "filtering" will be explained later). It's easy to see that after upsampling, the mask shape becomes 1×1×W×H, meaning the original resolution has been restored. During upsampling, n frames of low-resolution fused masks can be input as a whole into the filtering / upsampling module, which can then perform upsampling of n frames in batches.

[0098] In the above implementation, the target channel image is first downsampled to reduce its resolution, and then further processing is performed until a low-resolution fusion mask is obtained. Only then is the original resolution fusion mask obtained through upsampling. This allows most of the computation in the entire mask calculation process to be performed on the low-resolution image, thereby significantly improving the efficiency of mask calculation and, consequently, the efficiency of image fusion.

[0099] Furthermore, the inventors discovered that upsampling can cause a certain degree of mask quality degradation. Therefore, in one alternative approach, when upsampling the n-frame low-resolution fusion mask, edge-preserving smoothing filtering can be applied to improve the quality of the resulting n-frame fusion mask. Therefore, in Figure 3 The design employs an upsampling module with filtering capabilities instead of a simple upsampling module (although using a simple upsampling module would also be feasible).

[0100] Edge-preserving smoothing filters are a class of image filtering algorithms, such as bilateral filtering and guided filtering. These algorithms can smooth image content while preserving edge information as much as possible. Filtering can be performed before upsampling. Figure 3For example, we can first perform guided filtering on m_l (select x_l as the guide graph), and then upsample the filtering result to obtain m_h.

[0101] Alternatively, filtering and upsampling can be performed simultaneously (here, "simultaneously" should be understood as the upsampling and filtering processes being mixed together, with no clear boundary between them, and should not be interpreted as "parallel"), for example, Figure 3 The filtering / upsampling module can select the Guided Filtering for Joint Upsampling (GFU) module. The GFU module has three inputs: x_l, m_l, and x_h, and outputs m_h. Internally, the GFU module calculates two low-resolution mapping matrices A and B based on x_l and m_l, then upsamples A and B to obtain two original-resolution mapping matrices A* and B*. Finally, x_h is used as the guide map, and the formula m_h = A*⊙x_h + B is used to map it to m_h. Because the guide map in the GFU module uses the higher-resolution x_h instead of x_l, it can significantly reduce the loss of edge information caused by upsampling.

[0102] It should be understood that the downsampling and upsampling mechanisms in the image fusion process are used in conjunction; if downsampling is not performed, then upsampling is naturally unnecessary. For example, in some different... Figure 3 In this implementation, the backbone network does not contain a downsampling module, and there is no need to set up a filtering / upsampling module after the branch network. The branch network can directly output an n-frame fusion mask with the same resolution as the channel image.

[0103] Below, based on the above embodiments, the attention mechanism in the image fusion method is introduced:

[0104] In one implementation, the neural network model used in image fusion methods incorporates an attention mechanism. When observing an image, humans show varying degrees of attention to different parts—important parts receive higher attention, while less important parts receive lower attention. Therefore, for a specific image processing task (e.g., classification, segmentation, fusion, recognition), the features of different parts of an image have different levels of importance. The attention mechanism uses certain techniques to express these differences in importance. For example, different weights are assigned to features in different parts of the image; features with higher weights are considered more important, and features with lower weights are considered less important.

[0105] In neural network models, an attention mechanism can be implemented using a two-branch, multiplier structure. One branch inputs the feature map into the multiplier, while the other branch calculates the weights of features in different parts of the feature map and inputs these weights into the multiplier, which then performs feature weighting. How the feature map is divided into different parts will create different types of attention mechanisms.

[0106] In the scheme of this application, the feature map has four dimensions, namely... Figure 3 Taking the basic feature map as an example, its shape is n×c×w×h, where n represents the frame dimension, c represents the channel dimension, w represents the horizontal axis dimension, and h represents the vertical axis dimension. Alternatively, w and h can be considered as a single dimension, i.e., a spatial dimension, in which case the feature map has three dimensions. Therefore, the scheme in this application may use the following three attention mechanisms:

[0107] Channel attention, or channel-dimensional attention mechanism, refers to assigning corresponding weights to data in different channels of a feature map.

[0108] Frame attention, or frame-level attention mechanism, refers to assigning corresponding weights to data in different frames of a feature map.

[0109] Spatial attention, or attention mechanism in spatial dimension, refers to assigning corresponding weights to data at different spatial locations in a feature map.

[0110] In neural network models, attention mechanisms can be introduced into the backbone network, branch networks, or both. Specifically, the backbone network may employ at least one of channel attention, frame attention, and spatial attention mechanisms, while branch networks may employ at least one of channel attention and spatial attention mechanisms. (See reference...) Figure 3 Since each branch network's sub-feature map is only one frame, and according to the definition of frame attention, it is only meaningful to weight the data in different frames, there is no need to consider frame attention for the branch network.

[0111] Adding different attention mechanisms to neural networks can improve the accuracy of the calculated fusion mask. However, it should be understood that the image fusion method proposed in this application can still be achieved even without adding attention-related structures to the neural network model. Specific examples will be given later to illustrate how to introduce attention mechanisms into the backbone and / or branch networks.

[0112] In one implementation, the backbone network includes a multi-attention module, which comprises channel attention units and frame attention units connected sequentially, such as... Figure 4As shown. The location of the multi-attention module in the backbone network is not limited; for example, in... Figure 3 In this context, the multi-attention module is positioned after the downsampling module.

[0113] The input feature map of the channel attention unit is also the input feature map of the multi-attention module (here, "feature map" should be interpreted broadly; for example, if the multi-attention module is located at the beginning of the backbone network, the image to be fused is also considered a feature map). The channel attention unit is used to calculate the weights of the input feature map of the multi-attention module in the channel dimension, and multiplies the calculated weights with the data in different channels of the input feature map of the multi-attention module to obtain the output feature map of the channel attention unit. Comparing this to the definition of channel attention, it can be seen that the channel attention unit possesses channel attention.

[0114] Reference Figure 4 The input feature map of the channel attention unit has a shape of n×c×w×h (a convolutional layer is set between the downsampling module and the multi-attention unit to increase the number of channels of x_l from 1 to c, but...). Figure 3 (Not shown in the image), one branch of the channel attention unit directly inputs the feature map into the multiplier, while the other branch calculates a weight of shape n×c×1×1 based on the feature map. That is, each channel of the feature map is assigned a corresponding weight. The multiplier is responsible for multiplying the data in each channel of the feature map with the weight corresponding to that channel, so that the data in the more important channels are assigned larger weights, while the data in the relatively unimportant channels are assigned smaller weights.

[0115] Furthermore, the branches for calculating weights in the channel attention unit can, but are not limited to, employing... Figure 5 The structure in the diagram includes a global pooling layer, a first fully connected layer, a ReLU function, a second fully connected layer, and a Sigmoid function connected in sequence. The input to the global pooling layer is the input to the channel attention unit, and the output of the Sigmoid function is the weight to be calculated for that branch. Figure 5 The right side shows the shape of the feature map output by each layer / function. The first fully connected layer can change the number of channels of its input feature map from c to c / r, and the second fully connected layer can change the number of channels of its input feature map from c / r to c, where r is a preset hyperparameter.

[0116] The input feature map of the frame attention unit is also the output feature map of the channel attention unit. The frame attention unit transposes the channel dimension and frame dimension of its own input feature map, calculates the weight of the transposed feature map in the channel dimension, multiplies the calculated weight with the data in different channels of the transposed feature map to obtain the weighted feature map, and transposes the channel dimension and frame dimension of the weighted feature map again to obtain its own output feature map, which is the output feature map of the multi-attention module.

[0117] Although the frame attention unit weights the data in different channels, the weighting operation is performed after the data is transposed. Through the transpose operation, the original channel dimension in the feature map becomes the frame dimension, and the original frame dimension becomes the channel dimension. That is, weighting the data in different channels of the transposed feature map is essentially weighting the data in different frames of the feature map before transposition. Thus, by comparing with the definition of frame attention, we can see that the frame attention unit has frame attention.

[0118] Reference Figure 4 The input feature map of the frame attention unit has a shape of n×c×w×h. After transpose, its shape becomes c×n×w×h. One branch of the frame attention unit directly inputs the feature map into the multiplier, while the other branch calculates a weight of shape c×n×1×1 based on the feature map. That is, a corresponding weight is calculated for each channel of the feature map (the number of channels is n at this time). The multiplier is responsible for multiplying the data in each channel of the feature map (actually the data in the original frame) with the weight corresponding to that channel, so that the data in the more important channels are given larger weights, while the data in the relatively unimportant channels are given smaller weights. Finally, the dimensions in the feature map are restored to their original order by transposing it again.

[0119] It is easy to see that by transposing, the frame attention unit can reuse the structure of the channel attention unit, thus simplifying the network design. The branches for calculating weights in the frame attention unit can, but are not limited to, using... Figure 5 The structure in the text will not be repeated here.

[0120] The multi-attention module in the above implementation includes channel attention and frame attention. Channel attention enhances the differences in features in different channels, while frame attention enhances the differences in features in different frames. Furthermore, the multi-attention module is set in the backbone network shared by n frames of images to be fused. Therefore, this module is beneficial for enhancing information exchange and sharing between channels and frames, strengthening the connection between frames, and thus improving the accuracy of the calculated fusion mask.

[0121] It should be understood that in some other implementations, even if only channel attention and frame attention are introduced in the backbone network, the channel attention unit and frame attention unit do not necessarily have to be connected together. For example, they can be set independently in different positions in the backbone network.

[0122] In one implementation, each branch network includes a multi-receptive-field module, such as... Figure 3 As shown. The location of the multi-receptive-field module in the branch network is not limited, for example, in Figure 3 In this context, the multi-receptive field module is located at the beginning of each branch network.

[0123] The multi-receptive-field module includes m feature extraction branches and one feature fusion unit, where m is an integer greater than 1. Each feature extraction branch is used to extract features from the input feature map of the multi-receptive-field module to obtain its own output feature map. The feature fusion unit is used to fuse the output feature maps of the m feature extraction branches to obtain the output feature map of the multi-receptive-field module.

[0124] Figure 6 This illustrates a structure for a multi-receptive-field module. Figure 6 In the multi-receptive-field module, the input is a feature map with a shape of 1×c×w×h. The module includes 5 feature extraction branches (m=5) and a feature fusion unit. The feature fusion unit includes a stitching structure and an adder. The stitching structure is used to stitch the output feature maps of the top 4 feature extraction branches into a whole. The adder is used to add the stitched feature map to the output feature map of the bottom feature extraction branch and output the result. The output feature map is the low-resolution fusion mask mentioned earlier, with a shape of 1×1×w×h.

[0125] It should be understood that Figure 6 The feature fusion unit in the diagram omits some structures that are not shown: For example, the output feature maps of the four feature extraction branches above can have a shape of 1×c×w×h, and the concatenated shape is 1×4c×w×h (assuming concatenation by channel). Before inputting into the adder, a convolutional layer can be set to reduce its dimension to 1×c×w×h (in order to keep it the same as the other input shape of the adder); another example is that the original shape of the adder's output feature map is 1×c×w×h, which can be reduced to 1×1×w×h by setting a convolutional layer before output.

[0126] In the multi-receptive-field module, the receptive fields corresponding to the m feature extraction branches are different. The receptive field can be understood as the area of ​​the region corresponding to a pixel in the feature map within the original image. A larger receptive field allows the features in the feature map to better describe larger objects in the original image, while a smaller receptive field allows the features in the feature map to only describe smaller objects. In the scheme of this application, the aforementioned "original image" can be understood as a channel image or an image to be fused, and the receptive field corresponding to the feature extraction branch can be understood as the intermediate feature map of the feature extraction branch (e.g., ...). Figure 6 (Feature map output by the hollow convolutional layer) or the receptive field corresponding to the output feature map.

[0127] The multi-receptive-field module, by incorporating multiple feature extraction branches corresponding to different receptive fields, enables the extracted features to comprehensively describe different objects in the channel images at various scales. This improves the consistency of spatial information, prevents fusion layering in the channel-fused image, and ultimately enhances the quality of the fused image. Fusion layering manifests as discontinuities in image brightness and the appearance of banding. The inventors, after research, believe that fusion layering is caused by the extracted features failing to effectively describe the objects in the channel images; for example, different local regions of the same object are mistakenly identified as different objects.

[0128] The following explains in detail how each feature extraction branch achieves different receptive fields:

[0129] Method 1: Using convolutional layers with different void ratios

[0130] If m > 2, dilated convolutional layers are set in m-1 of the m feature extraction branches, and the dilation rates of the dilated convolutional layers in the m-1 feature extraction branches are all different. The remaining feature extraction branch is a direct connection branch (directly passing the input feature map of the multi-receptive field module). In particular, if m = 2, dilated convolutional layers are set in only one feature extraction branch according to the above rules, and its dilation rate only needs to be not equal to 1.

[0131] For example, in Figure 6 In the diagram, the four feature extraction branches all include dilated convolutional layers with dilation rates of 2, 4, 8, and 16 (increasing in powers of 2), resulting in progressively larger receptive fields (all larger than the receptive field of the input feature map of the multi-receptive field module). The remaining branch is a direct connection branch, with a receptive field equal to that of the input feature map of the multi-receptive field module, thus forming five feature extraction branches with distinct receptive fields. Furthermore, setting the remaining branch as a direct connection branch allows for the formation of a residual structure within the multi-receptive field module, which is beneficial for improving the computational accuracy of the fusion mask.

[0132] As an alternative, the direct branch above can also be replaced with a branch that only includes a regular convolutional layer (which can be regarded as a special dilated convolutional layer with a dilation rate of 1), and optionally, the size of the kernel of the regular convolution can be consistent with the dilated convolutions in the other feature extraction branches.

[0133] As an alternative, multiple receptive fields can also be achieved by setting dilated convolutional layers in each of the m feature extraction branches, and making the dilation rate of the dilated convolutional layers in the m feature extraction branches different.

[0134] Furthermore, an attention mechanism can be introduced into the multi-receptive-field module to improve the computational accuracy of the fusion mask. Taking the case where m-1 feature extraction branches have dilated convolutional layers, and the remaining feature extraction branch is a direct-connection branch as an example, assuming the introduced attention mechanism is spatial attention, since the direct-connection branch does not contain any network structure, spatial attention can be added to the remaining m-1 feature extraction branches. Specifically, after each feature extraction branch completes the dilated convolution calculation, it can further calculate the spatial weights of its own dilated convolutional layer's output feature map, and multiply the calculated weights by the data at different spatial locations of its own dilated convolutional layer's output feature map to obtain its own output feature map.

[0135] Reference Figure 6 Taking the topmost feature extraction branch as an example, its input feature map has a shape of 1×c×w×h. Assuming that the dilated convolution does not change its shape, the output feature map of the dilated convolution layer still has a shape of 1×c×w×h. The remaining part of the feature extraction branch is further divided into two sub-branches. One sub-branch directly inputs the feature map into the multiplier, while the other branch calculates a weight with a shape of 1×1×w×h based on the feature map. That is, a corresponding weight is calculated for each spatial location (i.e., pixel location) of the feature map. The multiplier is responsible for multiplying the data at the spatial location of the feature map with the weight corresponding to that location, so that the data at more important spatial locations are given larger weights, while the data at relatively unimportant spatial locations are given smaller weights.

[0136] The sub-branch for calculating weights in the feature extraction branch may, but is not limited to, using... Figure 7 The structure consists of two pooling layers connected in parallel, a concatenated structure, a convolutional layer, and a sigmoid function. The inputs to the two pooling layers are also the inputs to the sub-branch, and the output of the sigmoid function is the weight to be calculated for that sub-branch. The pooling operations of the two pooling layers are different: the left pooling layer performs max pooling (taking the maximum value of the data in each channel), while the right pooling layer performs average pooling (taking the average value of the data in each channel). Figure 7The right side shows the shape of the feature map output of each layer / structure / function (the shape of the output feature map of the pooling layer on the left is shown on the left).

[0137] It should be understood that if the remaining feature extraction branch is not a directly connected branch, but only contains ordinary convolutional layers, spatial attention can also be added to it; if dilated convolutional layers are set in all m feature extraction branches, spatial attention can be added to all feature extraction branches. It should also be noted that the attention mechanism is not mandatory; removing the attention mechanism (e.g., removing...) Figure 6 Even with the sub-branch that calculates weights and the multiplier, the multi-receptive field module can still perform feature extraction normally.

[0138] Method 2: Use ordinary convolutional layers with different kernel sizes

[0139] If m > 2, then ordinary convolutional layers are set in m-1 of the m feature extraction branches, and the kernel size of the ordinary convolutional layers in the m-1 feature extraction branches is different. The remaining feature extraction branch is a direct connection branch. In particular, if m = 2, then only one feature extraction branch has an ordinary convolutional layer according to the above rules, and its kernel size can be set freely (greater than 1×1 is sufficient).

[0140] For example, there are a total of 4 feature extraction branches, one of which is a direct connection branch, and the remaining 3 branches can use 3×3, 5×5, and 7×7 convolution kernels respectively.

[0141] As an alternative, multiple receptive fields can also be achieved by setting ordinary convolutional layers in each of the m feature extraction branches, and making the kernel size of the ordinary convolutional layers in the m feature extraction branches different.

[0142] It should be understood that an attention mechanism can also be introduced into the feature extraction branch of Method 2. For specific implementation, please refer to Method 1, which will not be repeated here.

[0143] It should be noted that methods 1 and 2 can be combined, as both the dilatation rate and the kernel size affect the receptive field size. Furthermore, different factors can be used to make each feature extraction branch have a different receptive field, such as using convolutional layers of different lengths, setting pooling parameters with different parameters, and so on.

[0144] In the above explanation, both Method 1 and Method 2 are based on the structure of a multi-receptive-field module comprising multiple parallel feature extraction branches. However, in other implementations, these feature extraction branches can be converted to a series connection, achieving the same multi-receptive-field effect. In this case, the multi-receptive-field module comprises m feature extraction units connected sequentially (i.e., in series), where m is an integer greater than 1. The first feature extraction unit extracts features from the input feature map of the multi-receptive-field module to obtain its own output feature map. Each subsequent feature extraction unit extracts features from the output feature map of the previous feature extraction unit to obtain its own output feature map. The output feature map of the last feature extraction unit is the output feature map of the multi-receptive-field module, and each feature extraction unit corresponds to a different receptive field.

[0145] As can be seen from the functional description of the feature extraction units, each feature extraction unit is roughly equivalent to a feature extraction branch mentioned earlier. When implementing feature extraction units, the structure of feature extraction branches can be directly adopted (including the addition of attention mechanisms), which will not be repeated here. Of course, directly connected branches do not have corresponding feature extraction units. Furthermore, since the feature extraction units are inherently connected in series, there is no need to set up a feature fusion unit in the multi-receptive field module.

[0146] For multi-receptive field modules with an internal serial structure, it is also beneficial to improve the consistency of spatial information, prevent the phenomenon of fusion layering in channel fusion images, and thus improve the quality of the fusion result image.

[0147] Finally, based on the above embodiments, a brief explanation of how the neural network model in the image fusion method is trained is provided. This model can be trained using either unsupervised or supervised training.

[0148] Taking supervised training as an example, n training images with varying exposure levels (for simplicity, let's consider single-channel images) can be used as training samples. After inputting the training samples into the neural network model, the fused result image can be calculated using the n-frame fusion mask generated by the model. Note that since the training images are assumed to be single-channel images, the fused result image can be obtained directly after fusion. Also, the fusion mask and fused result image here refer to the training phase and should not be confused with those in step S120. Then, the prediction loss can be calculated using the fused result image and the corresponding standard result image of the training samples. The form of the loss function is not limited. The standard result image has better HDR performance and can be pre-determined manually in conjunction with other image fusion algorithms. Finally, the parameters in the neural network model can be updated based on the calculated prediction loss, for example, using the backpropagation algorithm.

[0149] For unsupervised training, model parameters can be updated by calculating blind evaluation metrics such as MEF-SSIM, which will not be elaborated further.

[0150] Figure 8 The functional components included in the image fusion apparatus 200 provided in an embodiment of this application are shown. (Refer to...) Figure 8 The image fusion device 200 includes:

[0151] Image acquisition component 210 is used to acquire n frames of images to be fused that have different exposure levels; where n is an integer greater than 1, and each frame of images to be fused includes at least one channel, and the at least one channel includes a target channel;

[0152] Image fusion component 220 is used to fuse n channel images belonging to the same channel in the n frames of images to be fused, to obtain at least one channel fused image, and to determine a fusion result image based on the at least one channel fused image; wherein, for the n channel images belonging to the same target channel in the n frames of images to be fused, the corresponding channel fused image is calculated through the following steps: extracting the basic feature map of the n channel images using the backbone network in the pre-trained neural network model, and splitting the basic feature map into n sub-feature maps; calculating the n-frame fusion mask corresponding to the n channel images using the n branch networks in the neural network model; wherein, each branch network is used to calculate a one-frame fusion mask corresponding to a one-frame channel image based on a one-frame sub-feature map, and each fusion mask includes the weights used for fusion of the corresponding one-frame channel image; and performing weighted fusion of the n-frame channel images using the n-frame fusion mask to obtain the channel fused image.

[0153] In one implementation of the image fusion device 200, the backbone network is a network with an attention mechanism, the attention mechanism of which includes at least one of channel attention, frame attention, and spatial attention; and / or, the branch network is a network with an attention mechanism, the attention mechanism of which includes at least one of channel attention and spatial attention; wherein, channel attention refers to assigning corresponding weights to data in different channels of the feature map, frame attention refers to assigning corresponding weights to data in different frames of the feature map, and spatial attention refers to assigning corresponding weights to data at different spatial locations of the feature map.

[0154] In one implementation of the image fusion device 200, the backbone network includes a multi-attention module, which includes channel attention units and frame attention units connected in sequence. The channel attention units calculate the weights of the input feature map of the multi-attention module in the channel dimension, and multiply the calculated weights by data in different channels of the input feature map to obtain the output feature map of the channel attention unit. The frame attention units transpose the channel and frame dimensions of the output feature map of the channel attention unit, calculate the weights of the transposed feature map in the channel dimension, multiply the calculated weights by data in different channels of the transposed feature map to obtain a weighted feature map, and transpose the channel and frame dimensions of the weighted feature map to obtain the output feature map of the multi-attention module.

[0155] In one implementation of the image fusion device 200, the branch network includes a multi-receptive field module, which includes m feature extraction branches and a feature fusion unit; where m is an integer greater than 1, each feature extraction branch is used to extract features from the input feature map of the multi-receptive field module to obtain its own output feature map, and the receptive fields corresponding to each feature extraction branch are different. The feature fusion unit is used to fuse the output feature maps of the m feature extraction branches to obtain the output feature map of the multi-receptive field module.

[0156] In one implementation of the image fusion device 200, at least m-1 of the m feature extraction branches include dilated convolutional layers; wherein, when m is greater than 2, the dilation rates of the dilated convolutional layers in the at least m-1 feature extraction branches are different; when there are m-1 feature extraction branches including dilated convolutional layers, the remaining feature extraction branch among the m feature extraction branches includes only a regular convolutional layer or a direct-connection branch.

[0157] In one implementation of the image fusion device 200, each of the at least m-1 feature extraction branches is used to calculate the weight of its own dilated convolutional layer's output feature map in the spatial dimension, and multiply the calculated weight by the data at different spatial locations of its own dilated convolutional layer's output feature map to obtain its own output feature map.

[0158] In one implementation of the image fusion device 200, the branch network includes a multi-receptive field module, which includes m feature extraction units connected in sequence; where m is an integer greater than 1. The first feature extraction unit among the m feature extraction units is used to extract features from the input feature map of the multi-receptive field module to obtain its own output feature map. Each subsequent feature extraction unit is used to extract features from the output feature map of the previous feature extraction unit to obtain its own output feature map. The output feature map of the last feature extraction unit is the output feature map of the multi-receptive field module, and each feature extraction unit corresponds to a different receptive field.

[0159] In one implementation of the image fusion device 200, the backbone network is further used to downsample the n-frame channel images; the image fusion component 220 calculates the n-frame fusion mask corresponding to the n-frame channel images using the n branch networks in the neural network model, including: calculating the n-frame low-resolution fusion mask corresponding to the n-frame channel images using the n branch networks in the neural network model; and upsampling the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel images.

[0160] In one implementation of the image fusion device 200, the image fusion component 220 upsamples the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel images. This includes: performing edge-preserving smoothing filtering and upsampling on the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel images.

[0161] In one implementation of the image fusion device 200, if each frame of the image to be fused includes multiple channels, and the multiple channels are not all target channels, then the image fusion component 220 uses the fusion mask calculated when fusing the channel images of the target channels to perform weighted fusion on the n frame channel images belonging to the same non-target channel in the n frame images to be fused.

[0162] The image fusion apparatus 200 provided in this application embodiment has been described in the foregoing method embodiment in terms of its implementation principle and the resulting technical effects. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the method embodiment.

[0163] Figure 9 The structure of the electronic device 300 provided in an embodiment of this application is shown. (Refer to...) Figure 9 The electronic device 300 includes a processor 310, a memory 320, and a communication interface 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0164] The processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Graphics Processing Unit (GPU), a Neural-network Processing Unit (NPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 310, some can be general-purpose processors and others can be special-purpose processors.

[0165] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0166] Processor 310 and other possible components may access memory 320, reading and / or writing data therein. In particular, one or more computer program instructions may be stored in memory 320, which processor 310 may read and execute to implement the image fusion method provided in the embodiments of this application.

[0167] Communication interface 330 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices to exchange data. Communication interface 330 may include interfaces for wired and / or wireless communication.

[0168] Understandable. Figure 9 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 Different configurations are shown. For example, electronic device 300 may also include a camera for capturing images or videos, and frames from the captured images or videos can be used as the images to be fused in step S110; or, for example, if electronic device 300 does not need to communicate with other devices, it may not need to provide a communication interface 330.

[0169] Figure 9 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a mobile phone, camera, camcorder, PC, laptop, tablet, server, robot, etc., or a virtual device, such as a virtual machine, container, etc. Furthermore, electronic device 300 is not limited to a single device; it can also be a combination of multiple devices or a cluster of a large number of devices.

[0170] This application also provides a computer-readable storage medium storing computer program instructions. These instructions are read and executed by a processor to perform the image fusion method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 9 The memory 320 in the electronic device 300.

[0171] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the image fusion method provided in this application.

[0172] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that, include: Obtain n frames of images to be fused that have different exposure levels; where n is an integer greater than 1, and each frame of images to be fused includes at least one channel, and the at least one channel includes the target channel; The n-frame channel images belonging to the same channel in the n-frame images to be fused are fused to obtain at least one channel fused image, and the fusion result image is determined based on the at least one channel fused image; wherein, for the n-frame channel images belonging to the same target channel in the n-frame images to be fused, their corresponding channel fused images are calculated through the following steps: The basic feature map of the n-frame channel image is extracted using the backbone network in the pre-trained neural network model, and the basic feature map is split into n-frame sub-feature maps; The n-frame fusion mask corresponding to the n-frame channel images is calculated using the n-branch network in the neural network model; wherein, each branch network is used to calculate the fusion mask corresponding to the channel image based on a sub-feature map of a frame, and each fusion mask includes the weights used for fusion of the corresponding channel image. The n-frame fusion mask is used to perform weighted fusion on the n-frame channel images to obtain the channel fused image.

2. The image fusion method according to claim 1, characterized in that, The backbone network is a network with an attention mechanism, which includes at least one of channel attention, frame attention, and spatial attention; and / or, the branch network is a network with an attention mechanism, which includes at least one of channel attention and spatial attention. Here, channel attention refers to assigning corresponding weights to data in different channels of the feature map, frame attention refers to assigning corresponding weights to data in different frames of the feature map, and spatial attention refers to assigning corresponding weights to data at different spatial locations in the feature map.

3. The image fusion method according to claim 2, characterized in that, The backbone network includes a multi-attention module, which includes channel attention units and frame attention units connected in sequence. The channel attention unit is used to calculate the weights of the input feature map of the multi-attention module in the channel dimension, and multiply the calculated weights by the data in different channels of the input feature map of the multi-attention module to obtain the output feature map of the channel attention unit. The frame attention unit is used to transpose the channel dimension and frame dimension in the output feature map of the channel attention unit, calculate the weight of the transposed feature map in the channel dimension, multiply the calculated weight with the data in different channels of the transposed feature map to obtain a weighted feature map, and transpose the channel dimension and frame dimension in the weighted feature map to obtain the output feature map of the multi-attention module.

4. The image fusion method according to any one of claims 1-3, characterized in that, The branch network includes a multi-receptive field module, which includes m feature extraction branches and a feature fusion unit. Where m is an integer greater than 1, each feature extraction branch is used to extract features from the input feature map of the multi-receptive field module to obtain its own output feature map, and the receptive fields corresponding to each feature extraction branch are different. The feature fusion unit is used to fuse the output feature maps of the m feature extraction branches to obtain the output feature map of the multi-receptive field module.

5. The image fusion method according to claim 4, characterized in that, At least m-1 of the m feature extraction branches include dilated convolutional layers; Where m is greater than 2, the void ratios of the dilated convolutional layers in the at least m-1 feature extraction branches are different; When there are m-1 feature extraction branches including dilated convolutional layers, the remaining feature extraction branch among the m feature extraction branches only includes ordinary convolutional layers or is a direct connection branch.

6. The image fusion method according to claim 5, characterized in that, Each of the at least m-1 feature extraction branches is used to calculate the weight of its own dilated convolutional layer's output feature map in the spatial dimension, and multiply the calculated weight by the data at different spatial locations of its own dilated convolutional layer's output feature map to obtain its own output feature map.

7. The image fusion method according to any one of claims 1-3, characterized in that, The branch network includes a multi-receptive field module, which includes m feature extraction units connected in sequence. Where m is an integer greater than 1, the first feature extraction unit in the m feature extraction units is used to extract features from the input feature map of the multi-receptive field module to obtain its own output feature map, each subsequent feature extraction unit is used to extract features from the output feature map of the previous feature extraction unit to obtain its own output feature map, and the output feature map of the last feature extraction unit is the output feature map of the multi-receptive field module, and each feature extraction unit corresponds to a different receptive field.

8. The image fusion method according to claim 1, characterized in that, The backbone network is also used to downsample the n-frame channel images; The step of calculating the n-frame fusion mask corresponding to the n-frame channel images using the n-branch network in the neural network model includes: The n-frame low-resolution fusion mask corresponding to the n-frame channel images is calculated using the n-branch network in the neural network model; Upsample the n-frame low-resolution fusion mask to obtain the n-frame fusion mask with the same resolution as the n-frame channel image.

9. The image fusion method according to claim 8, characterized in that, The step of upsampling the low-resolution fusion mask of the n frames to obtain the n-frame fusion mask with the same resolution as the n-frame channel image includes: The n-frame low-resolution fusion mask is subjected to edge-preserving smoothing filtering and upsampling to obtain the n-frame fusion mask with the same resolution as the n-frame channel image.

10. The image fusion method according to claim 1, characterized in that, If each frame of the image to be fused includes multiple channels, and not all of the multiple channels are target channels, then the fusion mask calculated when fusing the channel images of the target channels is used to perform weighted fusion on the n frame channel images of the n frames of the image to be fused that belong to the same non-target channel.

11. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-10.

13. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores computer program instructions, which are read and executed by the processor to perform the method of any one of claims 1-10.

Images