Image acquisition method, device, apparatus, and non-transitory computer storage medium

Abstract

The present application discloses an image acquisition method, apparatus, device, and computer storage medium, belonging to the field of image technology. The method fuses multiple original images with different exposures obtained by shooting the same scene to obtain a high dynamic range image corresponding to the multiple original images. Among them, by introducing a first attention network into the fusion process of multiple original images, and using the first attention network to evaluate the importance of different image features, highlighting image features that are beneficial to obtaining high dynamic range images, and suppressing interference features caused by motion, the impact of motion on image fusion during the process of shooting multiple original images can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image acquisition method, apparatus, device, and non-transient computer storage medium. Background Art

[0002] High Dynamic Range (HDR) imaging is a technology used to achieve a greater dynamic range (i.e., greater differences between light and dark) than conventional digital imaging techniques. The goal of HDR imaging is to accurately represent the range of real-world brightness, from direct sunlight to the darkest shadows. The resulting HDR images provide greater dynamic range and detail.

[0003] In an image acquisition method, multiple original images are first acquired, each original image is then converted into a feature map, and feature fusion is performed on the multiple feature maps to obtain a high dynamic range image containing rich information.

[0004] However, in the above method, when there is slight motion during the process of capturing the multiple original images, the clarity of the high dynamic range image obtained from the multiple original images is poor. Summary of the Invention

[0005] The embodiment of the present application provides an image acquisition method. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for acquiring an image is provided, the method comprising:

[0007] Acquire multiple original images captured from the same scene, the multiple original images having different exposure levels;

[0008] fusing the plurality of original images through a fusion network to obtain a first fused feature map, wherein the fusion network includes a first attention network;

[0009] Acquire an adjusted feature map based on the first fused feature map and the multiple original images;

[0010] Based on the adjusted feature map, high dynamic range images corresponding to the multiple original images are obtained.

[0011] Optionally, the fusion network includes a first fusion sub-network and a second fusion sub-network, and both the first fusion sub-network and the second fusion sub-network include a first attention network;

[0012] The fusing the multiple original images through a fusion network to obtain a first fusion feature map includes:

[0013] determining one of the plurality of original images as a reference image;

[0014] Performing feature extraction on the reference image to obtain a reference feature map;

[0015] performing feature extraction on a plurality of the original images excluding the reference image among the plurality of original images respectively to obtain a plurality of original feature maps;

[0016] The reference feature map is respectively fused with the multiple original feature maps through the first fusion sub-network to obtain a second fused feature map corresponding to each of the original feature maps;

[0017] The plurality of second fused feature maps are fused through a second fusion sub-network to obtain the first fused feature map.

[0018] Optionally, fusing the reference feature map with the multiple original feature maps respectively through the first fusion sub-network to obtain a second fused feature map corresponding to each of the original feature maps includes:

[0019] Downsampling the reference feature map to obtain a first reference feature map;

[0020] Performing downsampling processing on the original feature map to obtain a first original feature map;

[0021] Merging the first reference feature map and the first original feature map, and inputting the merged features into the first attention network to obtain a first attention feature map;

[0022] The first attention feature map is upsampled to obtain the second fusion feature map.

[0023] Optionally, acquiring an adjusted feature map based on the first fused feature map and the multiple original images includes:

[0024] determining one of the plurality of original images as a reference image;

[0025] performing feature extraction on a plurality of original images excluding the reference image from the plurality of original images respectively to obtain a plurality of original feature maps;

[0026] Inputting the first fused feature map into a second attention network to obtain a second attention feature map;

[0027] Inputting the second attention feature maps into a plurality of feature transformation networks respectively, and inputting the plurality of original feature maps into the plurality of spatial feature transformation networks respectively, to obtain a plurality of adjusted feature maps corresponding to the plurality of original feature maps;

[0028] Among them, the spatial feature transformation network is used to determine the spatial parameter matrix based on the original feature map, and use the spatial parameter matrix to perform spatial transformation on the second attention feature map, and the spatial parameter matrix is ​​obtained by convolution processing on the original feature map.

[0029] Optionally, acquiring high dynamic range images corresponding to the multiple original images based on the adjusted feature map includes:

[0030] Obtaining a plurality of adjusted feature maps corresponding to the plurality of original feature maps;

[0031] Merging the multiple adjusted feature maps to obtain a merged feature map;

[0032] Inputting the merged feature map into a third attention network to obtain an important feature map;

[0033] Performing dimensionality reduction processing on the important feature map through at least one convolutional layer to obtain the high dynamic range image.

[0034] Optionally, determining one of the multiple original images as a reference image includes:

[0035] sorting the plurality of original images according to exposure from high to low;

[0036] An original image with a medium exposure is selected as the reference image.

[0037] Optionally, extracting features from the reference image to obtain a reference feature map includes:

[0038] The reference image is input into a residual densely connected network to obtain the reference feature map.

[0039] Optionally, before acquiring multiple original images by photographing the same shooting scene multiple times, the method further includes:

[0040] Acquire a training sample from a sample set, where the training set includes a plurality of training samples, the training sample including a plurality of sample images captured from the same scene, and a target sample high dynamic range image corresponding to the plurality of sample images, where the plurality of sample images have different exposure levels;

[0041] fusing the plurality of sample images through a fusion network to be trained to obtain a first fusion feature map, wherein the fusion network includes a first attention network to be trained;

[0042] Acquire an adjusted feature map based on the first fused feature map and the multiple sample images;

[0043] Based on the adjusted feature map, obtaining high dynamic range images corresponding to the multiple sample images;

[0044] Comparing the high dynamic range images corresponding to the multiple sample images with the target sample high dynamic range image to obtain a comparison difference;

[0045] If the contrast difference is greater than a preset result, adjusting the fusion network to be trained based on the contrast difference, and executing the step of training samples from the sample set;

[0046] If the comparison difference is less than or equal to the preset result, the to-be-trained fusion network is determined as the fusion network.

[0047] According to another aspect of the present application, there is provided an image acquisition device, the device comprising:

[0048] An acquisition module is used to acquire multiple original images obtained by shooting the same scene, wherein the multiple original images have different exposure levels;

[0049] a fusion module, configured to fuse the multiple original images through a fusion network to obtain a first fused feature map, wherein the fusion network includes a first attention network;

[0050] an adjustment module, configured to obtain an adjusted feature map based on the first fused feature map and the plurality of original images;

[0051] A reconstruction module is used to obtain high dynamic range images corresponding to the multiple original images based on the adjusted feature map.

[0052] Optionally, the fusion module includes:

[0053] A first determining module, configured to determine one of the plurality of original images as a reference image;

[0054] A first feature extraction module is used to extract features from the reference image to obtain a reference feature map;

[0055] a second feature extraction module, configured to extract features from a plurality of the original images excluding the reference image, to obtain a plurality of original feature maps;

[0056] A first fusion submodule is configured to fuse the reference feature map with the multiple original feature maps respectively through the first fusion subnetwork to obtain a second fused feature map corresponding to each of the original feature images;

[0057] The second fusion submodule is used to fuse the multiple second fusion feature maps through the second fusion subnetwork to obtain the first fusion feature map.

[0058] Optionally, the first fusion submodule is used to:

[0059] Downsampling the reference feature map to obtain a first reference feature map;

[0060] Performing downsampling processing on the original feature map to obtain a first original feature map;

[0061] Merging the first reference feature map and the first original feature map, and inputting the merged features into the first attention network to obtain a first attention feature map;

[0062] The first attention feature map is upsampled to obtain the second fusion feature map.

[0063] According to another aspect of the present application, an image acquisition device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the image acquisition method as described above.

[0064] According to another aspect of the present application, a non-volatile computer storage medium is provided, wherein the non-volatile computer storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image acquisition method as described above.

[0065] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image acquisition method.

[0066] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0067] Provided is an image acquisition method for fusing multiple original images of the same scene at different exposure levels to obtain a high dynamic range image corresponding to the multiple original images. This method incorporates a first attention network into the fusion process of the multiple original images and utilizes it to assess the importance of different image features, highlighting image features that are beneficial for obtaining the high dynamic range image and suppressing interfering features caused by motion. This method can mitigate the impact of motion on image fusion during the capture of the multiple original images, address the poor clarity of the high dynamic range images obtained in related technologies, and ultimately improve the clarity of the obtained high dynamic range images. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 This is a flow chart of an image acquisition method provided in an embodiment of the present application;

[0070] Figure 2 is a flowchart of another image acquisition method provided in an embodiment of the present application;

[0071] Figure 3 This is a schematic diagram of the structure of a residual densely connected network provided in an embodiment of the present application;

[0072] Figure 4 This is a schematic diagram of the structure of a first fusion sub-network provided in an embodiment of the present application;

[0073] Figure 5 This is a flow chart for obtaining a second fused feature map provided by an embodiment of the present application;

[0074] Figure 6 This is a schematic diagram of the structure of a first attention network provided by an embodiment of the present application;

[0075] Figure 7 This is a schematic diagram of the structure of a space transformation network provided in an embodiment of the present application;

[0076] Figure 8 This is a flow chart of obtaining a high dynamic range image provided by an embodiment of the present application;

[0077] Figure 9 This is a flow chart of a training fusion network provided by an embodiment of the present application;

[0078] Figure 10 This is a network architecture diagram provided by an embodiment of the present application;

[0079] Figure 11 This is a structural block diagram of an image acquisition device provided in an embodiment of the present application.

[0080] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0082] First, the application scenarios involved in the embodiments of the present application are introduced.

[0083] With the development of imaging technology, the issue of low image dynamic range has also received increasing attention. Dynamic range refers to the range of brightness that a device can capture or display, i.e., the range from the darkest black to the brightest white. To address this issue, high dynamic range imaging technology has rapidly developed. For example, some scenes that need to be captured have a large dynamic range, such as landscapes, night scenes, and indoor photography. These scenes are often relatively stable but have large contrasts between light and dark, making it impossible for the camera to capture all the details in a single shot. The process of acquiring a high dynamic range image can include: first, capturing multiple images of the same scene at different exposures, where each image captures a portion of the scene's full dynamic range. These multiple images are referred to as low dynamic range images, and then combining the multiple low dynamic range images into a single high dynamic range image. Compared to low dynamic range images, high dynamic range images can provide a wider dynamic range and image details, and can better reflect the visual effects of real environments.

[0084] It should be noted that the application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0085] The implementation environment may include a shooting scene, a shooting component, a server, and a display terminal. The shooting scene may include plants, scenery, etc. The shooting component may include a camera. The server includes a processor, and the server may establish a wired or wireless connection with the shooting component to generate a high dynamic range image based on the image captured by the shooting component and display the high dynamic range image on the display terminal.

[0086] Figure 1 This is a flow chart of an image acquisition method provided in an embodiment of the present application. This method can be applied to the server in the above implementation environment. The method may include the following steps:

[0087] Step 101: Acquire multiple original images captured of the same scene, where the multiple original images have different exposure levels.

[0088] Step 102: fuse multiple original images through a fusion network to obtain a first fusion feature map, where the fusion network includes a first attention network.

[0089] Step 103: Obtain an adjusted feature map based on the first fused feature map and the multiple original images.

[0090] Step 104: Based on the adjusted feature map, obtain high dynamic range images corresponding to the multiple original images.

[0091] In summary, the embodiments of the present application provide an image acquisition method, which obtains high dynamic range images corresponding to the multiple original images by fusing multiple original images with different exposures obtained by shooting the same scene. Among them, by introducing the first attention network into the fusion process of multiple original images, and using the first attention network to evaluate the importance of different image features, the image features that are beneficial to obtaining high dynamic range images are highlighted, and the interference features caused by motion are suppressed. In this way, the impact of motion on image fusion during the process of shooting multiple original images can be reduced, and the problem of poor clarity of high dynamic range images obtained in related technologies can be solved, thereby achieving the effect of improving the clarity of the obtained high dynamic range images.

[0092] Figure 2 This is a flow chart of another image acquisition method provided by an embodiment of the present application. This method can be applied to the server in the above implementation environment. The method may include the following steps:

[0093] Step 201: Acquire multiple original images captured from the same scene, where the multiple original images have different exposure levels.

[0094] By continuously shooting the same scene, you can obtain multiple original images with different exposures. For example, you can use a camera to quickly adjust the exposure to shoot the same scene in a short period of time to obtain multiple original images with different exposures. For example, you can select HDR mode in the camera and shoot the same scene three times in a row to obtain a low-exposure original image, a medium-exposure original image, and a high-exposure original image. The original image with a higher exposure has a higher brightness, while the original image with a lower exposure has a darker brightness.

[0095] Optionally, in the embodiment of the present application, the number of the multiple original images may be N, where N is an integer greater than or equal to 3.

[0096] Step 202: Determine one of the multiple original images as a reference image.

[0097] The original image with the middle exposure among multiple original images can be used as a reference image. The exposure of the reference image is more balanced and the image details are richer. The well-exposed parts are obtained from multiple original images with low exposure (underexposed compared to the reference image) and high exposure (overexposed compared to the reference image) and fused with the reference image. In this way, the obtained high dynamic range image can record relevant image details in both brighter highlights and darker shadows, making the high dynamic range image closer to the real scene seen by the human eye.

[0098] Optionally, multiple original images can be sorted from high to low according to exposure. The multiple original images can all be low dynamic range images. For example, three original images are sorted from high to low according to exposure as a first original image, a second original image, and a third original image.

[0099] Afterwards, an original image with a middle exposure level is selected from the sorted plurality of original images as a reference image, for example, the second original image is selected as the reference image.

[0100] Step 203: extract features from the reference image to obtain a reference feature map.

[0101] The reference image may be subjected to feature extraction by a feature extraction network, for example, by performing feature extraction on the reference image through convolution processing, thereby converting the reference image into a reference feature map. In the embodiment of the present application, the feature extraction network may be a residual dense block (RDB). During the process of extracting the reference feature map, the reference image may be input into the residual dense block to output the reference feature map.

[0102] like Figure 3 As shown, Figure 3This is a schematic diagram of the structure of a residual dense connection network provided by an embodiment of the present application. Conv represents a convolutional layer, C represents Concat, which is used to represent the merging or permutation operation of matrices, and R represents the Relu activation function. The Relu activation function is a linear rectification function. When the input is less than 0, the output is 0; when the input is greater than 0, the output is the input value. This activation function can make the network converge faster and reduce the interdependence of parameters. d-1 represents the input of the dth residual densely connected network, F d represents the output of the d-th residual densely connected network, F d,1 represents the output of the first convolutional layer in the dth residual densely connected network, F d,c represents the output of the cth convolutional layer in the dth residual densely connected network, F d,LF Represents local feature fusion.

[0103] The input of the residual dense connection network is a reference image, and the output is a reference feature map, which can be a matrix. Figure 3 The residual dense network shown in the figure can include three 3×3 convolutional layers and three activation layers (ReLU). It can be understood that the residual dense network can be considered a combination of the residual network structure and the dense network structure. Since there are multiple convolutional layers in the residual dense network, introducing the residual network structure can further improve the information flow between multiple convolutional layers and reduce the training difficulty of the residual dense network. The dense network structure can effectively alleviate the problem of vanishing gradients, strengthen feature propagation, and encourage feature reuse to improve the utilization of parameters in the residual dense network and reduce unnecessary computation.

[0104] Step 204 : performing feature extraction on the multiple original images excluding the reference image in the multiple original images to obtain multiple original feature maps.

[0105] In the process of extracting the reference feature map, multiple original images other than the reference image can be respectively input into the residual dense connection network to output multiple original feature maps. The residual dense connection network has the same structure as the residual dense connection network in step 203. In addition, the number of residual dense connection networks in this embodiment is determined according to the number of original images, that is, the number of residual dense networks is the same as the number of multiple original images. Exemplarily, in this embodiment of the application, there can be three residual dense networks.

[0106] Since the exposure levels of multiple original images are different, the brightness, contrast, texture, and contour information of the same scene in the original images with different exposure levels are different. If the same residual dense network is used to extract features from multiple original images, the generated shared parameters will destroy the inherent characteristics of the scene at different exposure levels. Therefore, a multi-channel architecture can be used in the feature extraction process, and no learning parameters are shared between multiple residual dense networks, and multiple residual dense networks can simultaneously extract features from multiple original images. That is, step 203 and step 204 can be executed simultaneously.

[0107] Step 205: Acquire the fused network.

[0108] The fusion network may be a trained fusion network, which may include multiple convolutional layers and a first attention network, wherein the first attention network may also be a trained first attention network.

[0109] Step 206: The reference feature map is fused with the multiple original feature maps respectively through the first fusion sub-network in the fusion network to obtain a second fused feature map corresponding to each original feature map.

[0110] Optionally, in an embodiment of the present application, the fusion network may include a first fusion sub-network and a second fusion sub-network, each of which may include a first attention network. The first fusion sub-network and the second fusion sub-network in the fusion network may have the same structure and may both be referred to as spatial adaptation networks (SAMs). No learning parameters are shared between the multiple fusion sub-networks.

[0111] like Figure 4 As shown, Figure 4 : This is a structural diagram of a first fusion sub-network provided in an embodiment of the present application. The input of the first fusion sub-network is a reference feature map and one of the multiple original feature maps, and the output is a second fusion feature map. Conv represents a convolution layer, C represents a matrix merge or permutation operation, and SConv represents a strided convolution. Here, the stride of the strided convolution is 2, that is, the reference feature map and the original feature map are reduced by 2 times after the strided convolution. Therefore, in the subsequent image processing process, up-sampling can be performed to restore the reference feature map and the original feature map.

[0112] During the image fusion process, since the exposure levels of the multiple input original images are different and not completely aligned, directly merging the reference feature map and the original feature map together and inputting the merged features into the subsequent convolutional layer may cause misalignment between the merged features. Therefore, in the embodiment of the present application, a first attention network is introduced during the image fusion process to highlight important image features that are beneficial to fusion and suppress image features of low-quality areas such as misalignment, ghosting, undersaturation, and oversaturation.

[0113] like Figure 5 As shown, step 206 may include the following four sub-steps:

[0114] Sub-step 2061: downsample the reference feature map to obtain a first reference feature map.

[0115] Downsampling the reference feature map can reduce the dimension of the reference feature map and retain valid information to avoid overfitting.

[0116] Sub-step 2062: downsample the original feature map to obtain a first original feature map.

[0117] Similarly, downsampling the original feature map can reduce the dimension of the reference feature map while retaining effective information, thereby avoiding overfitting. Step 2051 and step 2052 can be performed simultaneously to improve the efficiency of image fusion.

[0118] Sub-step 2063: merge the first reference feature map and the first original feature map, and input them into the first attention network to obtain the first attention feature map.

[0119] like Figure 6 As shown, Figure 6 This is a structural diagram of a first attention network provided by an embodiment of the present application. Wherein, Conv represents a convolutional layer, SConv represents strided convolution, Max-Pool represents maximum pooling, Concat represents a matrix merge or permutation operation, DConv represents a dilated convolution, DWConv represents a depthwise convolution, and S represents a Sigmoid activation function. The Sigmoid activation function is a logical activation function, also known as an S-shaped growth curve. The Sigmoid function can be used as an activation function of a neural network to map variables to between [0, 1]. Dilated convolution can expand the receptive field and help restore the loss of image details caused by oversaturated areas and motion dislocation.

[0120] It should be noted that the structure of the first attention network in the embodiment of the present application can be as follows: Figure 6 The first attention network shown may also be an attention network with other structures, which is not limited in this embodiment of the present application.

[0121] It should be noted that the reference feature map and the original feature map in the embodiment of the present application are essentially a matrix. Figure 4 In the illustrated embodiment, the merging of the first reference feature map and the first original feature map is essentially the merging of two matrices, which is a process of arranging or merging the two matrices without changing the order of the two matrices themselves.

[0122] Sub-step 2064: upsample the first attention feature map to obtain a second fusion feature map.

[0123] The above processing can be used to enlarge the image, and the second fused feature map can be a feature map with weight values.

[0124] Step 207: fuse the multiple second fused feature maps through the second fusion sub-network in the fusion network to obtain a first fused feature map.

[0125] The first fusion sub-network is similar to the feature extraction network and can also adopt a multi-channel design with the same structure. That is, multiple original feature maps can be fused with the reference feature map for the first time to obtain multiple second fused feature maps, and then the multiple second fused feature maps are fused for the second time. The input of the second fusion sub-network is the multiple second fused feature maps, and the output is the first fused feature map.

[0126] Step 208: Input the first fused feature map into the second attention network to obtain a second attention feature map.

[0127] The structure of the second attention network can be Figure 6 The structure of the first attention network shown is the same. The second attention network can be used to process the first fused feature map to retain the important image information in the first fused feature map and highlight the image features that are beneficial to fusion.

[0128] Step 209: Input the second attention feature map into multiple feature transformation networks respectively, and input the multiple original feature maps into multiple spatial feature transformation networks respectively, to obtain multiple adjusted feature maps corresponding to the multiple original feature maps.

[0129] Among them, the spatial feature transformation network is used to determine the spatial parameter matrix based on the original feature map, and use the spatial parameter matrix to perform spatial transformation on the second attention feature map. The spatial parameter matrix can be obtained by convolution processing on the original feature map.

[0130] like Figure 7 As shown, Figure 7: This is a schematic diagram of the structure of a spatial transformation network provided by an embodiment of the present application. Conv represents a convolutional layer, and C represents a matrix merge or permutation operation. The spatial feature transformation network is similar to the feature extraction network. The spatial feature transformation network can also adopt a multi-channel design with the same structure, that is, multiple original feature maps are spatially transformed into the second attention feature map to obtain multiple adjusted feature maps corresponding to the multiple original feature maps.

[0131] Exemplarily, the number of original feature maps and the number of spatial feature transformation networks may be equal, and the number of original feature maps and the number of spatial feature transformation networks may both be greater than or equal to 2.

[0132] The spatial feature transformation network can modulate the second attention feature map through the spatial parameter matrix so that the modulated second attention feature map has more features related to the image texture, which can correct the local distortion and information loss of the second attention feature map.

[0133] Step 210: Based on the adjusted feature map, obtain high dynamic range images corresponding to the multiple original images.

[0134] Since the feature information in the multiple original feature maps is different, a high dynamic range image with more image details can be obtained based on the multiple adjusted feature maps by obtaining multiple adjusted feature maps corresponding to the multiple original images.

[0135] like Figure 8 As shown, step 210 may include the following four sub-steps:

[0136] Sub-step 2101: Obtain multiple adjusted feature maps corresponding to multiple original feature maps.

[0137] Since the feature information in multiple original feature maps is different, the second attention feature map can be spatially transformed by multiple spatial parameter matrices generated by multiple original features to improve the accuracy of image spatial transformation.

[0138] Sub-step 2102: Merge multiple adjustment feature maps to obtain a merged feature map.

[0139] The above-mentioned multiple adjusted feature maps can be merged through Concat so that the merged feature map has more image details.

[0140] Sub-step 2103: Input the merged feature map into the third attention network to obtain an important feature map.

[0141] The structure of the third attention network can be Figure 6The structure of the first attention network shown in the figure is the same. Through the third attention network, the importance of different image features in the merged feature map can be evaluated again, highlighting image features that are beneficial for obtaining high dynamic range images and suppressing interfering features caused by motion. This can reduce the impact of motion on image fusion during the process of capturing multiple original images and improve the clarity of the obtained high dynamic range images.

[0142] Sub-step 2104: performing dimensionality reduction processing on the important feature map through at least one convolutional layer to obtain a high dynamic range image.

[0143] There may be redundant information in the above important feature maps. We can use a combination of two 3x3 convolutions and activation functions to reduce the dimensionality of the important feature maps to obtain a high dynamic range image.

[0144] In summary, the embodiments of the present application provide an image acquisition method, which obtains high dynamic range images corresponding to the multiple original images by fusing multiple original images with different exposures obtained by shooting the same scene. Among them, by introducing the first attention network into the fusion process of multiple original images, and using the first attention network to evaluate the importance of different image features, the image features that are beneficial to obtaining high dynamic range images are highlighted, and the interference features caused by motion are suppressed. In this way, the impact of motion on image fusion during the process of shooting multiple original images can be reduced, and the problem of poor clarity of high dynamic range images obtained in related technologies can be solved, thereby achieving the effect of improving the clarity of the obtained high dynamic range images.

[0145] Optionally, the fusion network in step 205 can be a pre-trained fusion network, or the fusion network can be trained in step 205. It should be noted that the various networks used in the embodiments of the present application (fusion network, first attention network, second attention network, third attention network, and feature transformation network) are all trained network structures, and each network can be trained through deep learning, which is a method of machine learning. The embodiments of the present application do not limit the training methods of these networks.

[0146] The training of the fusion network can use low dynamic range images as input, and the high dynamic range images corresponding to the low dynamic range images as the true value. During the training process, low dynamic range images can be randomly extracted from the sample library and input into the fusion network for training. The network optimizer can use the Adam optimizer with an initial learning rate of 1e-4. The loss function (Loss) includes the L1 loss and PSNR loss of the low dynamic range image and the high dynamic range image. The formula of the loss function (Loss) is as follows:

[0147]

[0148] The L1 loss, also known as the minimum absolute deviation, calculates the sum of the absolute differences between the actual value and the target value. PSNR is the peak signal-to-noise ratio, which is used to measure the difference between two images. Represents the true value, I gt Represents the test value during training.

[0149] like Figure 9 As shown, in the embodiment of the present application, the training process of the fusion network may include the following steps:

[0150] Step 301: Obtain training samples from a sample set.

[0151] The training set includes multiple training samples, which include multiple sample images obtained by shooting the same scene, and target sample high dynamic range images corresponding to the multiple sample images, and the multiple sample images have different exposures.

[0152] The target sample high dynamic range image is a high-definition high dynamic range image formed by fusing multiple training samples.

[0153] Step 302: fuse multiple sample images through the fusion network to be trained to obtain a first fusion feature map, where the fusion network includes a first attention network to be trained.

[0154] In sub-step 302, a residual densely connected network is included for extracting features from a plurality of samples. During the training process, the residual densely connected network can be trained synchronously with the fusion network.

[0155] Step 303: Obtain an adjusted feature map based on the first fused feature map and multiple sample images.

[0156] In step 303, a second attention network and a spatial feature transformation network may be included. Similarly, the second attention network and the spatial feature transformation network may be trained synchronously with the fusion network.

[0157] Step 304: Based on the adjusted feature map, obtain high dynamic range images corresponding to the multiple sample images.

[0158] In step 304, a third attention network may be included. Similarly, the third attention network may be trained synchronously with the fusion network.

[0159] Step 305: Compare the high dynamic range images corresponding to the multiple sample images with the target sample high dynamic range image to obtain a comparison difference.

[0160] Step 306: If the comparison difference is greater than the preset result, the fusion network to be trained is adjusted based on the comparison difference, and the step of training samples from the sample set is executed.

[0161] If the acquired high dynamic range image is significantly different from the target sample high dynamic range image, it may indicate that the parameters in the fusion network are not accurate enough, and the parameters in the fusion network can be further adjusted through multiple trainings. That is, sub-step 301 is executed after sub-step 306.

[0162] Step 307: If the comparison difference is less than or equal to the preset result, the fusion network to be trained is determined as the fusion network.

[0163] If the acquired high dynamic range image has little difference compared with the target sample high dynamic range image, it can be said that the parameters in the fusion network are relatively accurate and the training of the fusion network can be ended.

[0164] Alternatively, as Figure 10 As shown, Figure 10 : is a network architecture diagram provided by an embodiment of the present application, wherein the multiple inputs (input 1, input 2, and input 3) can be multiple original images with different exposures, wherein input 2 can be a reference image.

[0165] Multiple feature extraction modules (feature extraction module 1, feature extraction module 2 and feature extraction module 3) can be used to extract features from multiple primitives. For example, the above-mentioned Figure 2 Step 203 and step 204 in the embodiment shown.

[0166] The fusion module 1 and the fusion module 2 in the multiple fusion modules can be used to respectively fuse the multiple original feature maps with the reference feature maps for the first time to obtain multiple second fusion feature maps. For example, the above-mentioned Figure 2 In the embodiment shown in step 206, the fusion module 3 in the plurality of fusion modules can be used to fuse a plurality of second fusion feature maps, for example, the above-mentioned Figure 2 Step 207 in the embodiment shown.

[0167] Multiple spatial feature transformation modules (spatial feature transformation module 1 and spatial feature transformation module 2) and multiple attention modules (attention module 1 and attention module 2) can be used to adjust the fused image features according to the image features of the original image, for example, the above-mentioned Figure 2 Steps 208 and 209 in the embodiment shown.

[0168] C represents a matrix merging or permutation operation.

[0169] The attention module 3 and the output module can be used to obtain high dynamic range images corresponding to multiple original images based on the adjusted feature map, such as the above Figure 2 Step 210 in the embodiment shown.

[0170] Figure 11 1 is a structural block diagram of an image acquisition device provided in an embodiment of the present application. The image acquisition device 1100 includes:

[0171] An acquisition module 1110 is configured to acquire multiple original images obtained by photographing the same scene, where the multiple original images have different exposure levels.

[0172] a fusion module 1120 for fusing the multiple original images through a fusion network to obtain a first fused feature map, wherein the fusion network includes a first attention network;

[0173] An adjustment module 1130 is configured to obtain an adjusted feature map based on the first fused feature map and the multiple original images;

[0174] The reconstruction module 1140 is configured to obtain high dynamic range images corresponding to the multiple original images based on the adjusted feature maps.

[0175] Optionally, the fusion module 1120 includes:

[0176] A first determining module, configured to determine one of the plurality of original images as a reference image;

[0177] A first feature extraction module is used to extract features from a reference image to obtain a reference feature map;

[0178] a second feature extraction module, configured to extract features from a plurality of original images excluding the reference image, to obtain a plurality of original feature maps;

[0179] A first fusion submodule is configured to fuse the reference feature map with the multiple original feature maps through the first fusion subnetwork to obtain a second fused feature map corresponding to each original feature image;

[0180] The second fusion submodule is used to fuse the multiple second fusion feature maps through the second fusion subnetwork to obtain a first fusion feature map.

[0181] Optionally, the first fusion submodule is used to:

[0182] Downsampling the reference feature map to obtain a first reference feature map;

[0183] Downsampling the original feature map to obtain a first original feature map;

[0184] Merge the first reference feature map and the first original feature map, and input them into the first attention network to obtain the first attention feature map;

[0185] The first attention feature map is upsampled to obtain the second fusion feature map.

[0186] Optionally, the adjustment module 1130 includes:

[0187] The first determining module is used to determine an original image among the multiple original images as a reference image, and perform feature extraction on each original image among the multiple original images except the reference image to obtain an original feature map.

[0188] a first attention module, configured to input the first fused feature map into a second attention network to obtain a second attention feature map;

[0189] The first spatial module is used to input the second attention feature map and the original feature map into the spatial feature transformation network to obtain an adjusted feature map corresponding to the original feature map.

[0190] Among them, the spatial feature transformation network is used to determine the spatial parameter matrix based on the original feature map, and use the spatial parameter matrix to perform spatial transformation on the second attention feature map. The spatial parameter matrix is ​​obtained by convolution processing on the original feature map.

[0191] Optionally, the reconstruction module 1140 is configured to:

[0192] Obtain multiple adjusted feature maps corresponding to the multiple original feature maps;

[0193] Merging multiple adjusted feature maps to obtain a merged feature map;

[0194] Input the merged feature map into the third attention network to obtain the important feature map;

[0195] The important feature maps are reduced in dimension through at least one convolutional layer to obtain a high dynamic range image.

[0196] In summary, an embodiment of the present application provides an image acquisition device that fuses multiple original images with different exposures obtained by shooting the same scene to obtain high dynamic range images corresponding to the multiple original images. Specifically, by introducing a first attention network into the fusion process of multiple original images and utilizing the first attention network to evaluate the importance of different image features, image features that are beneficial to obtaining high dynamic range images are highlighted, and interference features caused by motion are suppressed. This can reduce the impact of motion on image fusion during the process of shooting multiple original images, solve the problem of poor clarity of high dynamic range images obtained in related technologies, and achieve the effect of improving the clarity of the obtained high dynamic range images.

[0197] In addition, an embodiment of the present application also provides a schematic diagram of the structure of an electronic device. The electronic device includes one or more processors, a camera component, a memory, and a terminal. The memory may include random access memory (RAM) and read-only memory (ROM), and the camera component and the terminal may be an integrated structure. The network training portion of the above-mentioned image acquisition method can be applied to a server, and the other image processing portions other than network training can be applied to both a server and a terminal.

[0198] In addition, an embodiment of the present application also provides an image acquisition device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement an image acquisition method as in any of the above embodiments.

[0199] In addition, an embodiment of the present application also provides a non-volatile computer storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement an image acquisition method as in any of the above embodiments.

[0200] In addition, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image acquisition method of any of the above embodiments.

[0201] In this application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless expressly limited otherwise.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0205] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image acquisition method, characterized in that: The method comprises: Acquire multiple original images captured from the same scene, the multiple original images having different exposure levels; The multiple original images are fused through a fusion network to obtain a first fused feature map, the fusion network includes a first attention network, the fusion network includes a first fusion sub-network and a second fusion sub-network, and the first fusion sub-network and the second fusion sub-network both include the first attention network; the fusing of the multiple original images through the fusion network to obtain the first fused feature map includes: determining one of the multiple original images as a reference image; performing feature extraction on the reference image to obtain a reference feature map; performing feature extraction on multiple original images other than the reference image in the multiple original images to obtain multiple original feature maps; fusing the reference feature map with the multiple original feature maps through the first fusion sub-network to obtain a second fused feature map corresponding to each original feature map; fusing multiple second fused feature maps through the second fusion sub-network to obtain a first fused feature map; determining one of the multiple original images as a reference image; performing feature extraction on a plurality of original images excluding the reference image from the plurality of original images respectively to obtain a plurality of original feature maps; Inputting the first fused feature map into a second attention network to obtain a second attention feature map; Inputting the second attention feature map into a plurality of spatial feature transformation networks respectively, and inputting the plurality of original feature maps into the plurality of spatial feature transformation networks respectively, to obtain a plurality of adjusted feature maps corresponding to the plurality of original feature maps; wherein the spatial feature transformation network is used to determine a spatial parameter matrix based on the original feature map, and perform spatial transformation on the second attention feature map using the spatial parameter matrix, wherein the spatial parameter matrix is ​​obtained by performing convolution processing on the original feature map; Based on the adjusted feature map, high dynamic range images corresponding to the multiple original images are obtained.

2. The method according to claim 1, characterized in that The fusing the reference feature map with the multiple original feature maps respectively through the first fusion sub-network to obtain a second fused feature map corresponding to each of the original feature maps includes: Downsampling the reference feature map to obtain a first reference feature map; Performing downsampling processing on the original feature map to obtain a first original feature map; Merging the first reference feature map and the first original feature map, and inputting the merged features into the first attention network to obtain a first attention feature map; The first attention feature map is upsampled to obtain the second fusion feature map.

3. The method according to claim 1, characterized in that The acquiring, based on the adjusted feature map, high dynamic range images corresponding to the multiple original images includes: Obtaining a plurality of adjusted feature maps corresponding to the plurality of original feature maps; Merging the multiple adjusted feature maps to obtain a merged feature map; Inputting the merged feature map into a third attention network to obtain an important feature map; Performing dimensionality reduction processing on the important feature map through at least one convolutional layer to obtain the high dynamic range image.

4. The method according to claim 1, wherein The determining one of the plurality of original images as a reference image comprises: sorting the plurality of original images according to exposure from high to low; An original image with a medium exposure is selected as the reference image.

5. The method according to claim 1, wherein The step of extracting features from the reference image to obtain a reference feature map includes: The reference image is input into a residual densely connected network to obtain the reference feature map.

6. The method according to claim 1, characterized in that Before acquiring a plurality of original images captured for the same scene, the method further includes: Acquire a training sample from a sample set, the sample set including a plurality of training samples, the training sample including a plurality of sample images obtained by photographing the same scene, and a target sample high dynamic range image corresponding to the plurality of sample images, the plurality of sample images having different exposure levels; fusing the plurality of sample images through a fusion network to be trained to obtain a first fusion feature map, wherein the fusion network includes a first attention network to be trained; Acquire an adjusted feature map based on the first fused feature map and the multiple sample images; Based on the adjusted feature map, obtaining high dynamic range images corresponding to the multiple sample images; Comparing the high dynamic range images corresponding to the multiple sample images with the target sample high dynamic range image to obtain a comparison difference; If the contrast difference is greater than a preset result, adjusting the fusion network to be trained based on the contrast difference, and executing the step of training samples from the sample set; If the comparison difference is less than or equal to the preset result, the to-be-trained fusion network is determined as the fusion network.

7. An image acquisition device, characterized in that: The device comprises: An acquisition module is used to acquire multiple original images obtained by shooting the same scene, wherein the multiple original images have different exposure levels; A fusion module, configured to fuse the multiple original images through a fusion network to obtain a first fused feature map, wherein the fusion network includes a first attention network; the fusion module includes: a first determination module, configured to determine one of the multiple original images as a reference image; a first feature extraction module, configured to extract features from the reference image to obtain a reference feature map; a second feature extraction module, configured to extract features from multiple original images other than the reference image in the multiple original images to obtain multiple original feature maps; a first fusion submodule, configured to fuse the reference feature map with the multiple original feature maps through a first fusion subnetwork to obtain a second fused feature map corresponding to each original feature image; a second fusion submodule, configured to fuse multiple second fused feature maps through a second fusion subnetwork to obtain the first fused feature map; both the first fusion subnetwork and the second fusion subnetwork include a first attention network; An adjustment module is configured to determine one of the multiple original images as a reference image; further configured to perform feature extraction on multiple original images other than the reference image in the multiple original images to obtain multiple original feature maps; further configured to input the first fused feature map into a second attention network to obtain a second attention feature map; further configured to input the second attention feature map into multiple spatial feature transformation networks, and input the multiple original feature maps into the multiple spatial feature transformation networks, to obtain multiple adjusted feature maps corresponding to the multiple original feature maps; wherein the spatial feature transformation network is configured to determine a spatial parameter matrix based on the original feature map, and perform spatial transformation on the second attention feature map using the spatial parameter matrix, wherein the spatial parameter matrix is ​​obtained by performing convolution processing on the original feature map; A reconstruction module is used to obtain high dynamic range images corresponding to the multiple original images based on the adjusted feature map.

8. The device according to claim 7, characterized in that The first fusion submodule is used for: Downsampling the reference feature map to obtain a first reference feature map; Performing downsampling processing on the original feature map to obtain a first original feature map; Merging the first reference feature map and the first original feature map, and inputting the merged features into the first attention network to obtain a first attention feature map; The first attention feature map is upsampled to obtain the second fusion feature map.

9. An image acquisition device, characterized in that: The image acquisition device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image acquisition method according to any one of claims 1 to 6.

10. A non-transitory computer storage medium, characterized in that The non-transitory computer storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image acquisition method according to any one of claims 1 to 6.

Images