High-dynamic-range imaging method and system, electronic equipment and medium

Abstract

The invention relates to the technical field of image processing, and discloses a high-dynamic-range imaging method and system, electronic equipment and a medium, and the method comprises the steps: obtaining a first exposure image and a second exposure image in the same scene; adjusting pixel values of the first exposure image and the second exposure image through an interpolation method; performing exposure improvement and then exposure reduction on the pixel value of the first virtual exposure image, and performing exposure reduction and then exposure improvement on the pixel value of the second virtual exposure image; and fusing the first target virtual image and the second target virtual image. According to the method and the device, the pixel value adjustment processing is performed on the pixel values of the different exposure images to obtain the intermediate exposure images corresponding to the two exposure images respectively, and the exposure image enhancement processing is performed on the intermediate exposure images through exposure brightening and exposure lowering, so that the difference between the corresponding exposure images under different exposure durations is reduced, and the exposure image enhancement effect is improved. And the phenomenon of upside-down brightness during high-exposure-ratio image fusion is avoided.

Description

Background Art

[0002] High dynamic range imaging is becoming increasingly important due to its wide range of applications, but due to the limitations of existing digital device sensors, multiple low dynamic range images with different exposures are usually captured in sequence to expand the dynamic range, and then the low dynamic range images are fused to obtain a high dynamic range image. However, ghosting is considered to be the Achilles' heel of this high dynamic range imaging solution.

[0003] In order to solve the problem of ghosting, two images with different exposures are usually captured to reduce the cost of new equipment. The ratio between the exposure times of the two images can be large in order to capture as much information as possible from high-dynamic scenes. However, image fusion algorithms based on large exposure ratios can cause the fused image to have a phenomenon of light and dark inversion. For example, the areas that should be bright in the real scene are darker than the areas that should be dark after fusion, which looks unnatural. Summary of the invention

[0004] In order to overcome the problem that a fused image may have a light-dark inversion phenomenon caused by an image fusion algorithm based on a large exposure ratio, the present invention provides a high dynamic range imaging method, system, electronic device and medium.

[0005] In a first aspect, in order to solve the above technical problems, the present invention provides a high dynamic range imaging method, comprising:

[0006] Acquire a first exposure image and a second exposure image of the same scene; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image;

[0007] By using an interpolation method, the pixel value of the first exposure image is increased to determine a first virtual exposure image, and the pixel value of the second exposure image is decreased to determine a second virtual exposure image;

[0008] The pixel values ​​of the first virtual exposure image are first exposed and then exposed to determine a first target virtual image, and the pixel values ​​of the second virtual exposure image are first exposed and then exposed to determine a second target virtual image;

[0009] The first target virtual image and the second target virtual image are fused to determine a composite image.

[0010] In a second aspect, the present invention provides a high dynamic range imaging system, comprising:

[0011] An exposure image acquisition module, used to acquire a first exposure image and a second exposure image of the same scene; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image;

[0012] A virtual exposure image determination module, configured to increase the pixel value of the first exposure image by an interpolation method to determine a first virtual exposure image, and to decrease the pixel value of the second exposure image to determine a second virtual exposure image;

[0013] a target virtual image determination module, configured to first increase the exposure and then decrease the exposure of the pixel values ​​of the first virtual exposure image to determine the first target virtual image, and first decrease the exposure and then increase the exposure of the pixel values ​​of the second virtual exposure image to determine the second target virtual image;

[0014] The fusion module is used to fuse the first target virtual image and the second target virtual image to determine a composite image.

[0015] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor implements the steps of a high dynamic range imaging method as described above when executing the program.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of a high dynamic range imaging method as described above.

[0017] The beneficial effect of the present invention is as follows: the present application obtains intermediate exposure images (a first virtual exposure image and a second virtual exposure image) corresponding to two exposure images respectively by increasing or decreasing the pixel values ​​of images with different exposures, and then performs exposure image enhancement processing on the intermediate exposure images by means of exposure brightening and exposure reducing, thereby reducing the difference between the exposure images corresponding to different exposure times, and avoiding the phenomenon of light and dark inversion when images with a large exposure ratio are fused. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 A schematic diagram of a high dynamic range imaging method according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a flow chart of another embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the network structure for exposure improvement;

[0022] Figure 4 Schematic diagram of the network structure for exposure reduction;

[0023] Figure 5 A schematic diagram of a process of a high dynamic range imaging method corresponding to a high exposure image;

[0024] Figure 6 The figure is a schematic diagram of the structure of a high dynamic range imaging system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following examples are provided to further explain and supplement the present invention and do not constitute any limitation to the present invention.

[0026] A high dynamic range imaging method, system, electronic device and medium according to an embodiment of the present invention will be described below in conjunction with the accompanying drawings.

[0027] like Figure 1 As shown, the present invention provides a high dynamic range imaging method, for each preset load detection area, comprising:

[0028] S1. Acquire a first exposure image and a second exposure image; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image.

[0029] S2. Using an interpolation method, the pixel values ​​of the first exposure image are adjusted higher to determine a first virtual exposure image, and the pixel values ​​of the second exposure image are adjusted lower to determine a second virtual exposure image.

[0030] S3, first increase the exposure and then decrease the exposure of the pixel values ​​of the first virtual exposure image to determine the first target virtual image, and first decrease the exposure and then increase the exposure of the pixel values ​​of the second virtual exposure image to determine the second target virtual image.

[0031] S4. Fusing the first target virtual image and the second target virtual image to determine a composite image.

[0032] In this embodiment, the present application obtains intermediate exposure images (a first virtual exposure image and a second virtual exposure image) corresponding to the two exposure images by increasing or decreasing the pixel values ​​of the different exposure images, and then performs exposure image enhancement processing on the intermediate exposure images by means of exposure brightening and exposure reducing, thereby reducing the difference between the exposure images corresponding to different exposure times and avoiding the phenomenon of light and dark inversion when the images with a large exposure ratio are fused.

[0033] Optionally, adjusting the pixel value of the first exposure image higher by an interpolation method to determine the first virtual exposure image, and adjusting the pixel value of the second exposure image lower to determine the second virtual exposure image, comprises:

[0034] Mapping each pixel value in the first exposure image to a corresponding first preset irradiance through an exponential function to determine a first virtual exposure image;

[0035] Each pixel value in the second exposure image is mapped to a corresponding second preset irradiance through an exponential function to determine a second virtual exposure image.

[0036] In this embodiment, Z 1 and Z 3 They represent two images with different exposure times under the same scene, namely the first exposure image and the second exposure image, where the exposure time of the first exposure image is Δt 1 , the exposure time of the second exposure image is Δt 3 , now it is necessary to generate an intermediate exposure image between the first exposure image and the second exposure image by interpolation, and its exposure time is Δt 2 , Δt 1 <Δt 2 <Δt 3 Therefore, it is necessary to adjust the pixel values ​​of the first exposure image and the second exposure image.

[0037] In this embodiment, an interpolation method is used to process the exposure image to ensure that all synthesized pixels are on a smooth curve. In addition, an exponential function is added to the interpolation method to fit the exposure image, so that the interpolated image is closer to the true value.

[0038] In this embodiment, each pixel value in the first exposure image is mapped to the corresponding first preset irradiance through an exponential function to determine the first virtual exposure image, and each pixel value in the second exposure image is mapped to the corresponding second preset irradiance through an exponential function to determine the second virtual exposure image. The formula is as follows:

[0039]

[0040] When i=1, Λ i→2，c (z) represents the first virtual exposure image. When i=3, Λ i→2，c (z) represents the second virtual exposure image, Indicates that each pixel value in the first exposure image is mapped to the first preset amplitude, or each pixel value in the second exposure image is mapped to the second preset amplitude, F c (·) represents an exponential function.

[0041] Optionally, the pixel values ​​of the first virtual exposure image are first exposed to a higher exposure level and then exposed to a lower exposure level to determine a first target virtual image, and the pixel values ​​of the second virtual exposure image are first exposed to a lower exposure level and then exposed to a higher exposure level to determine a second target virtual image, further comprising:

[0042] Acquire each pixel value in the first virtual exposure image, and adjust each pixel value using a preset first weight coefficient to determine a first adjusted exposure image;

[0043] Acquire each pixel value in the second virtual exposure image, and adjust each pixel value using a preset second weight coefficient to determine a second adjusted exposure image;

[0044] The pixel values ​​of the first adjusted exposure image are first exposed to be increased and then exposed to be decreased to determine a first target virtual image, and the pixel values ​​of the second adjusted exposure image are first exposed to be decreased and then exposed to be increased to determine a second target virtual image.

[0045] In this embodiment, the pixel values ​​of the intermediate exposure images (the first virtual image and the second virtual image) are adjusted by adding weight coefficients to reduce unreliable information from overexposed or underexposed areas, so that it is more reliable to map the pixel values ​​of the overexposed area of ​​the dark image to the corresponding bright image, or to map the pixel values ​​of the overexposed area of ​​the bright image to the corresponding dark image.

[0046] The formulas for the first weight coefficient and the second weight coefficient are as follows:

[0047]

[0048]

[0049]

[0050] Among them, W 1 (z) represents the first weight coefficient, W 3 (z) represents the second weight coefficient, z represents the pixel value corresponding to the first virtual exposure image or the second virtual exposure image, and h 1 (z), h 3 (z) represent preset parameters respectively.

[0051] Optionally, exposure improvement of pixel values ​​of the first virtual exposure image or the second virtual exposure image is achieved through an exposure improvement network, where the exposure improvement network includes two 3*3 convolutional layers and an activation layer, and the activation layer is arranged between the two convolutional layers.

[0052] like Figure 2 As shown, LowZ 1 △t 1 The first exposure image is shown. It can be seen that LowZ 1 △t 1 First, use the exposure improvement network EV Up-Net to obtain the image with increased pixel value exposure Then The exposure reduction network EV Down-Net is used to obtain the image Low2 after the pixel value exposure is reduced, which is the first target virtual image.

[0053] Similarly, HigZ 3 △t 3 The second adjusted exposure image shows that HigZ 3 △t 3 First, use the exposure improvement network EV Down-Net to obtain an image with reduced pixel value exposure Then The exposure reduction network EV Up-Net is used to obtain the image Hig2 after the pixel value exposure is increased, which is the second target virtual image.

[0054] Based on the above, the exposure enhancement network in this embodiment is an EV Up-Net network, and a compensation module CB (Compensation Block, CB) is added to the EV Up-Net network, wherein Figure 3 As shown, CB includes two 3*3 convolutional layers (Conv3*3) and an activation layer (PReLU), and the activation layer is set between the two convolutional layers.

[0055] Optionally, exposure reduction of pixel values ​​of the first virtual exposure image or the second virtual exposure image is achieved through an exposure reduction network, where the exposure reduction network includes multiple residual modules and a 3*3 convolution layer, and the 3*3 convolution layer is connected after the last residual module.

[0056] In this embodiment, Figure 4 As shown, the exposure reduction network includes multiple residual modules DAB (Dual Attention Blocks, DAB) and a 3*3 convolution layer (Conv3*3), and the 3*3 convolution layer is connected after the last residual module.

[0057] In this embodiment, the process of EV Up-Net and EV Down-Net is described in detail. Figure 5 As shown, with high exposure image Z 3 △t 3 For example, Z 3 △t 3 The second virtual exposure image Λ can be obtained through the interpolation method (IMF) 3→2 (Z 3 ), for 3→2 (Z 3) Using the second weight coefficient (Weight Function), the second adjusted exposure image can be obtained. The second adjusted exposure image first passes through the exposure reduction network (RRG, Conv3*3), and then passes through the exposure increase network (Compensation Block, Conv3*3 and PReLU), and the second target virtual image Z can be obtained. 2 △t 2 .

[0058] In this embodiment, the processing process of the low-exposure image is similar to the processing process of the high-exposure image, except that the order of the exposure-enhancing network and the exposure-reducing network is adjusted, so it is not described in detail.

[0059] Optionally, the exposure-enhancing network and the exposure-reducing network are trained by an unsupervised learning method until the loss function meets the preset requirements, and the corresponding models meeting the preset requirements are used as the exposure-enhancing network and the exposure-reducing network;

[0060] The loss function represents the difference between the first target virtual image and the second target imaginary image.

[0061] In this embodiment, the loss function also plays an important role in the proposed exposure perception enhancement network (exposure enhancement network and exposure reduction network). The exposure enhancement network and the exposure reduction network adopt an unsupervised learning method, that is, they are performed without obtaining the real intermediate exposure image, which greatly expands the scope of application scenarios. Specifically, the following loss function is minimized:

[0062]

[0063] Among them, L represents the loss function, Represents the low exposure image Z 1 The virtual low-exposure image is generated after passing through the exposure-raising network and then the exposure-lowering network. Represents the high exposure image Z 3 The virtual low-exposure image obtained after two exposure reduction networks, Represents Z 3 After passing through the exposure reduction network and then the exposure increase network, a virtual high-exposure image is obtained. Represents Z 1 The virtual high-exposure image obtained after two exposures to enhance the network, Represents Z 1 The virtual intermediate exposure image obtained by the exposure enhancement network, Represents Z 3 The virtual intermediate exposure image obtained by the exposure reduction network.

[0064] Optionally, the first target virtual image and the second target virtual image are fused to determine a composite image, and the formula is as follows:

[0065]

[0066] in, represents the synthetic image, W i (Z i (p)) when i=1, it indicates the first adjusted exposure image, W 1 Represents the first weight coefficient, Z 1 (p) represents the pixel value corresponding to the coordinate point p in the first virtual exposure image, represents the first target virtual image, W i (Z i (p)) When i = 3, W 3 (Z 3 (p)) represents the second adjusted exposure image, W 3 Represents the second weight coefficient, Z 3 (p) represents the pixel value corresponding to the coordinate point p in the second virtual exposure image, Represents the second target virtual image.

[0067] In this embodiment, a combination of data-driven and model-driven strategies (exposure-enhancing network and exposure-reducing network) is adopted through the loss function, which has strong robustness and reduces dependence on data. At the same time, it uses unsupervised training methods (loss function) to expand the application scope of the algorithm.

[0068] like Figure 6 As shown, the present invention provides a high dynamic range imaging system, comprising:

[0069] An exposure image acquisition module, used to acquire a first exposure image and a second exposure image of the same scene; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image;

[0070] A virtual exposure image determination module, configured to increase the pixel value of the first exposure image by an interpolation method to determine a first virtual exposure image, and to decrease the pixel value of the second exposure image to determine a second virtual exposure image;

[0071] a target virtual image determination module, configured to first increase the exposure and then decrease the exposure of the pixel values ​​of the first virtual exposure image to determine the first target virtual image, and first decrease the exposure and then increase the exposure of the pixel values ​​of the second virtual exposure image to determine the second target virtual image;

[0072] The fusion module is used to fuse the first target virtual image and the second target virtual image to determine a composite image.

[0073] Optionally, the virtual exposure image determination module is specifically used to:

[0074] Mapping each pixel value in the first exposure image to a corresponding first preset irradiance through an exponential function to determine a first virtual exposure image;

[0075] Each pixel value in the second exposure image is mapped to a corresponding second preset irradiance through an exponential function to determine a second virtual exposure image.

[0076] Optionally, the target virtual image determination module is further used to:

[0077] Acquire each pixel value in the first virtual exposure image, and adjust each pixel value using a preset first weight coefficient to determine a first adjusted exposure image;

[0078] Acquire each pixel value in the second virtual exposure image, and adjust each pixel value using a preset second weight coefficient to determine a second adjusted exposure image;

[0079] The pixel values ​​of the first adjusted exposure image are first exposed to be increased and then exposed to be decreased to determine a first target virtual image, and the pixel values ​​of the second adjusted exposure image are first exposed to be decreased and then exposed to be increased to determine a second target virtual image.

[0080] Optionally, a training module is also included, specifically for:

[0081] The exposure-enhancing network and the exposure-reducing network are trained by an unsupervised learning method until the loss function meets the preset requirements, and the corresponding models meeting the preset requirements are used as the exposure-enhancing network and the exposure-reducing network;

[0082] The loss function represents the difference between the first target virtual image and the second target imaginary image.

[0083] Optionally, the fusion module is specifically used to:

[0084] The first target virtual image and the second target virtual image are fused to determine the composite image. The formula is as follows:

[0085]

[0086] in, represents the synthetic image, W i (Z i (p)) when i=1, it indicates the first adjusted exposure image, W 1 Represents the first weight coefficient, Z 1 (p) represents the pixel value corresponding to the coordinate point p in the first virtual exposure image, represents the first target virtual image, W i (Z i (p)) When i = 3, W 3 (Z 3 (p)) represents the second adjusted exposure image, W 3 Represents the second weight coefficient, Z 3 (p) represents the pixel value corresponding to the coordinate point p in the second virtual exposure image, Represents the second target virtual image.

[0087] An embodiment of the present invention also provides an electronic device, including a memory, a manager, and a program stored in the memory and running on the manager. When the manager executes the program, it implements part or all of the steps of the above-mentioned high dynamic range imaging method.

[0088] Among them, the electronic device can be a computer, and correspondingly, its program is computer software. The above-mentioned parameters and steps in the electronic device of the present invention can refer to the parameters and steps in the embodiment of a high dynamic range imaging method mentioned above, and will not be repeated here.

[0089] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0090] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0091] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A high dynamic range imaging method, characterized in that: include: Acquire a first exposure image and a second exposure image of the same scene; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image; By using an interpolation method, the pixel values ​​of the first exposure image are adjusted higher to determine a first virtual exposure image, and the pixel values ​​of the second exposure image are adjusted lower to determine a second virtual exposure image; The pixel values ​​of the first virtual exposure image are first exposed to increase and then exposed to decrease to determine a first target virtual image, and the pixel values ​​of the second virtual exposure image are first exposed to decrease and then exposed to increase to determine a second target virtual image; The first target virtual image and the second target virtual image are fused to determine a composite image.

2. The method according to claim 1, characterized in that The method comprises: adjusting the pixel value of the first exposure image to be increased by an interpolation method to determine a first virtual exposure image, and adjusting the pixel value of the second exposure image to be decreased to determine a second virtual exposure image, comprising: Mapping each pixel value in the first exposure image to a corresponding first preset irradiance through an exponential function to determine a first virtual exposure image; Each pixel value in the second exposure image is mapped to a corresponding second preset irradiance through an exponential function to determine a second virtual exposure image.

3. The method according to claim 1, characterized in that The method further comprises: first increasing the exposure of the pixel values ​​of the first virtual exposure image and then decreasing the exposure to determine a first target virtual image; and first decreasing the exposure of the pixel values ​​of the second virtual exposure image and then increasing the exposure to determine a second target virtual image. Acquire each pixel value in the first virtual exposure image, and adjust each pixel value using a preset first weight coefficient to determine a first adjusted exposure image; Acquire each pixel value in the second virtual exposure image, and adjust each pixel value using a preset second weight coefficient to determine a second adjusted exposure image; The pixel values ​​of the first adjusted exposure image are first exposed to be increased and then exposed to be decreased to determine a first target virtual image, and the pixel values ​​of the second adjusted exposure image are first exposed to be decreased and then exposed to be increased to determine a second target virtual image.

4. The method according to claim 1, characterized in that: Exposure improvement of the pixel values ​​of the first virtual exposure image or the second virtual exposure image is achieved through an exposure improvement network, wherein the exposure improvement network includes two 3*3 convolutional layers and an activation layer, and the activation layer is arranged between the two convolutional layers.

5. The method according to claim 4, characterized in that: Exposure reduction of pixel values ​​of the first virtual exposure image or the second virtual exposure image is achieved through an exposure reduction network, wherein the exposure reduction network includes multiple residual modules and a 3*3 convolution layer, and the 3*3 convolution layer is connected after the last residual module.

6. The method according to claim 5, characterized in that The exposure-enhancing network and the exposure-reducing network are trained by an unsupervised learning method until the loss function meets the preset requirements, and the corresponding models meeting the preset requirements are used as the exposure-enhancing network and the exposure-reducing network; The loss function represents the difference between the first target virtual image and the second target imaginary image.

7. The method according to claim 3, characterized in that The first target virtual image and the second target virtual image are fused to determine a composite image, and the formula is as follows: in, represents the synthetic image, W i (Z i When i=1 in (p), it represents the first adjusted exposure image, W1 represents the first weight coefficient, and Z1(p) represents the pixel value corresponding to the coordinate point p in the first virtual exposure image. represents the first target virtual image, W i (Z i (p)) when i=3, W3(Z3(p)) represents the second adjusted exposure image, W3 represents the second weight coefficient, and Z3(p) represents the pixel value corresponding to the coordinate point p in the second virtual exposure image. Represents the second target virtual image.

8. A high dynamic range imaging system, characterized in that: include: An exposure image acquisition module, used to acquire a first exposure image and a second exposure image of the same scene; wherein the exposure time of the first exposure image is shorter than the exposure time of the second exposure image; a virtual exposure image determination module, configured to increase the pixel value of the first exposure image by an interpolation method to determine a first virtual exposure image, and to decrease the pixel value of the second exposure image to determine a second virtual exposure image; a target virtual image determination module, configured to first increase the exposure and then decrease the exposure of the pixel values ​​of the first virtual exposure image to determine a first target virtual image, and first decrease the exposure and then increase the exposure of the pixel values ​​of the second virtual exposure image to determine a second target virtual image; The fusion module is used to fuse the first target virtual image and the second target virtual image to determine a composite image.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a high dynamic range imaging method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the steps of a high dynamic range imaging method as described in any one of claims 1 to 7.

Images