HDR image reconstruction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an HDR image reconstruction method and device, computer equipment and a computer storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a low dynamic range (LDR) image, and performing feature extraction on the LDR image to obtain an attention information feature map for the LDR image; inputting the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range (HDR) image; wherein the image reconstruction model at least comprises a local enhancement module and a spatial feature transformation module, the local enhancement module is used for extracting corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used for performing spatial affine transformation on the local spatial information to obtain an HDR image. According to the method, ghosting and artifacts of the HDR image obtained by reconstruction can be avoided, so that the image quality is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to an HDR image reconstruction method, an HDR image reconstruction device, a computer device, and a computer storage medium. Background Art

[0002] In the field of digital image processing, dynamic range refers to the distribution range of light intensity from the darkest shadow part to the brightest highlight part in the image. Due to the hardware of existing image acquisition devices such as cameras and smart phones, which are limited by the design and size of their camera sensors, they can only capture limited low dynamic range (LDR) images, resulting in information loss in overexposed and underexposed areas of the image. High dynamic range (HDR) images have a wider dynamic range and can more realistically restore light and shadow effects that are close to real scenes, so that images with richer details, more realistic pictures, and higher quality can be obtained. Therefore, a series of processing is often performed based on the collected LDR images to obtain HDR images in order to achieve HDR image reconstruction.

[0003] In the related technical solutions, HDR images are often obtained by fusing multiple LDR images with different exposure parameters. However, this method can better fuse into an HDR image when the multiple LDR images are strictly aligned, for example, when there is no background jitter or foreground object motion.

[0004] However, in real scenes, on the one hand, since handheld camera shooting often causes the background of the picture to shake, and there are usually moving foreground objects in the shooting scene, the direct fusion of multiple frames of LDR images is prone to problems such as ghosting and artifacts. On the other hand, when there is a sharp change in brightness in the scene, the direct fusion of multiple frames of LDR images is also prone to detail loss, which ultimately leads to poor quality of the reconstructed HDR image.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] In the embodiments of the present application, a HDR image reconstruction method, an HDR image reconstruction method apparatus, a computer device and a computer storage medium are provided, thereby overcoming, at least to a certain extent, the technical problems that the reconstructed HDR image has ghosting, artifacts and poor image quality due to the limitations and defects of the relevant technology.

[0007] In a first aspect of an embodiment of the present application, a HDR image reconstruction method is provided, the method comprising: obtaining a low dynamic range (LDR) image, and performing feature extraction on the LDR image to obtain an attention information feature map for the LDR image; inputting the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range (HDR) image; wherein the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain an HDR image.

[0008] In a second aspect of an embodiment of the present application, a HDR image reconstruction device is provided, which includes: a feature extraction module, used to obtain a low dynamic range LDR image, and perform feature extraction on the LDR image to obtain an attention information feature map for the LDR image; an HDR image reconstruction module, used to input the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain an HDR image.

[0009] According to a third aspect of an embodiment of the present application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the above HDR image reconstruction methods are implemented.

[0010] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above HDR image reconstruction methods are implemented.

[0011] A fifth aspect of the embodiments of the present application provides a computer program product, including a computer program, which is executed by a processor to implement the steps of any one of the above HDR image reconstruction methods.

[0012] The technical solution of this application has the following beneficial effects: According to the HDR image reconstruction method, a low dynamic range LDR image is obtained, and features are extracted from the LDR image to obtain an attention information feature map for the LDR image; the attention information feature map is input into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model at least includes a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract the corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform a spatial affine transformation on the local spatial information to obtain an HDR image. On the one hand, the method reconstructs an HDR image for a single frame of LDR, thereby avoiding the artifact and ghosting problems of merging multiple frames of LDR to construct an HDR image in the related technical solution; on the other hand, the method takes into account the imaging differences between LDR and HDR images, and analyzes the various steps with differences in the camera imaging process, such as photoelectric conversion, quantization and tone mapping, respectively, to construct a neural network model that meets the characteristics of LDR and HDR images, so as to improve the generalization of image reconstruction based on artificial intelligence methods, thereby improving the image quality of the reconstructed HDR image and improving the image reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of the difference between LDR imaging and HDR imaging in one of the camera imaging processes provided in an embodiment of the present application; Figure 2 An application architecture diagram of one of the HDR image reconstruction systems provided in an embodiment of the present application; Figure 3 A flowchart of one of the HDR image reconstruction methods provided in an embodiment of the present application; Figure 4 A schematic diagram of a process for extracting distinguishing features of an LDR image based on visual attention provided in an embodiment of the present application; Figure 5A A schematic diagram of the structure of one of the HDR image reconstruction models provided in an embodiment of the present application; Figure 5B A schematic diagram of the structure of a local enhancement module composed of a UNet network structure and a residual module provided in an embodiment of the present application; Figure 5C A schematic diagram of the structure of one of the residual modules provided in an embodiment of the present application; Figure 6A schematic diagram of a structure of an HDR image visualization provided in an embodiment of the present application; Figure 7 A schematic diagram of the comparison effect before and after reconstruction of one of the HDR images provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an HDR image reconstruction device provided in one embodiment of the present application; Fig. 9 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0014] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0015] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0016] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0017] First, in order to facilitate those skilled in the art to understand the technical solution of the present application, the terms involved in the embodiments of the present application are explained as follows: 1) High Dynamic Range (HDR): It is mainly described relative to the Low Dynamic Range (LDR) and the standard dynamic range. In layman's terms, it refers to the range of light brightness that the camera can accommodate. In some major international standards, the standard parameters for high dynamic range in the display field mainly include color depth, color gamut, and dynamic range. Generally, the color depth of low dynamic range is 8 bits, the color gamut is BT709, and the dynamic range curve is usually the Gamma function. According to the international ITU standard T / UWA 005.1-2022, the color depth of high dynamic range is above 10 bits, the color gamut is above BT709, such as BT2020, and the dynamic range curve is usually the HLG or PQ curve. In the conversion from LDR to HDR, if one of the above three aspects meets the high dynamic range standard and the subjective effect is similar to that of the high dynamic range image, we call it the completion of the conversion from LDR to HDR.

[0018] 2) Color depth: Color depth is the color bit depth, which is a unit that uses the number of bits to represent the number of colors in a digital image. It is generally related to the quantization of an image. Image quantization is to divide the value range of each sample point of the sampled image into several intervals, and use only one value to represent all the values ​​in each interval. Another way of saying it is that quantization is a process that aims to convert the continuous change interval of the brightness corresponding to the image pixel point into a single specific value, that is, to discretize the spatial coordinate amplitude value of the original grayscale image. The more quantization levels, the richer the image hierarchy, the higher the grayscale resolution, and the better the image quality; the fewer quantization levels, the simpler the image hierarchy, the lower the grayscale resolution, and the phenomenon of image contour stratification will occur, which reduces the image quality. Generally, LDR images are quantized to 8 bits, and HDR images are quantized to more than 10 bits.

[0019] 3) Color gamut: Color gamut refers to the sum of the colors that a technical system can produce, that is, the range of colors that an image can express. The range of colors that can be expressed on each display device is different. For example, the display color gamut range of a normal display screen and a 4K display screen is different. HDTV is the display color range of a normal display screen, and the color gamut is expressed as Rec.709 or BT709; 4K display screens generally belong to the display range of UHDTV, and the color gamut is BT2020. In an image, the color gamut refers to the number of colors that a mode image can have. For example: for an image in gray mode, each pixel is represented by one byte, then the gray mode image can have a maximum of 2 8 = 256 colors, and its color range is 0~255. For an RGB image, if one primary color is represented by one byte, then the RGB image can have at most 224 Color, its color gamut range is 0~2 24 -1.

[0020] 4) Camera imaging: It is based on the principle of optoelectronics. Currently, the widely used digital cameras use photosensitive devices to convert the received light signals into electrical signals proportional to the exposure, which are then processed by analog / digital converters to convert them into digital signals. Finally, they are converted into original image files for storage through subsequent nonlinear operations. The process from photosensitive devices to obtaining digital images is mainly divided into the sensor to imaging process and the processing flow within the camera.

[0021] In the relevant technical background, in the field of digital images, the dynamic range is usually used to measure the ratio of the highest value to the lowest value of the brightness of an image or scene, that is, the ratio of the maximum value to the minimum value in the pixel value of the digital image, which is used to describe the distribution range of light intensity from the darkest shadow part to the brightest highlight part in the image. For digital images describing the same scene, the higher the dynamic range, the better the subjective visual effect. Based on the above dynamic range, low dynamic range LDR images and high dynamic range HDR images can be derived.

[0022] Among them, LDR image refers to the narrow range of light brightness that the image acquisition device can capture and display, usually with 8-bit precision to represent 256 grayscale levels. Therefore, the brightness difference that the LDR image can record is very limited. For example, in the case of sufficient light in the shooting scene, such as when there are bright objects, the traditional LDR image can only record part of the lighting information. When the actual lighting is greater than the light range that the image acquisition device can record, the image is very likely to show an over-exposure state, causing the pixels of the image to be saturated. Correspondingly, when the light in the shooting scene is extremely scarce, the weak light cannot be accurately recorded due to the low quantization level of the LDR image itself, and it is very easy to show an under-exposure state. When the LDR image exists for both of the above scenarios, it will show information loss in the over-exposed and under-exposed areas of the image.

[0023] HDR images refer to a device that can capture and display a very wide range of light brightness, usually exceeding 8-bit precision (e.g. 16-bit). It is a type of image that can more accurately record the complete scene lighting information and brightness differences. Compared with LDR images, HDR images can store the tiny brightness differences of a scene, displaying detailed information in high-exposure and low-exposure areas, close to the state in which the human eye can see both light and dark areas after adapting to the light. Therefore, HDR images can more completely record and display the texture details of light and dark areas that are basically the same as those in real scenes, improving the quality and realism of the image.

[0024] However, due to the limited dynamic range of imaging device hardware, the images obtained by standard digital cameras are of poor quality, often with areas of overexposure or underexposure. HDR imaging devices that can directly capture HDR images are expensive and difficult to popularize in a short period of time, making it difficult to directly obtain HDR images.

[0025] In some technical solutions, the most common method to obtain HDR images through existing lower-cost ordinary image acquisition devices (such as ordinary digital cameras or cameras on smartphones) is to obtain a series of LDR images under different exposure conditions, and then merge the above series of LDR images into the same image to form an HDR image. However, for dynamic scenes or handheld image acquisition devices, the above method will inevitably cause the synthesized HDR image to produce ghosting, artifacts, noise, information loss and other problems. Therefore, how to generate better HDR images faster is the focus of current research in the HDR field.

[0026] In response to the above technical problems, a reconstruction method of HDR images based on single-frame LDR images is often used, also known as a single-frame HDR image reconstruction algorithm. Its processing principle is usually to stretch the brightness range of the LDR image in a certain way, thereby enhancing the detail information of the overexposed and underexposed areas. Traditional single-frame HDR image reconstruction methods can be divided into two categories: based on the inverse camera response function (iCRF) and based on the inverse tone mapping (ITM) method. Among them, the iCRF method first estimates the corresponding iCRF curve through the input LDR image, and then uses the iCRF to map the image in the LDR domain to obtain the corresponding HDR image. The ITM algorithm uses a specific dynamic operator to map the LDR image, expands the brightness range of the LDR image, and obtains a clearer HDR image with corresponding details.

[0027] However, due to the large amount of information missing in the overexposed and underexposed areas of the LDR image, there are still some shortcomings. In this regard, the relevant technical solutions also use deep learning-based methods to reconstruct HDR images. The existing deep learning-based methods mainly include: (1) Direct deep learning from a single-frame LDR. For example, in 2017, Eilertsen et al. first proposed a method based on deep learning to recover detailed information in overexposed areas of an image. By setting appropriate parameters, such areas with severe information loss in the image are separated, and then a neural network with a codec structure similar to the UNet structure is used to perform targeted reconstruction of the overexposed area information. The disadvantage of this method is that it still cannot recover details well when there are large areas of saturation, and does not focus on the recovery of information in underexposed areas.

[0028] (2) Re-fusion of multiple exposure images to generate a single-frame LDR. In 2017, Endo et al. trained a network that can learn to generate multiple LDR images of different exposures from a single-exposure LDR image, and then obtained an HDR image by combining the basic fusion algorithm of multiple exposure images. This method obtains the details of the exposure by integrating multiple LDR images, so that the final generated result has richer details, but the training data of the network is extremely large and when the brightness of the overexposed image is too high, the final image will have color distortion.

[0029] (3) Use multi-branch networks to learn different tasks. The most important method in multi-branch network learning is based on the camera imaging process. For example, in order to avoid the checkerboard artifacts caused by upsampling in the network, Marnerides et al. proposed a three-branch network, which consists of a local branch, a dilation branch, and a global branch. Finally, the results of each branch are fused to obtain the final HDR image. In 2020, Liu et al. used the pipeline of learning camera photography to decompose the single-image HDR reconstruction problem into three subtasks: inverse quantization, linearization, and generation of the final HDR image, and developed three deep networks to specifically solve each task. In 2021, Chen et al. proposed a three-branch network end-to-end mapping HDRUNet to solve the noise and quantization problems in HDR images, and achieved good results in single-frame HDR reconstruction tasks, but there are still problems in highly oversaturated areas. In 2022, Chen et al. re-proposed a three-stage method based on the camera process, which achieved good results, but the process was relatively complicated. Currently, in single-frame tasks, various existing deep learning-based networks have an inherent limitation, that is, since they are completely dependent on a single input LDR image, these methods are often not applicable in scenes with high contrast in large-scale saturated and dark areas. Therefore, the fundamental difficulty of HDR image reconstruction tasks based on single-frame LDR images has not been completely solved.

[0030] It can be seen that the HDR reconstruction model based on deep learning shown in the above-mentioned related technical solutions has not deeply analyzed the theoretical level of image acquisition equipment and imaging, which leads to the construction model not targeting the specific characteristics of LDR images and HDR images, and there is a technical problem of weak generalization of image restoration quality, which in turn affects the image quality of the reconstructed HDR image. Among them, the generalization of image quality is used to evaluate whether the performance of the image reconstruction model is consistent on different data sets. Specifically, the weak generalization of the image reconstruction model means that the image reconstruction model has good performance on a known training data set, but it does not mean that the image reconstruction model can also perform well on an unknown data set.

[0031] In view of the above technical problems, the embodiment of the present application proposes a HDR image reconstruction method, which can be applied to any application scenario that requires reconstruction of HDR images based on LDR images. In this method, the inventor uses the camera imaging principle as a starting point to achieve HDR image reconstruction. The details are as follows: That is, the inventors found that the camera imaging process of LDR and HDR is consistent, both of which convert the initial optical signal into an image, but in the process from optical signal to digital signal, combined with the standard parameter requirements of HDR, its key processing flow includes four parts: photoelectric conversion, quantization, color gamut mapping and tone mapping. While the generation process from optical signal to digital signal of LDR and HDR is consistent, there are some differences in each mapping function that needs to be converted.

[0032] In the embodiment of the present application, the color gamut of the selected data set does not change much, so the main processes are photoelectric conversion, quantization and tone mapping. Figure 1 The difference between LDR imaging and HDR imaging in the camera imaging process is shown in the figure. The raw data (Raw Data) is subjected to photoelectric conversion and quantization under different mapping functions, and the corresponding raw image (Raw Image) is subjected to color gamut conversion (the difference between the data sets is not big) and tone mapping (different tone mapping functions), thereby obtaining LDR and HDR respectively. Therefore, a single-frame HDR image can be reconstructed based on the three aspects of photoelectric conversion, quantization and tone mapping in the camera imaging process to improve the quality of the reconstructed HDR image.

[0033] The following will be combined Figure 2 , the HDR image reconstruction method and device proposed in the present disclosure are applied to Figure 2 The embodiments are described in the system architecture of the exemplary application environment shown.

[0034] like Figure 2 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a terminal device 201 and a server 202 may be included.

[0035] The terminal device 201 can be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart vehicle-mounted device, a smart wearable device, a smart TV, an aircraft, or any other device that requires image reconstruction or HDR image display. The terminal device 201 can be installed with a target application, and the target application can have functions such as image display. The application involved in the embodiment of the present application can be a software client, or a client such as a web page or a small program. The server 202 is a server corresponding to the software or a web page, a small program, etc., and does not limit the specific type of the client.

[0036] The server 202 may be a background server of the target application, used to provide corresponding background services for it, such as HDR image reconstruction services, etc. It may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited thereto.

[0037] It should be noted that the HDR image reconstruction method in the embodiment of the present application can be executed by the terminal device 201 or the server 202 alone, or can be executed by the server 202 and the terminal device 201 together. Among them, the server 202 and the terminal device 201 can both include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 202 can also be configured with a database, which can be used to store model parameters, LDR and HDR images before and after reconstruction, etc. Among them, the memories of the server 202 and the terminal device 201 can also store the program instructions required to be executed in the HDR image reconstruction method provided in the embodiment of the present application, and these program instructions can be used to implement the image reconstruction process provided in the embodiment of the present application when executed by the processor.

[0038] It should be noted that when the HDR image reconstruction method provided in the embodiment of the present application is executed separately by the server 202 or the terminal device 201, the above-mentioned application scenario may also only include a single device of the server 202 or the terminal device 201, or the server 202 and the terminal device 201 may be considered to be the same device. Of course, in actual application, when the HDR image reconstruction method provided in the embodiment of the present application is jointly executed by the server 202 and the terminal device 201, the server 202 and the terminal device 201 may also be the same device, that is, the server 202 and the terminal device 201 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0039] In the embodiment of the present application, the terminal device 201 and the server 202 can be directly or indirectly connected to each other through one or more networks 203. The network 203 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiment of the present application. It should be noted that Figure 2 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0040] For example, in an exemplary embodiment, the server 202 obtains an LDR image from the terminal device 201, and performs feature extraction on the LDR image to obtain an attention information feature map for the LDR image.

[0041] The attention information feature map is input into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model at least includes a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract the corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform a spatial affine transformation on the local spatial information to obtain an HDR image. Thus, the reconstructed HDR image is displayed in the terminal device 201.

[0042] Below, in combination with the application scenarios described above, the method provided by the exemplary implementation of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. It should be noted that the following method can be executed by the above terminal device or server, or by the terminal device and the server together. Here, the server is specifically used as the execution subject for example.

[0043] The following uses the server as an execution subject and applies the HDR image reconstruction method to the terminal device as an example for illustration. Figure 3 This is a flowchart of one of the HDR image reconstruction methods provided in the embodiments of the present application; see Figure 3 The fluid simulation method provided in the embodiment of the present application includes the following steps 301-302: Step 301: Acquire an LDR image, and perform feature extraction on the LDR image to obtain an attention information feature map for the LDR image.

[0044] Step 302: input the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract the corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform a spatial affine transformation on the local spatial information to obtain the HDR image.

[0045] Through the above Figure 3 The method shown, on the one hand, is to reconstruct an HDR image for a single LDR frame, thereby avoiding the artifacts and ghosting problems of merging multiple LDR frames to construct an HDR image in the related technical solution; on the other hand, the method takes into account the imaging differences between LDR and HDR images, and analyzes each step with differences in the camera imaging process, such as photoelectric conversion, quantization and tone mapping, to construct a neural network model that meets the characteristics of LDR and HDR images, so as to improve the generalization of image reconstruction based on artificial intelligence methods, thereby improving the image quality of the reconstructed HDR image and improving the image reconstruction efficiency.

[0046] In conjunction with the specific embodiments Figure 3 The specific implementation methods of each step in the embodiment shown are described in detail: In step 301, an LDR image is acquired, and features are extracted from the LDR image to obtain an attention information feature map for the LDR image.

[0047] The LDR image in step 301 is a single-frame LDR image, not a multi-frame LDR image.

[0048] In the related technical solutions, a series of multi-frame LDR images are often used to merge to obtain an HDR image. However, for dynamic scenes or handheld image acquisition devices, the above method will inevitably cause the reconstructed HDR image to have problems such as ghosting, artifacts, noise, and information loss. In order to overcome the above technical problems of multi-frame LDR images, the embodiment of the present application will implement the reconstruction process of the HDR image based on a single-frame LDR image.

[0049] After obtaining a single-frame LDR image, the feature extraction of the LDR image can be performed in the above step 301 to obtain the attention information feature map for the single-frame LDR image. The above implementation process will be described in conjunction with a specific embodiment below.

[0050] In an optional embodiment of the present disclosure, the LDR image is input into a pre-trained Transformer model to extract multiple deep local feature representations from the LDR image based on the Transformer model, and the relative position of each deep local feature representation is determined to generate a relative position code of each deep local feature representation; the multiple deep local feature representations and the relative position code of each deep local feature representation are input into a pre-trained SwinIR model for feature extraction to obtain an attention information feature map for the LDR image.

[0051] Among them, the attention information feature map refers to the feature map with attention information. The Transformer model is a deep learning model based on the self-attention mechanism. Unlike traditional RNN and CNN, the Transformer model only uses the self-attention mechanism to process the input sequence and the output sequence, so it can be calculated in parallel, greatly improving the calculation efficiency. The SwinIR model is an image reconstruction model based on the Swin Transformer. It needs to be explained that the SwinIR model mainly consists of the following three parts: 1) Shallow feature extraction: Use convolutional layers to extract shallow features of the input LDR image.

[0052] 2) Deep feature extraction: Multiple residual Swin Transformer blocks (RSTBs) are used for deep feature extraction, each of which contains multiple Swin Transformer layers and a residual connection.

[0053] 3) High-quality image reconstruction: The shallow and deep features are input into the image reconstruction module through skip connections. The shallow features mainly contain low-frequency information, and the deep features focus on recovering the lost high-frequency information. It can be seen that the SwinIR model can remove severe noise interference and retain high-frequency image details, thereby obtaining clearer edges and more natural textures.

[0054] Exemplarily, different image regions of an LDR image exhibit different features. For this reason, the embodiment of the present application converts different regions of LDR image representation into the problem of extracting discriminative features for LDR images. It should be explained that in the field of artificial intelligence, the goal of feature extraction is to extract representative and discriminative features from raw data so that machine learning models can better understand and process data. For this reason, it is necessary to extract discriminative features / representative features in LDR images.

[0055] Taking the above problems into consideration, the present application adopts a Transformer model based on the visual attention mechanism and a SwinIR model in this embodiment to extract deep local features of LDR images, so that the feature extraction of visual attention is more targeted, which is beneficial to the accuracy and efficiency of feature extraction. Specifically, (1) the image content and attention weight interact based on the content, which can be explained as a spatially varying convolution. (2) The image restoration is achieved by using adjacent pixels through the shift window mechanism unique to SwinIR.

[0056] The following will be combined Figure 4 The detailed process of extracting the attention information feature map in the above embodiment is exemplified.

[0057] Figure 4 A schematic diagram of a process for extracting distinguishing features of an LDR image based on visual attention provided in an embodiment of the present application, referring to Figure 4 As shown in Figure 2, the LDR image is input into the discriminative feature extraction module based on visual attention. First, multiple deep local feature representations (i.e. Figure 4 The LDR discriminative feature representation shown in FIG2 is obtained by calculating the relative position of each deep local feature representation and generating a relative position encoding of each deep local feature representation; the multiple deep local feature representations and the relative position encoding of each deep local feature representation are input into the pre-trained SwinIR model for feature extraction to obtain an attention information feature map for the LDR image. Figure 4 The deep feature extraction module (Deep Feature Extraction) in the SwinIR model shown includes: multiple residual Swin Transformer blocks (RSTB) and convolutional layers (Conv) to perform deep feature extraction of LDR images based on multiple residual Swin Transformer blocks (RSTB) and convolutional layers (Conv).

[0058] Through this embodiment, the Transformer model and the SwinIR model are used to extract the deep local features of the LDR image, so that the feature extraction of visual attention is more targeted, which is beneficial to the accuracy and efficiency of feature extraction. And the self-attention mechanism of Swin Transformer can better capture the global interaction between contexts, so as to facilitate the accurate and rapid restoration of high-quality images.

[0059] In addition to using the above-mentioned embodiments, CNN (Convolutional Neural Network) and the Transformer model alone can also be used to extract distinctive features of LDR images. However, the above-mentioned method of visual attention feature extraction based on the SwinIR model is better and more targeted than CNN (Convolutional Neural Network) and the Transformer model alone.

[0060] In step 302, the attention information feature map is input into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract the corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform a spatial affine transformation on the local spatial information to obtain an HDR image.

[0061] Exemplarily, the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, wherein the local enhancement module is used for the quantization process, in which the fine granularity of quantization reflects the difference in spatial information at the feature level. Therefore, the local enhancement module can be used to capture the feature details in the attention information feature map to extract local spatial information, thereby improving the image restoration quality.

[0062] As for tone mapping, tone mapping is to map high dynamic range images to the display, which is convenient for subsequent image display and image processing. In the problem of tone mapping, the difference between the local mapping algorithm and the global mapping algorithm is that the local spatial domain difference of the image is considered, and different methods are used to dynamically map pixels in different areas of the same image. This local dynamic mapping algorithm is consistent with the spatial feature transformation algorithm SFT (Spatial Feature Transform), so the SFT module can be used to build the image reconstruction model.

[0063] SFT is a mapping function that learns and outputs modulation parameter pairs based on some specific conditions. Specifically, the learnable parameters of the spatial feature transform adaptively affect the output of an affine transformation space, which is applied to each intermediate feature in an image. This method can change the behavior of the basic network by transforming the features of certain intermediate layers of the network. In the underlying visual tasks, it is often necessary to consider more image spatial information and perform different processing at different locations. The parameter pairs of SFT correspond to the dynamic characteristics of the model design; secondly, the SFT algorithm has high parameter efficiency. By transforming the intermediate features of a single network, only one forward pass is required to reconstruct HDR images with rich semantic areas.

[0064] In an optional embodiment of the present disclosure, when executing step S302, the attention information feature map can be subjected to dimensionality reduction processing to convert the image size of the attention information feature map to the same size as the LDR image; the attention information feature map after dimensionality reduction processing is input into the image reconstruction model to obtain a reconstructed HDR image.

[0065] This embodiment is based on the photoelectric conversion part in the camera imaging process, which can perform dimensionality reduction processing on the attention information feature map to convert the image size of the attention information feature map to the same size as the LDR image, and finally restore it to the size of the original attention information feature map to represent the conversion process.

[0066] In this regard, before using the local enhancement module to extract local spatial information, the attention information feature map with attention information can be pre-processed by dimensionality reduction to achieve the transformation of the feature to the size of the original LDR image. Figure 5A In , the feature map with attention information is converted into a feature map of the same size as the LDR image to serve as the input feature map of the image reconstruction model.

[0067] On the basis of the above embodiments, we continue to focus on the image reconstruction model, and input the attention information feature map or the attention information feature map after dimensionality reduction processing into the local enhancement module in the image reconstruction model.

[0068] In an optional embodiment of the present disclosure, the local enhancement module included in the image reconstruction model includes at least a residual block structure, and the residual block structure includes a dense connection layer, a local feature fusion layer and a local residual learning layer.

[0069] Among them, the dense connection layer and the local residual learning layer are used to extract local features from the attention information feature map, and the local feature fusion layer is used to fuse the local features extracted by the dense connection layer to obtain local spatial information.

[0070] Exemplarily, since there is a certain degree of distinction and correlation between the various attention information feature maps extracted based on step 301, the extraction of local spatial information at the feature level can adopt a residual block structure, and the residual module (Residual Block) includes a dense connection layer, a local feature fusion layer and a local residual learning layer, thereby forming a continuous storage mechanism.

[0071] Figure 5C A schematic diagram of the structure of a residual module provided in an embodiment of the present application; Figure 5CAs shown in the figure, the residual block consists of two convolutions, a Relu activation function and a jump connection. The local residual learning layer corresponds to the jump connection. The jump connection uses the ⊕ operation mode to act on the feature map. This method not only takes into account the global feature relationship, but also realizes adaptability at different spatial scales, and prevents the model from overfitting during the training process.

[0072] It should be explained that the dense connection layer composed of the direct connection of each convolutional layer output in the previous RB module of each feature map and all the subsequent convolutional layers not only retains the feedforward characteristics, but also extracts local dense features. The skip connection simplifies the learning objectives and difficulty, and protects the integrity of the original image information. After the image dimension is reduced, data training and feature information extraction can be more effective.

[0073] In an optional embodiment of the present disclosure, the residual block structure includes at least a first residual block structure and a second residual block structure, and the local enhancement module further includes a UNet network structure, the UNet network structure includes a downsampling convolution layer and an upsampling convolution layer located after the downsampling convolution layer, and the downsampling convolution layer is located before the first residual block structure, and the upsampling convolution layer is located before the second residual block structure; the downsampling convolution layer and the upsampling convolution layer are both composed of a plurality of convolution layers and pooling layers of different sizes; Among them, the downsampling convolution layer is used to reduce the size information of the attention information feature map, and increase the number of feature channels of the attention information feature map to obtain a first feature map; the first residual block structure extracts local features for the first feature map, and uses the upsampling convolution layer to restore the size information and the number of feature channels of the first feature map to the same as the size information and the number of feature channels of the attention information feature map to obtain a second feature map; the local features of the second feature map are feature-fused through the second residual block structure to obtain local spatial information.

[0074] For example, a UNet network structure may be added before the residual block structure. Figure 5A The spatial information extraction based on the LE model is shown in FIG. 5 , wherein the LE model is the local enhancement module (Local Enhance, referred to as LE).

[0075] The following will be combined Figure 5B The structure of the overall LE model is described.

[0076] Figure 5B This is a schematic diagram of the structure of a local enhancement module provided in an embodiment of the present application. Figure 5BAs shown in the figure, the LE module is a local enhancement module, which includes a UNet network structure and a residual module (i.e., a residual block structure). The residual module includes a feature fusion module. The UNet network structure has the ability to reconstruct images, and it can eliminate a certain degree of noise during the encoding and decoding process; the residual module is a plurality of RB modules, which fuse and extract the features after downsampling; and the feature fusion module is used to integrate all feature information again.

[0077] In the downsampling encoding stage, UNet consists of convolutional layers and pooling layers of different sizes. The input feature map is downsampled, and the resolution of the feature map is reduced by selecting the maximum or average value in the local area of ​​the input feature map or sliding the convolution kernel with a step size greater than 1. This process not only reduces the spatial dimension of the data and the computational complexity of the model, but also helps to extract more abstract feature representations, while expanding the network's receptive field, allowing the network to capture a wider range of contextual information.

[0078] After downsampling, the feature map passes through the residual module. Each residual module consists of Figure 5C The two convolutional layers shown in the figure are composed of a Relu activation function and a skip connection. Each convolutional layer output in the previous RB module of each feature map is directly connected to all subsequent convolutional layers, which not only retains the feedforward characteristics but also extracts local dense features. The skip connection simplifies the learning objectives and difficulty, and protects the integrity of the original image information. After image dimensionality reduction, data training and feature information extraction can be more effective.

[0079] Finally, the original feature map size is restored through the upsampling stage of UNet (it should be explained that the upsampling convolution layer corresponds one-to-one with the downsampling stage). After the features of the original size map are restored, two layers of convolution are performed for feature fusion. The Relu activation function in the network introduces nonlinearity in the neural network, enabling the model to learn and simulate complex function mappings, while affecting the back propagation of gradients and the convergence of the model.

[0080] In the overall process, the original feature map is downsampled to compress its spatial information into a feature map of smaller size, and the features of the spatial information are fused through the residual module. The original feature map size is restored through upsampling, and finally the features of the overall spatial information are fused to achieve the purpose of extracting spatial feature information.

[0081] In this embodiment, UNet is used as the basic network structure before the entire residual module because UNet has the ability to reconstruct images and eliminate a certain degree of noise during the encoding and decoding process.

[0082] The local enhancement module is used to extract the corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain an HDR image. Figure 5A “Local dynamic mapping based on SPF model” is used to output the processed feature map.

[0083] In an optional embodiment of the present disclosure, a spatial affine transformation is performed on local spatial information according to a spatial feature transformation module in an image reconstruction model to obtain image features of an HDR image; feature fusion is performed on the image features of the HDR image to obtain an initial HDR image; and tone mapping is performed on the initial HDR image to obtain a reconstructed HDR image.

[0084] Exemplarily, the image features output by the image reconstruction model already have the original HDR information. In order to visualize it on the display, the image features of the HDR image can be fused to obtain an initial HDR image, and then the initial HDR image can be tone mapped to obtain a reconstructed HDR image.

[0085] The following will be combined Figure 6 The specific implementation process of the above embodiment is described.

[0086] Figure 6 This is a schematic diagram of a structure of an HDR image visualization provided in an embodiment of the present application, referring to Figure 6 As shown, feature fusion and graphic image reconstruction are performed on the processed feature map output based on the above embodiment, and tone mapping is performed on the initial HDR image to obtain a visualized HDR image.

[0087] It can be understood that tone mapping is a computer graphics technique that approximates the display of high dynamic range images on limited dynamic range media. The core of the tone mapping algorithm (TMO) is a function that maps from one domain to another.

[0088] The ultimate goal of TMO is to use the transformation M:L→l so that L∈R + Mapped to display reference pixel value l∈N d = {0, .., 255}. L can be in arbitrary units, but for most color scale reconstruction algorithms, it is expected to be linearly related to the captured brightness values. However, different methods handle color scale reconstruction differently. Some TMO methods directly map values ​​to the range of 0-255; some methods first map the image to display brightness values ​​and then use a display model to convert to pixel values.

[0089] In an optional embodiment of the present disclosure, when performing the step of tone mapping the initial HDR image to obtain the reconstructed HDR image, tone mapping is performed on the initial HDR image based on a Gamma tone mapping function to obtain the reconstructed HDR image.

[0090] Exemplarily, after tone mapping using the Gamma tone mapping function, a better color effect is presented, and the algorithm is simple, thereby improving the efficiency of image reconstruction.

[0091] It should be explained that in addition to the Gamma tone mapping function used in the above embodiments, other tone mapping methods may also be included, such as the μ-Law tone mapping method, a high dynamic range image tone mapping algorithm based on a threshold intensity function, a local contrast stretching-based tone mapping algorithm, a time-adaptive tone mapping algorithm, and a display-adaptive tone mapping function.

[0092] Among them, the method based on threshold intensity function is obtained according to the detection threshold experiment on the full field of view; the high dynamic range image tone mapping algorithm based on local contrast stretching realizes tone mapping by local contrast stretching, aiming to improve the visual effect of HDR images on display devices; the time adaptation tone mapping algorithm is obtained based on a model simulating the time adaptation mechanism of HSV; the display adaptation tone mapping function is obtained by reconstructing the color scale and modeling it as an optimization problem to maximize the similarity between the input HDR image and the tone mapped image viewed on a specific display device.

[0093] In order to facilitate those skilled in the art to understand the HDR image reconstruction provided by the embodiments of the present application, the following will be combined with Figure 7 The image comparison effect diagram after being processed by the HDR image reconstruction method provided by the embodiment of the present application. Figure 7 As shown in the figure, the first column of images is the input LDR image, the middle column of images is the output initial HDR image, and the last column is the output visualization HDR image. Figure 7 From the comparison, we can see that compared with the original LDR image, the output HDR image restores clearer detail information in the overexposed and underexposed areas, and the overall contrast and saturation of the image are better adjusted, and the image quality is also better.

[0094] After the HDR image is obtained in step 302, the method further includes performing content review on the HDR image by a text and image content review system based on a multimodal large model. The text and image content review system includes an unstructured data processing module, a multimodal data integration module, a content analysis module, a content anomaly quantification module, and an intelligent control module. The review process at least includes: The unstructured data processing module is used to perform semantic analysis on the text in the HDR image and the unstructured data of the HDR image using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in the text and image through in-depth analysis, and construct a preliminary review data set; The multimodal data integration module is used to integrate the text data in the HDR image with the HDR image data in a unified context, convert it into a text-image association vector space, and transmit it to the content analysis module for in-depth analysis of the content context and emotion; The content analysis module is used to preliminarily determine the matching degree between the text in the HDR image and the HDR image content based on the semantic association graph generated by the unstructured data processing module, generate the context factor Qycs, and analyze the potential sensitive content in each modal data to generate the sensitivity factor Gmyz; then, based on the context factor Qycs and the sensitivity factor Gmyz, the potential emotional factors, social sensitive information and potential misleading factors in the image and text content are hierarchically analyzed to generate the multimodal association factor Mmgyz; The content anomaly quantification module calculates the audit weight value Hqwz based on the situational factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz; then obtains the system audit risk factor Rxz through calculation and associates it with the audit weight value Hqwz, thereby generating the final content anomaly index Cxzs; Among them, the calculation formula of the audit weight value Hqwz is:

[0095] Among them, b1, b2 and b3 represent the preset weight coefficients of the situational factor Qycs, the sensitive factor Gmyz and the multimodal association factor Mmgyz respectively, and B represents the audit correction coefficient; The intelligent control module is used to set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards, and compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0096] In an optional embodiment of the present application, the unstructured data processing module is first used to receive the unstructured data of the HDR image to be reviewed, and perform format standardization processing on the unstructured data, including image resolution adjustment, text language normalization processing and audio and video transcoding; Then, the pre-trained network of the multimodal large model is called to extract multi-level features of the HDR image, including visual features, semantic features, and emotional features, and the extracted multi-level features are normalized to generate a unimodal feature matrix. Based on the unimodal feature matrix, the multi-level features of the HDR image are mapped in the multimodal association vector space, and a cross-modal semantic association graph is generated through the semantic embedding mechanism of the multimodal large model. Secondly, the cross-modal semantic association graph is used to conduct a detailed analysis of the deep-level association between the text in the HDR image and the HDR image, extract the text in the HDR image and the core contextual elements in the HDR image, and simultaneously analyze the emotional state and expression trend in the audio and video, so as to identify the potential sensitive information, misleading information and social influencing factors in the news content in real time, and form a multi-modal association semantic index; Finally, a preliminary review dataset is generated.

[0097] In an optional embodiment of the present application, the multimodal data integration module is used to receive the preliminary review data set generated by the unstructured data processing module, and perform uniform alignment processing of the text data in the HDR image and the HDR image data in terms of timing and content through a frame synchronization algorithm; Then, a multimodal feature mapping algorithm is applied to fuse the context, emotion and semantic features of the HDR image into a multimodal feature vector, and a correlation matrix is ​​generated based on the correlation between each modality to perform hierarchical mapping of information density, context relevance and semantic consistency. Subsequently, the image-text association vector space is constructed based on the association matrix, and the multimodal data is quantified in a unified semantic space to generate an integrated semantic association representation vector. At the same time, the semantic deviation caused by modality differences is corrected through the context adjustment algorithm. Finally, the integrated image-text association vector space is transmitted to the content analysis module.

[0098] In an optional embodiment of the present application, the content analysis module includes a preliminary discrimination unit, a multi-layer sensitivity analysis unit, and a correlation factor generation unit; The preliminary discrimination unit identifies semantic deviation-related data and content matching-related data by analyzing the semantic similarity and contextual consistency between the image and the text; extracts the semantic similarity Ycy, contextual consistency Sjy, contextual correlation deviation Qjl, and image-text emotional consistency Twq from the semantic deviation-related data and content matching-related data and performs dimensionless processing, and then calculates the contextual factor Qycs using the following formula:

[0099] The preset scenario threshold Q is compared with the scenario factor Qycs to evaluate the contextual relevance of the text in the HDR image and the HDR image in the audit scenario. The specific comparative evaluation contents are as follows: If the context factor Qycs ≥ the context threshold Q, it indicates that the context between the text in the HDR image and the HDR image matches, and the semantics, emotions and context of the two are consistent. The review is passed and marked as "normal context association"; If the context factor Qycs < context threshold Q, it indicates that the text in the HDR image does not match the context between the HDR image, and the semantics, emotions and context of the two are inconsistent. The review fails and further analysis of the specific reasons for the deviation, including potential sensitive content, is required.

[0100] In an optional embodiment of the present application, a multi-layer sensitivity analysis unit is used to further perform in-depth analysis on the potential sensitive content of the graphic data, including analyzing the text in the HDR image and the emotion-related data, social sensitivity-related data and potential misleading-related data in the HDR image, and constructing a sensitive content data set after summarizing and dimensionless processing the text in the HDR image and the emotion-related data, social sensitivity-related data and potential misleading-related data in the HDR image; extracting the sensitive content data set and generating a sensitivity factor Gmyz through the following formula:

[0101] Where Qmq represents the emotion intensity coefficient in the sensitive content data set, Mgc represents the sensitive word index in the sensitive content data set, and Xwg represents the possibility of information misleading in the sensitive content data set.

[0102] In an optional embodiment of the present application, the correlation factor generation unit is used to quantify the correlation degree between emotion, context and sensitive content based on the context factor Qycs and the sensitivity factor Gmyz by fusing multi-level information in the image-text correlation vector space, and generate a multimodal correlation factor Mmgyz by calculating through the following formula; .

[0103] In an optional embodiment of the present application, the content anomaly quantification module includes an audit calculation unit and an anomaly index acquisition unit; The content evaluation unit is used to construct the system audit risk factor Rxz. The semantic deviation degree Ycy and content consistency index Nry are obtained by extracting semantic deviation related data and content matching related data. The high sensitivity trigger rate Gmg is obtained by extracting sensitive content data sets. The system audit risk factor Rxz is calculated by combining the following formula: .

[0104] In an optional embodiment of the present application, the abnormality index obtaining unit is used to calculate and obtain the final content abnormality index Cxzs, and its specific calculation formula is as follows:

[0105] In an optional embodiment of the present application, the intelligent control module compares and evaluates the audit weight value Hqwz with the compliance threshold T, and compares and evaluates the audit risk threshold R with the content anomaly index Cxzs, and specifically generates the following evaluation content: Compliance standards comparison: If the audit weight value Hqwz ≥ the compliance threshold T, it means that the content meets the audit standards and the system marks it as "compliant" If the audit weight value Hqwz < the compliance threshold T, it means that the content does not meet the audit standards and is insufficiently compliant, requiring further review or adjustment.

[0106] Abnormal risk assessment: If the content anomaly index Cxzs ≥ the audit risk threshold R, it indicates that the content has an abnormal risk, and the system generates an "abnormal content" mark, triggering the manual review process; If the content anomaly index Cxzs < the audit risk threshold R, it means that there is no abnormal risk in the content, and the system generates a "normal content" mark, and then enters the automatic publishing process; When the evaluation results of the audit weight value Hqwz and the content anomaly index Cxzs are marked as "compliant" and "normal content" at the same time, the content will be automatically marked as "passed" and published directly; When the evaluation result of the audit weight value Hqwz and the content anomaly index Cxzs is that the content does not meet the audit standards or has abnormal risks, the content will be automatically marked as "pending review" and the selective review process will be triggered.

[0107] It is used to set the compliance threshold T and the audit risk threshold R, and to determine whether the current content meets the audit standards, obtain the final audit results, and selectively trigger manual review or automatic release control plans based on the audit results.

[0108] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0109] To implement the above HDR image reconstruction method, see Figure 8 An embodiment of the present application provides an HDR image reconstruction device. The HDR image reconstruction device 800 may include: a feature extraction module 801 and an HDR image reconstruction module 802.

[0110] Among them, the feature extraction module 801 is used to obtain a low dynamic range LDR image, and perform feature extraction on the LDR image to obtain an attention information feature map for the LDR image; the HDR image reconstruction module 802 is used to input the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model includes at least a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain an HDR image.

[0111] In an optional embodiment of the present disclosure, the feature extraction module 801 is specifically used to input the LDR image into a pre-trained Transformer model to extract multiple deep local feature representations from the LDR image based on the Transformer model, determine the relative position of each deep local feature representation, and generate a relative position code of each deep local feature representation; input the multiple deep local feature representations and the relative position code of each deep local feature representation into a pre-trained SwinIR model for feature extraction to obtain an attention information feature map for the LDR image.

[0112] In an optional embodiment of the present disclosure, the HDR image reconstruction module 802 is specifically used to perform dimensionality reduction processing on the attention information feature map to convert the image size of the attention information feature map to the same size as the LDR image; the attention information feature map after dimensionality reduction processing is input into the image reconstruction model to obtain a reconstructed HDR image.

[0113] In an optional embodiment of the present disclosure, the local enhancement module includes at least a residual block structure, which includes a densely connected layer, a local feature fusion layer and a local residual learning layer; wherein the densely connected layer and the local residual learning layer are used to extract local features from the attention information feature map, and the local feature fusion layer is used to fuse the local features extracted by the densely connected layer to obtain local spatial information.

[0114] In an optional embodiment of the present disclosure, the residual block structure includes at least a first residual block structure and a second residual block structure, and the local enhancement module further includes a UNet network structure, the UNet network structure includes a downsampling convolution layer and an upsampling convolution layer located after the downsampling convolution layer, and the downsampling convolution layer is located before the first residual block structure, and the upsampling convolution layer is located before the second residual block structure; the downsampling convolution layer and the upsampling convolution layer are both composed of a plurality of convolution layers and pooling layers of different sizes; Among them, the downsampling convolution layer is used to reduce the size information of the attention information feature map, and increase the number of feature channels of the attention information feature map to obtain a first feature map; the first residual block structure extracts local features for the first feature map, and uses the upsampling convolution layer to restore the size information and the number of feature channels of the first feature map to the same as the size information and the number of feature channels of the attention information feature map to obtain a second feature map; the local features of the second feature map are feature-fused through the second residual block structure to obtain local spatial information.

[0115] In an optional embodiment of the present disclosure, the device may also include a transformation module, a feature fusion module and a tone mapping module, the transformation module is used to perform spatial affine transformation on local spatial information according to the spatial feature transformation module in the image reconstruction model to obtain image features of the HDR image; the feature fusion module is used to perform feature fusion on the image features of the HDR image to obtain an initial HDR image; the tone mapping module is used to perform tone mapping on the initial HDR image to obtain a reconstructed HDR image.

[0116] In an optional embodiment of the present disclosure, tone mapping is performed on the initial HDR image based on a Gamma tone mapping function or a μ-Law tone mapping function to obtain a reconstructed HDR image.

[0117] For the specific definition of the above-mentioned HDR image reconstruction device, please refer to the definition of the HDR image reconstruction method above, which will not be repeated here. Each module in the above-mentioned HDR image reconstruction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0118] In one embodiment, a computer device is provided, the internal structure diagram of which can be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an HDR image reconstruction method as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the HDR image reconstruction method as described above is implemented.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above HDR image reconstruction method can be implemented.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0125] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A HDR image reconstruction method, characterized in that: include: Acquire a low dynamic range (LDR) image, and perform feature extraction on the LDR image to obtain an attention information feature map for the LDR image; The attention information feature map is input into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model at least includes a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain the HDR image.

2. The method according to claim 1, characterized in that The step of extracting features from the LDR image to obtain an attention information feature map for the LDR image includes: Inputting the LDR image into a pre-trained Transformer model, extracting a plurality of deep local feature representations from the LDR image based on the Transformer model, determining a relative position of each of the deep local feature representations, and generating a relative position encoding of each of the deep local feature representations; The multiple deep local feature representations and the relative position encoding of each of the deep local feature representations are input into a pre-trained SwinIR model for feature extraction to obtain an attention information feature map for the LDR image.

3. The method according to claim 1, characterized in that The step of inputting the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image comprises: Performing dimensionality reduction processing on the attention information feature map to convert the image size of the attention information feature map to be the same as the size of the LDR image; The attention information feature map after dimensionality reduction processing is input into the image reconstruction model to obtain a reconstructed HDR image.

4. The method according to claim 1, characterized in that: The local enhancement module at least includes a residual block structure, which includes a densely connected layer, a local feature fusion layer and a local residual learning layer; wherein the densely connected layer and the local residual learning layer are used to extract local features from the attention information feature map, and the local feature fusion layer is used to perform feature fusion on the local features extracted by the densely connected layer to obtain the local spatial information.

5. The method according to claim 4, characterized in that The residual block structure at least includes a first residual block structure and a second residual block structure, and the local enhancement module further includes a UNet network structure, the UNet network structure includes a downsampling convolution layer and an upsampling convolution layer located after the downsampling convolution layer, and the downsampling convolution layer is located before the first residual block structure, and the upsampling convolution layer is located before the second residual block structure; the downsampling convolution layer and the upsampling convolution layer are both composed of a plurality of convolution layers and pooling layers of different sizes; Among them, the downsampling convolution layer is used to reduce the size information of the attention information feature map and increase the number of feature channels of the attention information feature map to obtain a first feature map; the first residual block structure extracts the local features for the first feature map, and uses the upsampling convolution layer to restore the size information and the number of feature channels of the first feature map to the same as the size information and the number of feature channels of the attention information feature map to obtain a second feature map; the local features of the second feature map are feature-fused through the second residual block structure to obtain the local spatial information.

6. The method according to claim 1, characterized in that The method further comprises: Performing spatial affine transformation on the local spatial information according to the spatial feature transformation module in the image reconstruction model to obtain image features of the HDR image; Performing feature fusion on the image features of the HDR image to obtain an initial HDR image; Perform tone mapping on the initial HDR image to obtain a reconstructed HDR image.

7. The method according to claim 6, characterized in that The step of performing tone mapping on the initial HDR image to obtain a reconstructed HDR image includes: Based on the Gamma tone mapping function or the μ-Law tone mapping function, tone mapping is performed on the initial HDR image to obtain a reconstructed HDR image.

8. An HDR image reconstruction device, characterized in that: include: A feature extraction module, used for acquiring a low dynamic range (LDR) image, and performing feature extraction on the LDR image to obtain an attention information feature map for the LDR image; An HDR image reconstruction module is used to input the attention information feature map into a pre-trained image reconstruction model to obtain a reconstructed high dynamic range HDR image; wherein the image reconstruction model at least includes a local enhancement module and a spatial feature transformation module, the local enhancement module is used to extract corresponding local spatial information from the attention information feature map, and the spatial feature transformation module is used to perform spatial affine transformation on the local spatial information to obtain the HDR image.

9. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the HDR image reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the HDR image reconstruction method according to any one of claims 1 to 7 are implemented.

Images