Image reconstruction method, electronic device, storage medium and computer program product

Abstract

Embodiments of the present application provide an image reconstruction method, an electronic device, a computer readable storage medium and a computer program product. The method comprises: obtaining a plurality of to-be-processed images, the to-be-processed images being LDR images; performing feature extraction on the plurality of to-be-processed images to obtain extracted image features; performing global context information extraction and local context information extraction based on the extracted image features to obtain global context feature information and local context feature information, and combining the global context feature information and the local context feature information to obtain a reconstructed image, the reconstructed image being a high dynamic range image. The ghosting problem in a large foreground motion scene can be effectively solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically to an image reconstruction method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In the field of digital image processing, dynamic range refers to the range of light intensity in a scene that an image can capture. The dynamic range of a natural scene observable by the human eye can reach 10,000:1, while ordinary consumer-grade photography devices (such as mobile phones) can often only capture limited low dynamic range (LDR) images (with a dynamic range of, for example, 100:1 to 300:1). Compared to LDR images, high dynamic range (HDR) images have a wider dynamic range and can more realistically reproduce the light and shadow effects of a scene, thus producing photos with richer detail, more realistic images, and higher quality.

[0003] There are two ways to obtain HDR images: the first is to capture HDR images directly with a specific device, but such devices are bulky and expensive, making them unavailable in consumer electronics (such as smartphones); the second method is to obtain HDR images by fusing multiple LDR images with different exposures (i.e., multi-exposure HDR image reconstruction), but due to camera shake or movement of foreground objects, multi-frame fusion is prone to ghosting problems.

[0004] Of the two methods described above, the second method, multi-exposure HDR image reconstruction in dynamic scenes, is a more commonly used method for obtaining HDR images. To obtain high-quality HDR images, it is necessary to address the ghosting problem caused by foreground motion or camera shake when fusing multiple LDR images with different exposure intensities. In particular, the ghosting problem becomes extremely difficult to resolve when there are large moving objects or rapid changes in brightness in the scene. Summary of the Invention

[0005] This application is made in view of the above-mentioned problems. This application provides an image reconstruction method, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this application, an image reconstruction method is provided, comprising: acquiring multiple images to be processed, wherein the images to be processed are low dynamic range images; performing feature extraction on the multiple images to be processed to obtain extracted image features; performing global context information extraction and local context information extraction based on the extracted image features to obtain global context feature information and local context feature information, and combining the global context feature information and local context feature information to obtain a reconstructed image, wherein the reconstructed image is a high dynamic range image.

[0007] For example, the method of extracting global and local context information based on extracted image features to obtain global context feature information and local context feature information, and combining the global and local context feature information to obtain a reconstructed image, includes: sequentially performing at least one reconstruction operation, wherein the reconstruction operation includes at least one feature processing operation performed sequentially, the feature processing operation including: extracting global context information from the image features to be processed to obtain global context features, wherein the image features to be processed corresponding to the first feature processing operation in the first reconstruction operation are extracted image features, the image features to be processed corresponding to the first feature processing operation in non-first reconstruction operations are the execution result of the previous reconstruction operation, and the image features to be processed corresponding to non-first feature processing operations in all reconstruction operations are the execution result of the previous feature processing operation, and the global context feature information includes global context features; extracting local context information from the image features to be processed to obtain local context features, the local context feature information including local context features; fusing the global context features and local context features together to obtain the execution result of the feature processing operation; and combining the global and local context feature information to obtain a reconstructed image, including: obtaining a reconstructed image based on the execution result of the last reconstruction operation, wherein the reconstructed image is a high dynamic range image.

[0008] For example, extracting local context information from the features of the image to be processed to obtain local context features includes: performing layer normalization on the features of the image to be processed to obtain normalized features; shaping the normalized features to convert them into two-dimensional image feature forms to obtain shaped features; convolving the shaped features to obtain first convolution features; performing channel attention processing on the first convolution features to obtain channel attention features; and shaping the channel attention features to convert them into block-embedded feature forms to obtain local context features.

[0009] For example, feature extraction of multiple images to be processed to obtain extracted image features includes: performing initial feature extraction on multiple images to be processed to obtain initial image features; and shaping the initial image features to convert them into block-embedded feature form to obtain extracted image features.

[0010] For example, initial feature extraction is performed on multiple images to be processed to obtain initial image features, including: convolving each of the multiple images to be processed to obtain multiple second convolution features corresponding one-to-one with the multiple images to be processed; for any non-reference image, spatial attention processing is performed on the second convolution features corresponding to the non-reference image and the second convolution features corresponding to the reference image to obtain spatial attention features corresponding to the non-reference image, wherein the reference image is one of the multiple images to be processed, and the non-reference images are the images other than the reference image among the multiple images to be processed; the spatial attention features corresponding to all non-reference images and the second convolution features corresponding to the reference images are fused to obtain the initial image features.

[0011] For example, the reference image is the image whose exposure value is closest to 0 among a plurality of images to be processed, or the reference image is one of the images whose exposure value is in the middle among a plurality of images to be processed.

[0012] For example, obtaining a reconstructed image based on the execution result of the last reconstruction operation includes: merging the execution result of the last reconstruction operation with the initial image features and / or the second convolutional features corresponding to the reference image to obtain intermediate output features; convolving the intermediate output features to obtain third convolutional features; and mapping the third convolutional features through an activation function to obtain the reconstructed image.

[0013] For example, obtaining a reconstructed image based on the execution result of the last reconstruction operation includes: merging the execution result of the last reconstruction operation with the initial image features to obtain intermediate output features; convolving the intermediate output features to obtain fourth convolutional features; and mapping the fourth convolutional features through an activation function to obtain the reconstructed image.

[0014] For example, the reconstruction operation also includes a convolution operation performed on the result of the last feature processing operation. In each reconstruction operation, the image features to be processed corresponding to the first feature processing operation are combined with the result of the convolution operation to obtain the result of the reconstruction operation.

[0015] For example, the operations of feature extraction from multiple images to be processed, global context information extraction based on the extracted image features, and local context information extraction combined with global and local context features to obtain a reconstructed image are implemented through a reconstruction network model. The method further includes: acquiring training images, which include multiple sample images and corresponding labeled images, wherein the sample images are low dynamic range images and the labeled images are high dynamic range images; processing multiple sample images using an initial reconstruction network model to obtain a predicted reconstructed image; calculating a reconstruction loss term based on the predicted reconstructed image and the labeled image; acquiring the predicted image features and labeled image features corresponding to the predicted reconstructed image and the labeled image respectively; calculating a perceptual loss term based on the predicted image features and the labeled image features; calculating a total loss based on the reconstruction loss term and the perceptual loss term; and optimizing the parameters of the initial reconstruction network model based on the total loss to obtain a reconstruction network model.

[0016] For example, obtaining the predicted image features and labeled image features corresponding to the predicted reconstructed image and the labeled image respectively includes: performing tone mapping on the predicted reconstructed image and the labeled image respectively to obtain a new predicted reconstructed image and a new labeled image; and inputting the new predicted reconstructed image and the new labeled image into a pre-trained network model respectively to obtain their respective predicted image features and labeled image features.

[0017] According to another aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the above-described image reconstruction method.

[0018] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program / instructions are stored, characterized in that the computer program / instructions, when executed by a processor, implement the above-described image reconstruction method.

[0019] According to another aspect of this application, a computer program product is provided, including a computer program / instructions, characterized in that the computer program / instructions implement the above-described image reconstruction method when executed by a processor.

[0020] According to the image reconstruction method, electronic device, computer-readable storage medium, and computer program product of this application, global context information extraction and local context information extraction are performed based on the image features of LDR images, and the global and local context feature information are combined to obtain HDR images. This approach can simultaneously aggregate long-range and local context information of an image. When this processing method is applied to HDR image reconstruction, it can effectively solve the ghosting problem in scenes with large foreground motion. Attached Figure Description

[0021] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 A schematic block diagram of an example electronic device for implementing the image feature processing or image reconstruction method and apparatus according to embodiments of this application is shown;

[0023] Figure 2 A schematic flowchart of an image feature processing method according to an embodiment of this application is shown;

[0024] Figure 3 This diagram illustrates the network structure of a context-aware visual converter module according to an embodiment of the present application.

[0025] Figure 4 A schematic flowchart illustrating an image reconstruction method according to an embodiment of this application is shown;

[0026] Figure 5 A schematic diagram illustrating the processing flow of reconstructing a network model according to an embodiment of this application is shown;

[0027] Figure 6 A schematic block diagram of an image feature processing apparatus according to an embodiment of the present application is shown;

[0028] Figure 7 A schematic block diagram of an image reconstruction apparatus according to an embodiment of this application is shown;

[0029] Figure 8 A schematic block diagram of an electronic device according to one embodiment of this application is shown; and

[0030] Figure 9 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0031] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, image reconstruction, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0032] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0033] To at least partially address the aforementioned technical problems, embodiments of this application provide an image reconstruction method, an electronic device, a computer-readable storage medium, and a computer program product. The image reconstruction method according to embodiments of this application can simultaneously aggregate both long-range and local contextual information of an image. When this processing method is applied to HDR image reconstruction, it can effectively solve the ghosting problem in scenes with large foreground motion. The image reconstruction technology according to embodiments of this application can be applied to any field requiring the generation of HDR images.

[0034] First, refer to Figure 1This describes an example electronic device 100 for implementing image feature processing or image reconstruction methods and apparatus according to embodiments of this application.

[0035] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may also include an input device 106, an output device 108, and an image acquisition device 110, these components being interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0036] The processor 102 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic array (PLA), and microprocessor. The processor 102 may be one or a combination of several of the following: central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or other processing units with data processing capabilities and / or instruction execution capabilities. It may also control other components in the electronic device 100 to perform the desired functions.

[0037] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this application described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0038] The input device 106 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0039] The output device 108 can output various information (e.g., images and / or sound) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0040] The image acquisition device 110 can acquire images and store the acquired images in the storage device 104 for use by other components. The image acquisition device 110 can be a standalone camera or a camera in a mobile terminal, etc. It should be understood that the image acquisition device 110 is only an example, and the electronic device 100 may not include the image acquisition device 110. In this case, other devices with image acquisition capabilities can be used to acquire images and send the acquired images to the electronic device 100.

[0041] For example, an example electronic device for implementing the image feature processing or image reconstruction method and apparatus according to the embodiments of this application can be implemented on a device such as a personal computer, terminal device, time and attendance machine, panel machine, camera, or remote server. The terminal device includes, but is not limited to, tablet computers, mobile phones, PDAs (Personal Digital Assistants), touchscreen all-in-one machines, wearable devices, etc.

[0042] This application employs a novel image feature processing method to aggregate long-range and local contextual information of an image, and then applies this image feature processing method to the field of image reconstruction to solve the ghosting problem. For ease of understanding, the following first refers to... Figure 2-3 This application describes an image feature processing method according to an embodiment of the present application. Figure 2 A schematic flowchart of an image feature processing method 200 according to an embodiment of this application is shown. Figure 2 As shown, the image feature processing method 200 includes steps S210, S220, S230 and S240.

[0043] In step S210, the features of the image to be processed are obtained, which are based on the image features extracted from the image to be processed.

[0044] The image to be processed here can be any image. Exemplarily, but not limitingly, the image to be processed can be a raw Low Dynamic Range (LDR) image acquired by an image acquisition device, or an LDR image obtained by performing certain preprocessing on a raw LDR image. Exemplarily, preprocessing can include operations such as smoothing, filtering, and normalization.

[0045] Image features extracted from the image to be processed can undergo any suitable further processing to obtain the final image features. Such further processing can include things like reshaping. For example, reshaping can convert the image features extracted from the image to be processed from a two-dimensional H×W×C form into a patchbeddings form, where H represents height, W represents width, and C represents the number of channels. The image features to be processed are the input features of the Context-aware Vision Transformer (CA-ViT) module, i.e., the features that need to be processed by the CA-ViT module.

[0046] In step S220, global context information is extracted from the features of the image to be processed to obtain global context features. For example, the visual converter module in the context-aware visual converter module can be used to extract global context information from the features of the image to be processed to obtain global context features.

[0047] The image features to be processed can be input into the Vision Transformer (ViT) module to obtain the global context features output by the ViT module. The ViT module used in step S220 can be a conventional ViT. Those skilled in the art will understand the network structure and working principle of a conventional ViT, which will not be elaborated here.

[0048] In step S230, local context information is extracted from the features of the image to be processed to obtain local context features. For example, the local context extraction module in the context-aware visual converter module can be used to extract local context information from the features of the image to be processed to obtain local context features.

[0049] Optionally, steps S220 and S230 can be executed simultaneously.

[0050] Figure 3 A schematic diagram of the network structure of a CA-ViT module according to an embodiment of this application is shown. Figure 3 As shown, the CA-ViT module includes two parallel branches: the ViT module 302 and the local context extraction module 304. For example... Figure 3As shown, the ViT module 302 mainly includes a multi-head self-attention (MSA) layer and a multi-layer perceptron (MLP). Furthermore, the ViT module 302 may also include at least one layer normalization (LN) layer. Those skilled in the art will understand the network structure and working principle of the ViT module 302, which will not be elaborated upon here. The image features to be processed can be patch embeddings. By inputting the image features to be processed into the ViT module 302, the ViT module 302 can output global contextual features. That is, global contextual information can be extracted through the ViT module 302.

[0051] Local context information can be extracted using the local context extraction module 304. For example... Figure 3 As shown, the local context extraction module 304 may include a local context extractor. The local context extractor may include a first convolutional module and a channel attention module. The first convolutional module and the channel attention module can extract local context information from the image. However, it is understood that... Figure 3 The network structure shown is merely an example and not a limitation of this application. This application is not limited to this. Figure 3 The network structure is shown. For example, the first convolutional module in the local context extraction module 304 can be replaced by other network structures such as pooling layers, as long as the extraction of local context information can be achieved.

[0052] In step S240, the global context features and local context features are fused together to obtain the execution result of the image feature processing method 200. Fusing the global and local context features together yields the output features of the context-aware visual converter module, i.e., the execution result of the image feature processing method 200.

[0053] like Figure 3 As shown, a fusion module exists after the output of the ViT module 302 and the local context extraction module 304, which is used to fuse global context features and local context features together.

[0054] Optionally, global context features and local context features can be fused together by channel concatenation.

[0055] The image feature processing method according to the embodiments of this application obtains features (i.e., execution results) that carry global context information and local context information. That is, it can simultaneously aggregate the long-range and local context information of the image, and significantly improves the performance of existing visual converters in tasks such as HDR image reconstruction.

[0056] For example, the image feature processing method according to the embodiments of this application can be implemented in a device, apparatus or system having a memory and a processor.

[0057] The image feature processing method according to the embodiments of this application can be deployed at the image acquisition end, for example, at a personal terminal or a server.

[0058] Alternatively, the image feature processing method according to the embodiments of this application can also be deployed distributedly on a server (or cloud) and a personal terminal. For example, an LDR image can be captured on the client side, and the client side can transmit the captured LDR image to the server (or cloud) side, whereby the server (or cloud) side extracts features from it to obtain the image features to be processed for image feature processing.

[0059] According to an embodiment of this application, extracting local context information from the features of the image to be processed to obtain local context features (step S230) may include: performing layer normalization processing on the features of the image to be processed to obtain normalized features; shaping the normalized features to convert them into two-dimensional image features to obtain shaped features; performing convolution on the shaped features to obtain first convolution features; performing channel attention processing on the first convolution features to obtain channel attention features; and shaping the channel attention features to convert them into block-embedded features to obtain local context features.

[0060] For example, the local context extraction module includes a layer normalization module, a first shaping module, a first convolution module, a channel self-attention module, and a second shaping module. The local context extraction module in the context-aware visual converter module extracts local context information from the image features to be processed, obtaining local context features by: inputting the image features to be processed into the layer normalization module for layer normalization processing to obtain normalized features; inputting the normalized features into the first shaping module for shaping to convert the normalized features into a two-dimensional image feature form to obtain shaped features; inputting the shaped features into the first convolution module for convolution to obtain first convolution features; inputting the first convolution features into the channel attention module for channel attention processing to obtain channel attention features; and inputting the channel attention features into the second shaping module for shaping to convert the channel attention features into a block-embedded feature form to obtain local context features.

[0061] Figure 3 The local context extraction module 304 includes an LN module, a first convolution module, and a channel self-attention module. The first shaping module and the second shaping module are not shown.

[0062] When the image features to be processed are patch embeddings, they can be further processed before being input into the local context extractor, for example, converted into two-dimensional image features. For example, the local context extraction module 304 may include a layer normalization (LN) layer and / or a reshape layer. For instance, the image features to be processed can be input into the LN layer for normalization. The normalized features can then be input into the reshape layer for feature transformation, converting the features from patch embeddings to two-dimensional image features, i.e., features that can be expressed as H×W×C. Subsequently, the H×W×C dimensional features can be input into the first convolutional module. The dimension of the features output by the first convolutional module can also be H×W×C. Of course, for any parameter among height H, width W, and number of channels C, the output features of the first convolutional module may or may not be the same as the input features.

[0063] The first convolutional module may include at least one convolutional layer. The number of convolutional layers in the first convolutional module can be arbitrary, and this application does not impose any limitation on this. The channel attention module may include pooling layers, linear layers, activation function layers, dimensionality expansion, and other network layers. Figure 3 The pooling layer and dimension expansion layer are not explicitly shown, but their operations are illustrated at their corresponding points in the workflow. The number of layers within the channel attention module can be set to an appropriate size as needed, and this application does not impose any limitations on this. Figure 3 In the example shown, the channel attention module includes two linear layers and two activation function layers. The two activation function layers are a ReLU layer and a Sigmoid layer.

[0064] The features output from the first convolutional module are input into a pooling layer, which can output features of 1×1×C dimension. The pooling layer can be an average pooling layer or a max pooling layer, etc. After several linear layers and several activation function layers, the output features still maintain a 1×1×C dimension. To multiply with the features output from the first convolutional module, the features output from the last activation function layer can be dimension-expanded, transforming them back to H×W×C size. Multiplying the transformed features element-wise with the features output from the first convolutional module yields the output features of the channel attention module, i.e., the channel attention features.

[0065] Channel attention features are in the form of H×W×C. The features output by the MLP in the ViT module are in the form of patch embeddings. To fuse these two types of features, one of them can be transformed. For example, patch embedding can be computed on the channel attention features to obtain features in the form of patch embeddings as local context features. Subsequently, the global context features and local context features can be fused together using methods such as channel concatenation.

[0066] According to an embodiment of this application, the channel attention module includes at least one linear layer and an activation function layer.

[0067] This embodiment has been described above and will not be repeated here.

[0068] The following describes an implementation scheme according to an embodiment of this application for solving the ghosting problem in scenes with large foreground motion using the CA-ViT module described above.

[0069] According to another aspect of this application, an image reconstruction method is provided. Figure 4 A schematic flowchart of an image reconstruction method 400 according to an embodiment of this application is shown. Figure 4 As shown, the image reconstruction method 400 includes steps S410, S420, S430 and S440.

[0070] In step S410, multiple images to be processed are acquired. These images are low dynamic range images.

[0071] Those skilled in the art will understand that to reconstruct a given HDR image, the multiple LDR images used as the basis are theoretically images with similar content but different brightness. For example, multiple LDR images can be multiple images continuously captured by the same camera from the same viewpoint of the same scene. The scene can include the target object (foreground) and the parts other than the target object (background). In multiple LDR images, the background can remain basically unchanged, while the foreground can be static or moving. When the foreground has significant movement, the aforementioned ghosting problem will exist.

[0072] In step S420, feature extraction is performed on multiple images to be processed to obtain extracted image features. For example, multiple images to be processed can be input into the feature extraction module of the reconstruction network model for feature extraction to obtain extracted image features.

[0073] In step S430, global context information extraction and local context information extraction are performed based on the extracted image features to obtain global context feature information and local context feature information. The reconstructed image is obtained by combining the global context feature information and local context feature information. The reconstructed image is a high dynamic range image.

[0074] Global contextual feature information and local contextual feature information can be extracted using any suitable existing or future global contextual information extraction and local contextual information extraction methods, such as using a neural network model. Exemplary extraction methods will be described below.

[0075] According to the image reconstruction method of this application, global context information is extracted and local context information is extracted based on the image features of the LDR image, and the global and local context features are combined to obtain the HDR image. This approach can simultaneously aggregate the long-range and local context information of the image. When this processing method is applied to HDR image reconstruction, it can effectively solve the ghosting problem in scenes with large foreground motion.

[0076] For example, the image reconstruction method according to the embodiments of this application can be implemented in a device, apparatus or system having a memory and a processor.

[0077] The image reconstruction method according to the embodiments of this application can be deployed at the image acquisition end, for example, at a personal terminal or a server.

[0078] Alternatively, the image reconstruction method according to embodiments of this application can also be deployed distributedly on a server (or cloud) and at a personal terminal. For example, an LDR image can be acquired on the client side, and the client side can transmit the acquired LDR image to the server (or cloud) side for image reconstruction.

[0079] For example, extracting global and local context information based on extracted image features to obtain global and local context feature information, and combining the global and local context feature information to obtain a reconstructed image, wherein the reconstructed image is a high dynamic range image, may include: sequentially executing at least one reconstruction operation, wherein the reconstruction operation includes at least one feature processing operation executed sequentially, the feature processing operation including: extracting global context information from the image features to be processed to obtain global context features, wherein the image features to be processed corresponding to the first feature processing operation in the first reconstruction operation are extracted image features, the image features to be processed corresponding to the first feature processing operation in non-first reconstruction operations are the execution result of the previous reconstruction operation, and the image features to be processed corresponding to non-first feature processing operations in all reconstruction operations are the execution result of the previous feature processing operation, and the global context feature information includes global context features; extracting local context information from the image features to be processed to obtain local context features, the local context feature information including local context features; fusing the global and local context features together to obtain the execution result of the feature processing operation; and obtaining a reconstructed image based on the execution result of the last reconstruction operation, wherein the reconstructed image is a high dynamic range image.

[0080] Reconstruction operations can be performed once or multiple times, and feature processing operations within any reconstruction operation can also be performed once or multiple times. The result of the last reconstruction operation can be further processed to obtain an HDR image. For example, and not limitingly, the result of the last reconstruction operation can be convolved, and the features obtained from the convolution can then be processed through an activation function to obtain the desired HDR image.

[0081] For example, the extracted image features can be input into the reconstruction module of the reconstruction network model for processing to obtain a reconstructed image, which is a high dynamic range image. The reconstruction module includes at least one sequentially connected perception module. Each perception module includes at least one sequentially connected sub-perception module and a second convolutional module connected to the last sub-perception module. Each sub-perception module is the aforementioned context-aware visual converter module, and the input features of each sub-perception module are the image features to be processed corresponding to that sub-perception module. The feature processing operation is the operation included in the aforementioned image feature processing method 200. The feature processing operation can correspond one-to-one with the sub-perception module, meaning each sub-perception module can perform one feature processing operation. The reconstruction operation corresponds one-to-one with the perception module, meaning each perception module can perform one reconstruction operation.

[0082] Figure 5This diagram illustrates the processing flow of a reconstruction network model according to an embodiment of the present application. The reconstruction network model is a network model used to reconstruct an HDR image based on multiple LDR images. Figure 5 As shown, the reconstruction network model may include a feature extraction module 502 and a reconstruction module 504.

[0083] The feature extraction module 502 can be implemented using any suitable network structure, as long as it can extract features that meet the input requirements of the reconstruction module 504. For example, the features output by the feature extraction module 502 can be features in the form of patch beddings.

[0084] The reconstruction module 504 may include at least one perception module, and each perception module may include at least one sub-perception module and a second convolutional module. Each sub-perception module is the CA-ViT module described above. There can be N perception modules, where N ≥ 1. Note that... Figure 5 Only a single sensing module is shown; in the case of multiple sensing modules, they can be connected sequentially end-to-end. The second convolutional module can be any suitable convolutional module, which may include at least one convolutional layer. In one example, the second convolutional module may include a dilated convolutional layer.

[0085] The extracted features output by feature extraction module 502 are input to the first perceptual module in reconstruction module 504, and the features output by the first perceptual module are input to the second perceptual module in reconstruction module 504, and so on. Within each perceptual module, sub-perceptual modules are also sequentially connected, so the input features of a later sub-perceptual module are the output features of the previous sub-perceptual module. For example, the output features of each sub-perceptual module can also be features in the form of patch embeddings.

[0086] According to embodiments of this application, extracting local contextual information from the features of the image to be processed to obtain local contextual features includes: performing layer normalization processing on the features of the image to be processed to obtain normalized features; shaping the normalized features to convert them into two-dimensional image feature forms to obtain shaped features; performing convolution on the shaped features to obtain first convolutional features; performing channel attention processing on the first convolutional features to obtain channel attention features; and shaping the channel attention features to convert them into block-embedded feature forms to obtain local contextual features.

[0087] The method for extracting local contextual features has already been described above, and will not be repeated here.

[0088] According to an embodiment of this application, feature extraction of multiple images to be processed to obtain extracted image features includes: performing initial feature extraction on multiple images to be processed to obtain initial image features; and shaping the initial image features to convert them into block-embedded feature form to obtain extracted image features.

[0089] For example, the above feature extraction module may include an initial feature extraction module and a third shaping module. The process of inputting multiple images to be processed into the feature extraction module in the reconstruction network model to extract features and obtain extracted image features includes: inputting multiple images to be processed into the initial feature extraction module to extract initial features and obtain initial image features; and inputting the initial image features into the third shaping module to shape them into block-embedded feature form to obtain extracted image features.

[0090] Features for each LDR image can be extracted separately in the initial feature extraction module. This module then outputs the combined result of these features, i.e., the initial image features. Subsequently, the initial image features output by the initial feature extraction module can be converted from an H×W×C feature format to a patch embeddings feature format. This feature format conversion facilitates obtaining suitable features for processing in the sub-perception module.

[0091] According to an embodiment of this application, initial feature extraction is performed on multiple images to be processed to obtain initial image features, including: performing convolution on each of the multiple images to be processed to obtain multiple second convolution features corresponding one-to-one with the multiple images to be processed; for any non-reference image, performing spatial attention processing on the second convolution features corresponding to the non-reference image and the second convolution features corresponding to the reference image to obtain spatial attention features corresponding to the non-reference image, wherein the reference image is one of the multiple images to be processed, and the non-reference images are images other than the reference image among the multiple images to be processed; and fusing the spatial attention features corresponding to all non-reference images and the second convolution features corresponding to the reference images to obtain initial image features.

[0092] For example, the initial feature extraction module may include a third convolution module, a spatial attention module, and a fusion module. Inputting multiple images to be processed into the initial feature extraction module for feature extraction to obtain initial image features includes: inputting the multiple images to be processed into the third convolution module for convolution to obtain multiple second convolution features corresponding one-to-one with the multiple images to be processed; for any non-reference image, inputting the second convolution features corresponding to the non-reference image and the second convolution features corresponding to the reference image into the spatial attention module to obtain the spatial attention features corresponding to the non-reference image, wherein the reference image is one of the multiple images to be processed, and the non-reference images are images other than the reference image among the multiple images to be processed; inputting the spatial attention features corresponding to all non-reference images and the second convolution features corresponding to the reference images into the fusion module for fusion to obtain the initial image features.

[0093] The third convolutional module may include at least one sub-convolutional module. In one example, the third convolutional module may include multiple sub-convolutional modules corresponding one-to-one with multiple images to be processed, and each sub-convolutional module may include at least one convolutional layer. Each sub-convolutional module can be used to convolve the corresponding image to be processed.

[0094] Optionally, a reference image can be pre-selected from multiple images to be processed, and the remaining images can be used as non-reference images. The second convolutional features corresponding to each non-reference image can be concatenated with the second convolutional features corresponding to the reference image and then input into the spatial attention module for spatial attention calculation. Exemplarily, but not limitingly, the number of spatial attention modules can be one less than the number of images to be processed. This facilitates inputting the corresponding features into the corresponding spatial attention module for processing for each pair of non-reference and reference image combinations. Of course, there can also be, for example, only one spatial attention module, into which each pair of non-reference and reference image combinations can be sequentially input for processing.

[0095] refer to Figure 5 This shows three images to be processed. Figure 5In the illustrated embodiment, the second image to be processed, with the exposure value at the middle, is used as the reference image, while the other two are used as non-reference images. First, the three images to be processed are processed through three convolutional layers (Conv) to extract shallow features, yielding three convolutional features f1, f2, and f3 respectively. Then, convolutional features f1 and f2 are concatenated (Concat), and the concatenated features are fed into a spatial attention module to obtain spatial attention feature f1'. The spatial attention module may include at least one convolutional layer and at least one activation function layer. In one example, the spatial attention module consists of two convolutional layers and a sigmoid activation function layer. Furthermore, convolutional features f3 and f2 can be concatenated (Concat), and the concatenated features are fed into another spatial attention module to obtain spatial attention feature f3'. Finally, spatial attention features f1', spatial attention feature f3', and convolutional feature f2 are fused together to obtain the initial image feature f. att By way of example, and not limitation, the fusion module may include a stitching module and a fourth convolution module. First, the three features mentioned above are stitched together. Then, the stitched features can be input into the fourth convolution module for convolution to obtain the initial image features f. att Of course, the fusion module can also consist of only the stitching module. The above three features can be directly stitched together to obtain the initial image features f. att It should be noted that... Figure 5 The m1 and m3 shown are the spatial weights calculated by the spatial attention module. Multiplying these weights by the original features (i.e., f1 and f3) yields the spatial attention features.

[0096] The above feature extraction methods are merely examples and not limitations of this application. For instance, multiple images to be processed can each undergo feature extraction through convolutional layers. After obtaining multiple convolutional features, these convolutional features can be directly concatenated together as the initial image features.

[0097] By calculating the spatial attention relative to the reference image, it becomes easier to generate an HDR reconstructed image that is closer to the reference image.

[0098] According to an embodiment of this application, the reference image may be the image whose exposure value is closest to 0 among a plurality of images to be processed.

[0099] Ideally, when selecting LDR images, at least one image with a 0EV value should be included. Alternatively, a 0EV image can be directly set as the reference image. Even if the LDR image does not contain a 0EV image, the image closest to 0EV can still be selected as the reference image.

[0100] For example, suppose there are three images to be processed. In one example, the exposure values ​​of the three images are +2EV, 0EV, and -1EV, respectively. The image with 0EV can be selected as the reference image. In another example, the exposure values ​​are +3EV, +2EV, and +1EV, respectively. The image with +1EV can be selected as the reference image. In yet another example, the exposure values ​​are -1EV, -2EV, and -3EV, respectively. The image with -1EV can be selected as the reference image. In yet another example, the exposure values ​​are +3EV, +2EV, and -1EV, respectively. The image with -1EV can be selected as the reference image.

[0101] A 0EV image is a standard exposure image, which contains relatively balanced image information. The brightness of the reconstructed HDR image is usually close to that of the 0EV image. Therefore, using the image closest to 0EV as the reference image is beneficial for obtaining a more accurate HDR image through reconstruction.

[0102] According to an embodiment of this application, the reference image may be one of the images with the middle exposure value among a plurality of images to be processed.

[0103] When the number of images to be processed is odd, the image with the middle exposure value can be directly selected as the reference image from all the images to be processed. For example, suppose there are three images to be processed. In one example, the exposure values ​​of the three images to be processed are +2EV, 0EV, and -1EV, so the image with 0EV can be selected as the reference image. In another example, the exposure values ​​of the three images to be processed are +3EV, +2EV, and +1EV, so the image with +2EV can be selected as the reference image. In yet another example, the exposure values ​​of the three images to be processed are -1EV, -2EV, and -3EV, so the image with -2EV can be selected as the reference image. In yet another example, the exposure values ​​of the three images to be processed are +3EV, +2EV, and -1EV, so the image with +2EV can be selected as the reference image.

[0104] When the number of images to be processed is even, there are two images in the middle. One of these two images can be selected as the reference image. This selection can be achieved using any suitable method. For example, one can be randomly selected from the two images, or the image with the exposure value closest to 0 can be chosen as the reference image. For instance, suppose there are four images to be processed. In one example, the exposure values ​​of the four images are +3EV, +2EV, -1EV, and -3EV, respectively. The images in the middle are +2EV and -1EV, and the image closest to 0EV (i.e., the -1EV image) can be selected as the reference image.

[0105] According to an embodiment of this application, obtaining a reconstructed image based on the execution result of the last reconstruction operation includes: merging the execution result of the last reconstruction operation with the initial image features and / or the second convolutional features corresponding to the reference image to obtain intermediate output features; performing convolution on the intermediate output features to obtain third convolutional features; and mapping the third convolutional features through an activation function to obtain the reconstructed image.

[0106] For example, the reconstruction module may further include an output module connected after the last perception module, which processes the extracted image features into the reconstruction module to obtain a reconstructed image by: processing the extracted image features into at least one perception module; merging the features output by the last perception module with the initial image features and / or the convolutional features corresponding to the reference image to obtain intermediate output features; and processing the intermediate output features into the output module to obtain a reconstructed image.

[0107] The output module may include at least one convolutional layer and at least one activation function layer. Optionally, the activation function layer may be a sigmoid activation layer.

[0108] refer to Figure 5 At the output of the last perception module, the initial image features f are processed via a skip connection. att And the convolutional feature f2 is connected, so that the features output by the last perceptual module are similar to the initial image features f. att The convolutional features f2 are then combined to obtain intermediate output features. This combination can be achieved through element-wise summation.

[0109] Initial image features f att Furthermore, one or both of the convolutional features f2 can be merged with the features output by the last perceptual module. Alternatively, the features output by the last perceptual module can be output directly as intermediate features without merging with other features.

[0110] Combining the features output by the last perception module with the convolutional features corresponding to the initial image and / or the reference image is a residual processing method that optimizes the training of the reconstruction network model.

[0111] According to an embodiment of this application, obtaining a reconstructed image based on the execution result of the last reconstruction operation includes: merging the execution result of the last reconstruction operation with the initial image features to obtain intermediate output features; performing convolution on the intermediate output features to obtain fourth convolution features; and mapping the fourth convolution features through an activation function to obtain the reconstructed image.

[0112] For example, the reconstruction module further includes an output module connected after the last perception module. The process of inputting the extracted image features into the reconstruction module for processing to obtain a reconstructed image includes: inputting the extracted image features into at least one perception module for processing; merging the features output by the last perception module with the initial image features to obtain intermediate output features; and inputting the intermediate output features into the output module for processing to obtain a reconstructed image.

[0113] The example of merging the features output by the last perception module with the initial image features has already been described above, and will not be repeated here.

[0114] According to the embodiments of this application, the reconstruction operation further includes a convolution operation performed on the execution result of the last feature processing operation. In each reconstruction operation, the image features to be processed corresponding to the first feature processing operation are combined with the execution result of the convolution operation to obtain the execution result of the reconstruction operation.

[0115] For example, in each perception module, the input features of the first sub-perception module are combined with the output features of the second convolution module to form the output features of that perception module.

[0116] refer to Figure 5 This illustrates how the input features of the first CA-ViT module are skipped to the output of the second convolutional module, allowing the input features of the first CA-ViT module to be merged with the output features of the second convolutional module. This merging can be element-wise summation. Similar to the embodiment described above, which merges the features output by the last perception module with the initial image features and / or the second convolutional features corresponding to the reference image, this merging scheme is a residual processing method that optimizes the training of the reconstructed network model.

[0117] According to the embodiments of this application, the operations of feature extraction from multiple images to be processed, global context information extraction based on the extracted image features, and local context information extraction combined with global and local context features to obtain a reconstructed image are implemented through a reconstruction network model. Method 400 may further include: acquiring training images, which include multiple sample images and corresponding labeled images, wherein the sample images are LDR images and the labeled images are HDR images; processing the multiple sample images using an initial reconstruction network model to obtain a predicted reconstructed image; calculating a reconstruction loss term based on the predicted reconstructed image and the labeled image; acquiring the predicted image features and labeled image features corresponding to the predicted reconstructed image and the labeled image respectively; calculating a perceptual loss term based on the predicted image features and the labeled image features; calculating a total loss based on the reconstruction loss term and the perceptual loss term; and optimizing the parameters of the initial reconstruction network model based on the total loss to obtain a reconstruction network model.

[0118] The initial reconstruction network model is the reconstruction network model whose parameters are the initialization parameters. The method of using the initial reconstruction network model to process multiple sample images is the same as the method of using the reconstruction network model to process multiple images to be processed described above. Those skilled in the art can understand how the initial reconstruction network model is used during training and how the predicted reconstructed images are obtained, so this article will not elaborate on them.

[0119] Those skilled in the art will understand that the training process of a reconstruction network model may include: inputting a set of sample images into an initial reconstruction network model to obtain the corresponding predicted reconstructed images; then substituting the predicted reconstructed images and labeled images into a loss function to calculate the loss term; and optimizing the parameters of the initial reconstruction network model using backpropagation and gradient descent algorithms until the value of the loss term meets the requirements. After the current optimization is completed, the next set of sample images is input into the optimized reconstruction network model, and the above optimization process is repeated until the maximum number of iterations is reached or the model converges, ending the training process. The process of optimizing the parameters of the reconstruction network model based on the loss term can be implemented using conventional optimization methods, which will not be elaborated upon in this paper. This paper mainly describes the calculation method of the loss term involved in the training of the reconstruction network model.

[0120] Unlike existing loss function calculation methods that only use pixel-level loss, the loss function used in this application embodiment can include a reconstruction loss term and a perception loss term, as follows:

[0121]

[0122] In formula (1), Indicates the total loss. and These represent the reconstruction loss term and the perceptual loss term, respectively, and λ represents a preset scaling factor. The preset scaling factor λ can be any suitable value, such as 0.01. The reconstruction loss term is used to calculate the error between the predicted reconstructed image and the labeled image (ground truth label). Exemplarily, and not limitingly, this error can be such as the minimum absolute error (LEE). Error), that is:

[0123]

[0124] Among them, I GT Indicates an annotated image, I H Indicates the predicted reconstructed image. This represents the μ-law function. The expression for the μ-law function is as follows:

[0125]

[0126] The μ-law function represents tone mapping of the input image (represented by x in formula (3)), which means transforming the original input image to another brightness domain. For better display, the reconstructed HDR image typically undergoes... It is only displayed on the monitor after the transformation.

[0127] The perceptual loss term can be used to calculate the error in predicting the image features corresponding to the reconstructed image and the labeled image, respectively. Exemplarily, and not limitingly, this error can also be, for example, ... Error. The perceptual loss term is used to constrain the quality of the reconstructed HDR image at the feature level. Because of the inclusion of the perceptual loss term, the errors in the image features corresponding to the predicted and labeled images can also be incorporated into the optimization factors during the optimization of the reconstruction network model based on the loss term. This helps improve the quality of the HDR image output by the reconstruction network model.

[0128] For example, obtaining the predicted image features and labeled image features corresponding to the predicted reconstructed image and the labeled image respectively includes: performing tone mapping on the predicted reconstructed image and the labeled image respectively to obtain a new predicted reconstructed image and a new labeled image; and inputting the new predicted reconstructed image and the new labeled image into a pre-trained network model respectively to obtain their respective predicted image features and labeled image features.

[0129] The perceptual loss term can be defined as:

[0130]

[0131] Where, Φ i () represents the output feature of the i-th layer among the m layers in the pre-trained network model that are involved in training the reconstructed network model, i = 1, 2, ..., m.

[0132] The m layers in the pre-trained network model that participate in training the reconstructed network model can include any m network layers other than the input and output layers, i.e., any m hidden layers in the pre-trained network model. m is an integer greater than or equal to 1. The size of m can be set as needed, and this application does not impose any restrictions on it. Preferably, the m layers in the pre-trained network model that participate in training the reconstructed network model are the m layers closest to the output layer in the pre-trained network model (i.e., the last m hidden layers).

[0133] As mentioned above, to achieve better display, the reconstructed HDR image typically undergoes [processing / processing]. The image is displayed on the screen only after transformation. Therefore, when extracting features using a pre-trained network model, the predicted reconstructed image and the labeled image can be transformed separately. Features are extracted after transformation, rather than directly extracting features from the predicted reconstructed image and the labeled image. This is based on the transformation process... The HDR image reconstructed by the feature calculation of the transformed image, and then trained based on the loss term, can have a better display effect.

[0134] By way of example, and not limitation, the pre-trained network model can be a VGG16 model.

[0135] During training, the training images can first be cropped to a size of 128x128. For example, the Adam optimizer can be used to train the reconstructed network model. The parameters of the Adam optimizer can be set as follows: β1 = 0.9, β2 = 0.999, ∈ = 10^(-8), with a learning rate of 2e-4. The training code can be implemented using the PyTorch framework.

[0136] Compared with existing image reconstruction methods, the reconstruction network model based on the CA-ViT module according to the embodiments of this application can better remove ghosting and restore visually consistent details in overexposed and occluded areas, thus enabling the reconstruction of higher quality HDR images.

[0137] According to another aspect of this application, an image feature processing apparatus is provided. Figure 6 A schematic block diagram of an image feature processing apparatus 600 according to an embodiment of this application is shown.

[0138] like Figure 6 As shown, the image feature processing apparatus 600 according to an embodiment of this application includes an acquisition module 610, a first extraction module 620, a second extraction module 630, and a fusion module 640. Each module can respectively perform the combined functions described above. Figure 2 The image feature processing method described herein comprises various steps / functions. The following description focuses only on the main functions of each component of the image feature processing apparatus 600, omitting the details already described above.

[0139] The acquisition module 610 is used to acquire features of the image to be processed, which are obtained based on image features extracted from the image to be processed. The acquisition module 610 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0140] The first extraction module 620 is used to extract global context information from the features of the image to be processed, thereby obtaining global context features. The first extraction module 620 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0141] The second extraction module 630 is used to extract local contextual information from the features of the image to be processed, thereby obtaining local contextual features. The second extraction module 630 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0142] The fusion module 640 is used to fuse global context features and local context features together to obtain the execution result of the image feature processing device 600. The fusion module 640 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0143] According to another aspect of this application, an image reconstruction apparatus is provided. Figure 7 A schematic block diagram of an image reconstruction apparatus 700 according to one embodiment of this application is shown.

[0144] like Figure 7 As shown, the image reconstruction apparatus 700 according to an embodiment of this application includes an acquisition module 710, a first extraction module 720, and a second extraction module 730. Each module can respectively perform the functions described above in conjunction with... Figure 4 The image reconstruction method described herein comprises various steps / functions. The following description focuses only on the main functions of each component of the image reconstruction apparatus 700, omitting the details already described above.

[0145] The acquisition module 710 is used to acquire multiple images to be processed, which are low dynamic range images. The acquisition module 710 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0146] The first extraction module 720 is used to extract features from multiple images to be processed, thereby obtaining the extracted image features. The first extraction module 720 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0147] The second extraction module 730 is used to extract global and local context information based on the extracted image features, obtain global and local context feature information, and combine the global and local context feature information to obtain a reconstructed image, which is a high dynamic range image. The second extraction module 730 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0148] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] Figure 8 A schematic block diagram of an electronic device 800 according to an embodiment of this application is shown. The electronic device 800 includes a storage device (i.e., a memory) 810, a processor 820, and a computer program stored in the memory.

[0150] Storage device 810 stores computer program instructions for implementing corresponding steps in the image feature processing method according to embodiments of the present application.

[0151] The processor 820 is used to run computer program instructions stored in the storage device 810 to perform corresponding steps of the image feature processing method according to the embodiments of this application.

[0152] In one embodiment, computer program instructions are executed by processor 820 to perform the following steps: acquiring image features to be processed, the image features to be processed being obtained based on image features extracted from the image to be processed; extracting global context information from the image features to be processed to obtain global context features; extracting local context information from the image features to be processed to obtain local context features; and fusing the global context features and local context features together to obtain an execution result.

[0153] Figure 9 A schematic block diagram of an electronic device 900 according to an embodiment of this application is shown. The electronic device 900 includes a storage device (i.e., a memory) 910, a processor 920, and a computer program stored in the memory.

[0154] Storage device 910 stores computer program instructions for implementing corresponding steps in the image reconstruction method according to embodiments of the present application.

[0155] The processor 920 is used to run computer program instructions stored in the storage device 910 to perform corresponding steps of the image reconstruction method according to the embodiments of this application.

[0156] In one embodiment, computer program instructions are executed by processor 920 to perform the following steps: acquiring multiple images to be processed, the images to be processed being low dynamic range images; performing feature extraction on the multiple images to be processed to obtain extracted image features; performing global context information extraction and local context information extraction based on the extracted image features to obtain global context feature information and local context feature information, and combining the global context feature information and local context feature information to obtain a reconstructed image, the reconstructed image being a high dynamic range image.

[0157] Furthermore, according to embodiments of this application, a computer-readable storage medium is also provided, on which a computer program / instruction is stored. When the computer program / instruction is run by a computer or processor, it is used to execute corresponding steps of the image feature processing method of this application embodiment and to implement corresponding modules in the image feature processing apparatus according to embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0158] In one embodiment, a computer program / instruction is used at runtime to perform the following steps: acquiring image features to be processed, the image features to be processed being obtained based on image features extracted from the image to be processed; extracting global context information from the image features to be processed to obtain global context features; extracting local context information from the image features to be processed to obtain local context features; and fusing the global context features and local context features together to obtain an execution result.

[0159] Furthermore, according to embodiments of this application, a computer-readable storage medium is also provided, on which a computer program / instructions are stored. When the computer program / instructions are run by a computer or processor, they are used to execute corresponding steps of the image reconstruction method of this application embodiment and to implement corresponding modules in the image reconstruction apparatus according to embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0160] In one embodiment, a computer program / instruction is used to perform the following steps at runtime: acquiring multiple images to be processed, the images to be processed being low dynamic range images; performing feature extraction on the multiple images to be processed to obtain extracted image features; performing global context information extraction and local context information extraction based on the extracted image features to obtain global context feature information and local context feature information, and combining the global context feature information and local context feature information to obtain a reconstructed image, the reconstructed image being a high dynamic range image.

[0161] Furthermore, according to embodiments of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the above-described image feature processing method or the above-described image reconstruction method.

[0162] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0165] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0166] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the various applications, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0167] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0168] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0169] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the image feature processing apparatus or image reconstruction apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0170] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0171] The above are merely specific embodiments or descriptions of specific embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. An image reconstruction method, comprising: Multiple images to be processed are acquired. The images to be processed are low dynamic range images. The multiple images to be processed are multiple images continuously acquired from the same viewpoint for the same scene, and the brightness of the multiple images to be processed is different. Feature extraction is performed on the multiple images to be processed to obtain the extracted image features; Based on the extracted image features, global context information and local context information are extracted to obtain global context feature information and local context feature information. The reconstructed image is obtained by combining the global context feature information and the local context feature information. The reconstructed image is a high dynamic range image. The step of extracting global and local context information based on the extracted image features to obtain global and local context feature information, and then combining the global and local context feature information to obtain a reconstructed image, includes: At least one reconstruction operation is executed sequentially, wherein the reconstruction operation includes at least one feature processing operation executed sequentially, the feature processing operation including: The visual converter module in the context-aware visual converter module extracts global context information from the features of the image to be processed to obtain global context features. The image features to be processed corresponding to the first feature processing operation in the first reconstruction operation are the extracted image features. The image features to be processed corresponding to the first feature processing operation in non-first reconstruction operations are the execution result of the previous reconstruction operation. The image features to be processed corresponding to non-first feature processing operations in all reconstruction operations are the execution result of the previous feature processing operation. The global context feature information includes the global context features. The local context extraction module in the context-aware visual converter module is used to extract local context information from the features of the image to be processed to obtain local context features, and the local context feature information includes the local context features. The global context features and the local context features are fused together to obtain the execution result of the feature processing operation; A reconstructed image is obtained based on the result of the last reconstruction operation; the reconstructed image is a high dynamic range image.

2. The method as described in claim 1, wherein, The step of extracting local context information from the features of the image to be processed to obtain local context features includes: The features of the image to be processed are subjected to layer normalization to obtain normalized features; The normalized features are shaped to convert them into two-dimensional image features, thus obtaining shaped features; The shaped features are convolved to obtain the first convolutional features; The first convolutional features are subjected to channel attention processing to obtain channel attention features; The channel attention features are shaped to convert them into block-embedded feature forms to obtain the local context features.

3. The method as described in claim 1 or 2, wherein, The step of extracting features from the plurality of images to be processed to obtain extracted image features includes: Initial feature extraction is performed on the multiple images to be processed to obtain initial image features; The initial image features are shaped to convert them into block-embedded feature forms, thereby obtaining the extracted image features.

4. The method of claim 3, wherein, The initial feature extraction of the plurality of images to be processed to obtain initial image features includes: Convolution is performed on the multiple images to be processed to obtain multiple second convolution features that correspond one-to-one with the multiple images to be processed; For any non-reference image, spatial attention processing is performed on the second convolutional feature corresponding to the non-reference image and the second convolutional feature corresponding to the reference image to obtain the spatial attention feature corresponding to the non-reference image. The reference image is one of the plurality of images to be processed, and the non-reference image is an image other than the reference image among the plurality of images to be processed. The initial image features are obtained by fusing the spatial attention features corresponding to all non-reference images and the second convolution features corresponding to the reference image.

5. The method of claim 4, wherein, The reference image is the image whose exposure value is closest to 0 among the plurality of images to be processed, or the reference image is one of the images whose exposure value is in the middle among the plurality of images to be processed.

6. The method of claim 4, wherein, The process of obtaining the reconstructed image based on the result of the last reconstruction operation includes: The result of the last reconstruction operation is combined with the initial image features and / or the second convolutional features corresponding to the reference image to obtain intermediate output features; The intermediate output features are convolved to obtain the third convolutional features; The reconstructed image is obtained by mapping the third convolutional feature using an activation function.

7. The method of claim 3, wherein, The process of obtaining the reconstructed image based on the result of the last reconstruction operation includes: The result of the last reconstruction operation is combined with the initial image features to obtain intermediate output features; The intermediate output features are convolved to obtain the fourth convolutional features; The reconstructed image is obtained by mapping the fourth convolutional feature using an activation function.

8. The method as claimed in claim 1 or 2, wherein, The reconstruction operation also includes a convolution operation performed on the result of the last feature processing operation. In each reconstruction operation, the image features to be processed corresponding to the first feature processing operation are combined with the execution result of the convolution operation to obtain the execution result of the reconstruction operation.

9. The method as claimed in claim 1 or 2, wherein, The operations of feature extraction from the plurality of images to be processed, global context information extraction based on the extracted image features, and local context information extraction, combined with the global context information and the local context information to obtain the reconstructed image, are implemented through a reconstruction network model. The method further includes: Acquire training images, which include multiple sample images and labeled images corresponding to the multiple sample images, wherein the sample images are low dynamic range images and the labeled images are high dynamic range images; The initial reconstruction network model is used to process the multiple sample images to obtain the predicted reconstructed image; Calculate the reconstruction loss term based on the predicted reconstructed image and the labeled image; Obtain the predicted image features and labeled image features corresponding to the predicted reconstructed image and the labeled image, respectively; Calculate the perceptual loss term based on the predicted image features and the labeled image features; The total loss is calculated based on the reconstruction loss term and the perception loss term; The parameters of the initial reconstructed network model are optimized based on the total loss to obtain the reconstructed network model.

10. The method of claim 9, wherein, The step of obtaining the predicted image features and the labeled image features corresponding to the predicted reconstructed image and the labeled image respectively includes: Tone mapping is performed on the predicted reconstructed image and the labeled image respectively to obtain a new predicted reconstructed image and a new labeled image; The new predicted reconstructed image and the new labeled image are respectively input into the pre-trained network model to obtain the corresponding predicted image features and labeled image features.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the image reconstruction method as described in any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image reconstruction method as described in any one of claims 1-10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image reconstruction method as described in any one of claims 1-10.

Images