Image reconstruction method, apparatus, device, medium, and product

Abstract

The present application discloses an image reconstruction method, an apparatus, a device, a medium, and a product. According to the image reconstruction method in the embodiments of the present application, all Swin Transformer modules of an encoding module are connected by means of a patch merging layer; all Swin Transformer modules of a decoding module are connected by means of a patch expanding layer; the encoding module and the decoding module are symmetrical in structure; and finally, an upsampled feature map is input into a reconstruction module so as to obtain an RGB image by means of conversion. By constructing a network on the basis of a U-Net structure, the present solution is able to restore a corresponding high-resolution RGB image from a feature map, and interpolate missing values and defective pixels in a RAW image captured by an event camera while performing demosaicing, thereby obtaining a high-quality RGB image. Moreover, the use of Swin Transformer architecture as a feature extractor allows for the effective capturing of long-range dependencies, and ensures that consistency of local and global information is maintained.

Description

Image reconstruction method, device, equipment, medium and product TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image reconstruction method, device, equipment, medium and product. BACKGROUND

[0002] The original RAW image is usually sampled using a Bayer arrangement, that is, three color channels of red, green and blue are sampled in different arrangements to form a mosaic pattern. When the RAW image is reconstructed into a color RGB image, demosaicing is needed. The demosaicing process is to calculate the red, green and blue values of each pixel point according to the Bayer arrangement and the information of adjacent pixels, so as to generate a full-color image. After a general camera captures a RAW image, only demosaicing needs to be considered in the process of converting the RAW image into an RGB image.

[0003] However, due to the limitations of the design of the event camera sensor, when the event camera captures a RAW image, some values in the original domain of the red and blue information will be missing, and some random bad points will also appear. The traditional scheme cannot fill in the missing values and bad points, resulting in a low-quality RGB image. SUMMARY

[0004] The embodiments of the application provide an image reconstruction method, device, equipment, medium and product, which can fill in the missing values and bad points when a RAW image is reconstructed into an RGB image, and generate a high-quality RGB image.

[0005] In one aspect, the embodiments of the application provide an image reconstruction method, comprising:

[0006] inputting a RAW image captured by an event camera into a target image reconstruction model; the target image reconstruction model comprises, connected in sequence: an encoding module, a decoding module and a reconstruction module;

[0007] inputting the RAW image into the encoding module to obtain a down-sampling feature map; the encoding module comprises a plurality of Swin Transformer modules connected in series, and the Swin Transformer modules of the encoding module are connected through tile merging layers;

[0008] inputting the down-sampling feature map into the decoding module to obtain an up-sampling feature map; wherein the decoding module comprises a corresponding number of Swin Transformer modules as the encoding module, and the Swin Transformer modules of the decoding module are connected through patch expansion layers; the structure of the encoding module and the decoding module is symmetrical;

[0009] inputting the up-sampling feature map into the reconstruction module to reconstruct an RGB image.

[0010] In another aspect, the encoding module and the decoding module each include a normalization layer, a window-based multi-head self-attention module, and a multi-layer perceptron layer.

[0011] In another aspect, after the up-sampling feature map is inputted into the reconstruction module to reconstruct an RGB image, the method further includes:

[0012] obtaining a quality indicator of the RGB image, the quality indicator including a peak signal-to-noise ratio and / or a structural similarity index;

[0013] In a case where the quality indicator does not satisfy a first preset condition, adjusting the target image reconstruction model based on a standard quality indicator.

[0014] In another aspect, before the RAW image captured by the event camera is inputted into the target image reconstruction model, the method further includes:

[0015] constructing an initial image reconstruction model;

[0016] inputting a RAW image training set with missing pixel values into the initial image reconstruction model for image reconstruction to obtain a predicted RGB image;

[0017] training the initial image reconstruction model based on the predicted RGB image and an actual RGB image corresponding to the RAW image training set to obtain the target image reconstruction model.

[0018] In another aspect, before the RAW image training set with missing pixel values is inputted into the initial image reconstruction model for image reconstruction, the method further includes:

[0019] performing data enhancement processing on the RAW image training set; the data enhancement processing includes any one or any combination of the following: random cropping, rotation, and flipping;

[0020] The inputting of the RAW image training set with missing pixel values into the initial image reconstruction model for image reconstruction includes:

[0021] inputting the RAW image training set after the data enhancement processing into the initial image reconstruction model for image reconstruction.

[0022] In another aspect, the training of the initial image reconstruction model based on the predicted RGB image and the actual RGB image corresponding to the RAW image training set to obtain the target image reconstruction model includes:

[0023] determine a function value of a loss function based on the predicted RGB image and the actual RGB image;

[0024] In a case where the function value does not satisfy a second preset condition, adjust a parameter of a model, and return to the step of inputting the RAW image training set with missing pixel values into the initial image reconstruction model for image reconstruction to obtain a predicted RGB image.

[0025] In another aspect, the loss function is specifically:

[0026] In a segment of 0 < d < a:

[0027]

[0028] In a segment of a ≤ d ≤ 1:

[0029]

[0030] d is a difference value between the predicted RGB image and the actual RGB image, a is a threshold parameter, and b and g are scaling factors for controlling the gradient amplitude.

[0031] In another aspect, the loss function is specifically:

[0032] In a segment of 0 < d < a:

[0033]

[0034] In a segment of a ≤ d ≤ 1:

[0035]

[0036] d is a difference value between the predicted RGB image and the actual RGB image, a is a threshold parameter, b and g are scaling factors for controlling the gradient amplitude, and λ is a hyperparameter.

[0037] In still another aspect, an embodiment of the present application provides an image reconstruction device, which comprises a processor and a memory storing computer program instructions.

[0038] The processor implements the image reconstruction method as described above when executing the computer program instructions.

[0039] In still another aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the image reconstruction method as described above.

[0040] In still another aspect, the embodiments of the present application provide a computer program product, instructions in the computer program product are executed by a processor of an electronic device to cause the electronic device to perform the image reconstruction method as described above.

[0041] An image reconstruction method of the embodiments of the present application, after an event camera shoots a RAW image, inputs a target image reconstruction model to reconstruct the RAW image into an RGB image. The target image reconstruction model includes, in sequence, an encoding module, a decoding module and a reconstruction module. The RAW image is input into the encoding module to obtain a down-sampling feature map; the encoding module includes a plurality of serially connected Swin Transformer modules, and the Swin Transformer modules of the encoding module are connected through tile merging layers. Then the down-sampling feature map is input into the decoding module to obtain an up-sampling feature map; finally, the up-sampling feature map is input into the reconstruction module to convert to obtain the RGB image. The decoding module includes a corresponding number of Swin Transformer modules as the encoding module, and the Swin Transformer modules of the decoding module are connected through patch expansion layers; the structure of the encoding module and the decoding module is symmetrical, and the Swin Transformer modules of the encoding module and the decoding module are jump-connected. The scheme adopts a network structure combining Swin Transformer and U-net, constructs the network based on the U-Net structure, can restore the corresponding high-resolution RGB image from the feature map, and at the same time of demosaicking, can fill in the missing values and bad points in the RAW image shot by the event camera, so as to obtain a high-quality RGB image. And using Swin Transformer as a feature extractor can effectively capture long-distance dependency and keep the consistency of local and global information. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. For those skilled in the art, without creative labor, other drawings can also be obtained according to these drawings.

[0043] FIG. 1 shows a flowchart of an image reconstruction method provided by an embodiment of the present application;

[0044] FIG. 2 shows a structural diagram of a target image reconstruction model provided by an embodiment of the present application;

[0045] FIG. 3 shows a hardware structure diagram of an image reconstruction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. For the purpose of clarity, the description is divided into the following sections: technical field, brief description of the drawings, detailed description of the embodiments, and conclusion. It should be understood that the specific embodiments described herein are intended to be illustrative only and not limiting of the present application. The present application can be implemented in ways other than those specifically described herein without departing from the spirit of the present application. The following description is merely provided to give a better understanding to the present application.

[0047] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the listed element.

[0048] Based on the event camera imaging research, most of them focus on enhancing the RGB domain image using event information, ignoring the process of RAW domain image to RGB domain image. Specifically, the camera generates RAW domain images by capturing light, each pixel independently recording a single red, green or blue signal. Then, these RAW domain images need to be converted to RGB domain images through processing algorithms.

[0049] However, due to the limitations of event camera sensor design, some values in the partial red and blue information in the RAW domain will be missing, and some random bad pixels will also appear. Traditional demosaicing methods can be divided into two categories: one is model-based method, and the other is machine learning-based method. The traditional algorithm cannot solve the problem of missing pixel values.

[0050] In order to complete the missing color information, new demosaicing technology needs to be designed for event cameras to convert incomplete single-channel RAW images into complete RGB images. Specifically, the embodiments of the present application provide an image reconstruction method, device, equipment, medium and product. First, the image reconstruction method provided by the embodiments of the present application will be introduced. Fig. 1 shows a flowchart of the image reconstruction method provided by an embodiment of the present application; as shown in Fig. 1, the method comprises the following steps:

[0051] S101: input a RAW image captured by an event camera into a target image reconstruction model.

[0052] The target image reconstruction model is pre-trained according to a RAW image training set, and is used to reconstruct a RAW image into an RGB image. The target image reconstruction model comprises, in sequence, an encoding module, a decoding module, and a reconstruction module.

[0053] Generally, a preprocessing module can be further included before the encoding module, which is used to convert the format of the input RAW image into a data format suitable for subsequent encoding and decoding stages. The preprocessing module aims to convert the data format of the input RAW image into a data format suitable for subsequent encoding and decoding stages, while reducing the computational complexity.

[0054] FIG. 2 shows a structural schematic diagram of a target image reconstruction model according to an embodiment of the present application. As shown in FIG. 2, the target image reconstruction model comprises, in sequence, a preprocessing module, an encoding module, a decoding module, and a reconstruction module. The preprocessing module first performs a space-to-depth operation on the input RAW image I RAW ∈R H×W×1 , H is the height, W is the width, and an S2D operation is first applied. Let s be the factor, the spatial resolution is reduced to (H / s)×(W / s), and the channel dimension is increased to s 2 . This operation can effectively reduce the model complexity, because the computational cost is linearly related to the number of channels and quadratically related to the spatial resolution. Subsequently, a 1×1 convolution can be used to project the high-dimensional tensor to a low-dimensional space with C channels, denoted as F0∈R (H / s)×(W / s)×C, R represents the dimension. In order to incorporate the position information, a learnable position information E pos ∈R (H / s)×(W / s)×C is also added:

[0055] E pos (2i)=sin(p / 10000 2i / C );

[0056] E pos (2i+1)=cos(p / 10000 2i / C );

[0057] where p represents the position and i represents the dimension, and the preprocessed feature F0′=F0+E pos is obtained.

[0058] S102: input the RAW image into the encoding module to obtain a down-sampled feature map.

[0059] The encoding module includes a plurality of Swin Transformer modules (encoder blocks) connected in series, and the Swin Transformer modules of the encoding module are connected through tile merging layers. After the RAW image is input into the encoding module, the down-sampling feature map is obtained through the Swin Transformer modules and the tile merging layers connected alternately inside the encoding module. The tile merging layer reduces the resolution of the image, and the Swin Transformer module realizes feature conversion.

[0060] The encoding module is designed to extract multi-scale features and capture long-range dependencies, and the application adopts a U-Net-like architecture with Swin Transformer as the core. The encoding module includes N stages, each stage containing a Swin Transformer block followed by a tile merging layer. The Swin Transformer block includes a normalization (LN) layer, a window-based multi-head self-attention (W-MSA) module, and a multi-layer perceptron (MLP) layer.

[0061] The W-MSA module allows the model to capture multi-scale features and structural details at different granularities. The Swin Transformer modules of the encoding module use the Shifted Window mechanism alternately to facilitate cross-window interaction and enhance the representation ability of the model. After the Swin Transformer block, a tile merging layer is applied to generate a down-sampling feature map and increase the channel dimension. This multi-scale architecture enables the model to process information at multiple scales while gradually reducing the spatial resolution. Each down-sampling operation reduces the computational complexity by a factor (which can be 4 times) and doubles the receptive field, enabling the model to capture global information more effectively.

[0062] S103: input the down-sampling feature map into the decoding module to obtain an up-sampling feature map.

[0063] The decoding module includes a corresponding number of Swin Transformer modules (decoder blocks) corresponding to the encoding module, and the Swin Transformer modules of the decoding module are connected through patch expansion layers. The structure of the encoding module and the decoding module is symmetrical, that is, the resolution is first reduced through the tile merging layer of the encoding module, and then the resolution is increased through the patch expansion layer of the decoding module, and the up-sampling feature map is obtained through the symmetrical structure.

[0064] In particular, the decoding module aims to gradually generate the up-sampled feature maps and recover the spatial resolution of the output image. It follows a similar symmetric structure as the encoder module, consisting of N stages, each decoding stage containing a Swin Transformer module followed by an expansion patch layer.

[0065] The Swin Transformer module of the decoding module also includes an LN layer, a W-MSA module and an MLP layer. Here the W-MSA module operates in a similar way to the encoder block, capturing local dependencies within the window. The Swin Transformer modules of the decoding module are followed by a patch expansion layer to increase the spatial resolution and reduce the channel dimension. In addition, a skip connection is introduced between the corresponding encoder and decoder stages to facilitate information flow and help recover details. The multi-scale architecture of the decoding module enables the model to gradually improve the reconstructed image while combining features of different scales, thereby improving the demosaicing performance.

[0066] S104: input the up-sampled feature map into the reconstruction module to reconstruct the RGB image.

[0067] The up-sampled feature map is input into the reconstruction module, and the goal of the reconstruction module is to generate and output the final RGB image based on the up-sampled feature map.

[0068] To achieve this goal, a depth-to-space operation can be applied first, which precisely reverses the space-to-depth operation applied in the preprocessing module described above. This step is crucial for maintaining image quality and reducing distortion introduced in the preprocessing process. The depth-to-space operation rearranges the features and achieves an increase in spatial resolution by s times, restoring the original spatial dimensions of the input image. Finally, a 1x1 convolution can be used to map the high-dimensional features to the desired output space, generating an RGB image with a shape of HxWx3.

[0069] To solve the demosaicing problem in the RAW domain signal generated based on the event camera, the present application designs a neural network architecture combining the advantages of Swin Transformer and U-Net. First, data preprocessing is performed to obtain the original data from the event camera, resulting in a RAW format image, and batch packaging of RAW format data is performed to unify the picture data format and length, preparing for subsequent training.

[0070] The network design specifically uses Swin Transformer as a feature extractor, which can effectively capture long-range dependencies and maintain the consistency of local and global information. Based on the U-Net structure, the network can restore the corresponding high-resolution RGB image from the feature map.

[0071] During the training process, the RAW image with missing pixel values is used as input, and the actual RGB image is used as target output. By using the difference between the reconstructed RGB image and the actual RGB image, a loss function is designed to train the network to improve the network performance. In the training process, appropriate data augmentation techniques such as random cropping, rotation, and flipping can be used to enhance the generalization ability of the network.

[0072] Finally, the module can also be tested and evaluated, and the trained model can be used to perform demosaicing on new RAW images, and the results can be compared with the actual RGB images. Here, standard image quality evaluation indicators such as peak signal-to-noise ratio and structural similarity index can be used to evaluate the quality of the generated RGB images.

[0073] The present application solves the demosaicing problem for RAW domain signals generated by event cameras, and uses the advantages of Swin Transformer module and U-Net architecture to reconstruct high-quality RGB images from RAW images with missing pixel values. In addition, a special loss function is designed for areas with a large number of high-frequency details, edges, and textures to improve the network performance.

[0074] The image reconstruction method of the embodiment of the application is used to input a target image reconstruction model after an event camera shoots a RAW image, so as to reconstruct the RAW image into an RGB image. The target image reconstruction model comprises, in sequence, an encoding module, a decoding module and a reconstruction module. The RAW image is input into the encoding module to obtain a down-sampling feature map; the encoding module comprises a plurality of serially connected Swin Transformer modules, and the Swin Transformer modules of the encoding module are connected through tile merging layers. Then, the down-sampling feature map is input into the decoding module to obtain an up-sampling feature map; finally, the up-sampling feature map is input into the reconstruction module to convert to obtain the RGB image. The decoding module comprises a corresponding number of Swin Transformer modules corresponding to the encoding module, and the Swin Transformer modules of the decoding module are connected through patch expansion layers; the structure of the encoding module is symmetrical to that of the decoding module, and the Swin Transformer modules of the encoding module are jump-connected to the Swin Transformer modules of the decoding module. The network structure of the present scheme combines Swin Transformer and U-net, constructs the network based on the U-Net structure, can restore the corresponding high-resolution RGB image from the feature map, and can fill in the missing values and bad points in the RAW image shot by the event camera while removing the mosaics, so as to obtain a high-quality RGB image. Moreover, the Swin Transformer is used as a feature extractor, which can effectively capture long-distance dependency and keep the consistency of local and global information.

[0075] In the above embodiment, it is mentioned that the Swin Transformer modules of the encoding module and the decoding module can each comprise an LN layer, a W-MSA module and an MLP layer. The LN layer normalizes all feature dimensions of each sample, and is not affected by the sample batch size. The W-MSA module allows the model to capture multi-scale features and structural details of different granularities. The MLP layer is a kind of feedforward neural network, which is composed of a plurality of neurons and establishes a mapping relationship between the input and the output by learning appropriate weights and biases.

[0076] Because the performance of the target image reconstruction model obtained by training may not meet the requirements, the quality of the image can be evaluated after the RGB image is generated.

[0077] Specifically, after the up-sampling feature map is input into the reconstruction module to reconstruct the RGB image, the method further comprises: obtaining a quality index of the RGB image, and adjusting the target image reconstruction model based on a standard quality index in a case where the quality index does not meet a first preset condition. The quality index comprises a peak signal-to-noise ratio and / or a structural similarity index. The specific content of the standard quality index is not limited and is subject to actual requirements.

[0078] The embodiment of the present application evaluates the quality of the generated RGB image, adjusts the target image reconstruction model when the quality is poor, and thus improves the quality of the generated RGB image.

[0079] The embodiment of the present application provides a construction and training process of a target image reconstruction model. Before inputting a RAW image captured by an event camera into the target image reconstruction model, an initial image reconstruction model is constructed, and then a RAW image training set with missing pixel values is input into the initial image reconstruction model for image reconstruction to obtain a predicted RGB image. Based on the predicted RGB image and an actual RGB image corresponding to the RAW image training set, the initial image reconstruction model is trained to obtain the target image reconstruction model.

[0080] The embodiment of the present application trains the model based on the predicted RGB image and the actual RGB image, and improves the performance of the model.

[0081] In actual application, when the model is trained by the original RAW image training set, the effect may not be satisfactory. In order to improve the performance of the model, the RAW image training set can be subjected to data enhancement processing before being input into the initial image reconstruction model for image reconstruction. The data enhancement processing includes any one or any combination of the following: random cropping, rotation and flipping. The RAW image training set with missing pixel values is input into the initial image reconstruction model for image reconstruction, including: inputting the RAW image training set subjected to data enhancement processing into the initial image reconstruction model for image reconstruction.

[0082] Through the data enhancement processing scheme provided by the embodiment of the present application, the generalization ability of the model can be enhanced.

[0083] The above training process is not specifically limited, and in actual cases, the training can be completed by designing a loss function. Specifically, based on the predicted RGB image and the actual RGB image corresponding to the RAW image training set, the initial image reconstruction model is trained to obtain the target image reconstruction model, including: determining the function value of the loss function based on the predicted RGB image and the actual RGB image; in the case where the function value does not satisfy a second preset condition, adjusting the parameters of the model, and returning to the step of inputting the RAW image training set with missing pixel values into the initial image reconstruction model for image reconstruction to obtain the predicted RGB image.

[0084] Through the loss function designed by the present application, the image reconstruction model is continuously trained, the model is efficiently optimized, and finally the target image reconstruction model meeting the requirements is obtained.

[0085] The embodiment of the present application designs a special loss function. In the demosaicing task, the area with a large number of high-frequency details, edges and textures has greater importance and deserves more attention than the area that is easy to recover in the training process. Therefore, the scheme designs and uses a pixel attention loss function in the network training process, thereby improving the performance of the network.

[0086] Two loss functions are provided here to better capture edge-related differences.

[0087] One of the loss functions is:

[0088] In the 0 < d < a segment:

[0089]

[0090] In the a <= d <= 1 segment:

[0091]

[0092] d is the difference value between the predicted RGB image and the actual RGB image, a is the threshold parameter, b and g are scaling factors that control the gradient amplitude.

[0093] The other loss function is:

[0094] In the 0 < d < a segment:

[0095]

[0096] In the a <= d <= 1 segment:

[0097]

[0098] d is the difference value between the predicted RGB image and the actual RGB image, a is the threshold parameter, b and g are scaling factors that control the gradient amplitude, and lambda is a hyperparameter.

[0099] Both versions can enhance the model's ability to capture edge details by emphasizing the gradient of edge-related differences to enhance the demosaicing performance.

[0100] FIG. 3 shows a hardware structure diagram of an image reconstruction device according to an embodiment of the present application. As shown in FIG. 3, the image reconstruction device can include a processor 301 and a memory 302 storing computer program instructions.

[0101] Specifically, the processor 301 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0102] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 302 can include removable or non-removable (or fixed) media, where appropriate. The memory 302 can be internal or external to the integrated gateway disaster recovery appliance. In particular embodiments, the memory 302 is non-volatile, solid-state memory.

[0103] The memory 302 can include read-only memory (ROM), random-access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to

[0104] The processor 301 implements any one of the image reconstruction methods in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0105] In one example, the image reconstruction device can further include a communication interface 303 and a bus 310. The processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication with each other.

[0106] The communication interface 303 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0107] Bus 310 includes hardware, software, or both, to couple components of the image reconstruction device to each other and to couple components to other components, such as one or more other image reconstruction devices or peripheral devices. While bus 310 is shown for the sake of clarity as a single bus, bus 310 can include one or more buses operating together. Bus 310 can be implemented using any suitable type of bus or buses, including, but not limited to, an Accelerated Graphics Port (AGP) bus or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infmiband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or any other suitable bus or interconnect, or a combination of two or more of these. Where appropriate, bus 310 can include one or more buses operating together. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0108] In addition, in combination with the image reconstruction method in the above-mentioned embodiments, the embodiments of the present application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any of the image reconstruction methods in the above-mentioned embodiments.

[0109] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement any of the image reconstruction methods in the above-mentioned embodiments.

[0110] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above-mentioned embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0111] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0112] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0113] The above flowcharts and / or block diagrams of image reconstruction methods, apparatuses, devices, media, and products according to embodiments of the present disclosure have described various aspects of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0114] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method of image reconstruction, characterized by, The method comprises: inputting an original image captured by an event camera into a target image reconstruction model; the target image reconstruction model comprises, connected in sequence: an encoding module, a decoding module, and a reconstruction module; inputting the original image into the encoding module to obtain a down-sampling feature map; the encoding module comprises a plurality of serially connected Swin Transformer modules, and the Swin Transformer modules of the encoding module are connected through tile merging layers; inputting the down-sampling feature map into the decoding module to obtain an up-sampling feature map; wherein the decoding module comprises a corresponding number of Swin Transformer modules as the encoding module, and the Swin Transformer modules of the decoding module are connected through patch expansion layers; the structure of the encoding module and the decoding module is symmetrical, and the Swin Transformer modules of the encoding module and the decoding module are jump-connected; inputting the up-sampling feature map into the reconstruction module to reconstruct a color image.

2. The image reconstruction method of claim 1, wherein, The Swin Transformer modules of the encoding module and the decoding module each comprise a normalization layer, a window-based multi-head self-attention module, and a multi-layer perceptron layer.

3. The image reconstruction method of claim 1, wherein, After the step of inputting the up-sampling feature map into the reconstruction module to reconstruct a color image, the method further comprises: obtaining a quality indicator of the color image, the quality indicator comprising: a peak signal-to-noise ratio and / or a structural similarity index; in a case where the quality indicator does not satisfy a first preset condition, adjusting the target image reconstruction model based on a standard quality indicator.

4. The image reconstruction method of any one of claims 1 to 3, characterized in that, Before the step of inputting an original image captured by an event camera into a target image reconstruction model, the method further comprises: constructing an initial image reconstruction model; inputting an original image training set with missing pixel values into the initial image reconstruction model for image reconstruction to obtain a predicted color image; training the initial image reconstruction model based on the predicted color image and an actual color image corresponding to the original image training set to obtain the target image reconstruction model.

5. The image reconstruction method of claim 4, wherein, Before the step of inputting an original image training set with missing pixel values into the initial image reconstruction model for image reconstruction, the method further comprises: performing data enhancement processing on the original image training set; the data enhancement processing comprises any one or any combination of the following: random cropping, rotation, and flipping; the step of inputting an original image training set with missing pixel values into the initial image reconstruction model for image reconstruction comprises: inputting the original image training set after the data enhancement processing into the initial image reconstruction model for image reconstruction.

6. The image reconstruction method of claim 4, wherein, the step of training the initial image reconstruction model based on the predicted color image and an actual color image corresponding to the original image training set to obtain the target image reconstruction model comprises: determining a function value of a loss function based on the predicted color image and the actual color image. In a case where the function value does not satisfy a second preset condition, parameters of the model are adjusted, and the step of inputting the original image training set with the missing pixel value into the initial image reconstruction model for image reconstruction to obtain the predicted color image is returned.

7. The image reconstruction method of claim 6, wherein, The loss function is specifically: In the segment 0 < d < a: In the segment a < d < 1: d is a difference value between the predicted color image and the actual color image, a is a threshold parameter, b and g are scaling factors for controlling the gradient amplitude.

8. The image reconstruction method of claim 6, wherein, The loss function is specifically: In the segment 0 < d < a: In the segment a < d < 1 : d is a difference value between the predicted color image and the actual color image, a is a threshold parameter, b and g are scaling factors for controlling the gradient amplitude, and λ is a hyperparameter.

9. An image reconstruction device, characterized by The device comprises a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the image reconstruction method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the image reconstruction method of any one of claims 1 to 8.

11. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device executes the image reconstruction method of any one of claims 1 to 8.

Images