Blurred image super-division reconstruction method and device based on event stream, equipment and storage medium

Abstract

The invention discloses a blurred image super-division reconstruction method, device and equipment based on an event stream, and a storage medium, relates to the technical field of image processing, and discloses a blurred image super-division reconstruction method based on the event stream, and the method comprises the steps: obtaining a first blurred image, a second blurred image and a corresponding event stream, the two frames of event streams are respectively converted into event double-integral features; generating a first blurred image and a second reconstructed image according to the first blurred image, the second blurred image and the corresponding event double-integral features; performing up-sampling on the two frames of reconstructed images to obtain a first blurred image and a second clear image, and adaptively selecting a target clear image; and constructing a target loss function based on the input image, the event stream and the images in each stage, and optimizing the network parameters of the target clear image according to the target loss function to obtain a super-resolution image. And high-quality super-resolution reconstruction of a blurred image in a real dynamic scene is realized through a double-branch reconstruction network of event double-integral features and introduction of a self-supervised optimization loss function.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a blurred image super-resolution reconstruction method and device based on event stream, equipment and storage medium. BACKGROUND

[0002] In the field of intelligent vision and image processing today, the super-resolution reconstruction technology of blurred images is attracting more and more attention. Especially in practical applications such as automatic driving, intelligent monitoring, and unmanned aerial reconnaissance, image blurring is extremely easy to occur during the movement of the shooting device, and most images have low resolution, resulting in a significant decrease in the quality of visual information.

[0003] The widely used method in traditional technology is the image reconstruction method based on supervised learning, which mainly constructs a pair of reconstructed blurred images and their corresponding high-resolution clear images, and uses these paired data to train a deep neural network, so that the model can learn the restoration features of clear images. After training, it is applied to the reconstruction task of real images. However, since these training data are often generated by simulation rather than real shooting, there are obvious differences in texture details, noise distribution, and motion patterns between synthetic images and real images, so the performance of traditional methods in real scenes often decreases significantly.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a blurred image super-resolution reconstruction method and device based on event stream, equipment and storage medium, aiming to solve the technical problems of insufficient generalization ability and poor reconstruction effect of image super-resolution reconstruction in real complex scenes.

[0006] To achieve the above purpose, the present application provides a blurred image super-resolution reconstruction method based on event stream, which comprises:

[0007] Obtain a first blurred image, a first event stream, a second blurred image, and a second event stream, and convert the first event stream and the second event stream into first event double integral features and second event double integral features respectively;

[0008] Generate a first reconstructed image according to the first blurred image and the first event double integral features, and generate a second reconstructed image according to the second blurred image and the second event double integral features;

[0009] Upsample the first reconstructed image and the second reconstructed image respectively to obtain a first clear image and a second clear image, and adaptively select a target clear image according to the first clear image and the second clear image;

[0010] construct a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image;

[0011] optimize network parameters of the target clear image according to the loss function, to obtain a super-resolution image.

[0012] In an embodiment, the step of constructing a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image comprises:

[0013] obtain a preset image sample set and a downsampling operator, and construct a super-resolution consistency loss function according to the preset image sample set and the downsampling operator;

[0014] construct a first resolution latent clear image consistency loss function according to the first reconstructed image and the second reconstructed image, wherein the first resolution latent clear image consistency loss function is used to constrain the consistency of brightness and texture of an initial clear image;

[0015] construct a remodified blurred consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image and the second blurred image;

[0016] construct a second resolution latent clear image consistency loss function according to the first clear image and the second clear image, wherein the second resolution latent clear image consistency loss function is used to constrain the consistency of brightness and texture of an optimized clear image;

[0017] construct a first blurred clear consistency loss function and a second blurred clear consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream and the downsampling operator;

[0018] construct a target loss function according to the super-resolution consistency loss function, the first resolution latent clear image consistency loss function, the remodified blurred consistency loss function, the first blurred clear consistency loss function, the second resolution latent clear image consistency loss function and the second blurred clear consistency loss function.

[0019] In an embodiment, the step of constructing a reblur consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image, and the second blurred image comprises:

[0020] determining a first reblurred image according to mapping of the first reconstructed image and the first event stream;

[0021] determining a second reblurred image according to mapping of the second reconstructed image and the second event stream;

[0022] constructing a reblur consistency loss function according to the first reblurred image, the first blurred image, the second reblurred image, and the second blurred image.

[0023] In an embodiment, the step of constructing a first blur-sharp consistency loss function and a second blur-sharp consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream, and the down-sampling operator comprises:

[0024] obtaining a mask matrix;

[0025] constructing a first blur-sharp consistency loss function according to the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream;

[0026] constructing a second blur-sharp consistency loss function according to the down-sampling operator, the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream.

[0027] In an embodiment, the step of adaptively selecting a target sharp image according to the first sharp image and the second sharp image comprises:

[0028] obtaining a current time, a first exposure period, and a second exposure period;

[0029] determining a first adaptive value according to the current time and the first exposure period;

[0030] determining a second adaptive value according to the current time and the second exposure period;

[0031] when the first adaptive value is less than the second adaptive value, selecting the first sharp image as the target sharp image;

[0032] When the first adaptive value is greater than or equal to the second adaptive value, the second clear image is selected as a target clear image.

[0033] In an embodiment, the step of generating a first reconstructed image according to the first blurred image and the first event bilinear feature and generating a second reconstructed image according to the second blurred image and the second event bilinear feature comprises:

[0034] Down-sampling according to the first blurred image and the second blurred image to obtain a first down-sampled image and a second down-sampled image;

[0035] Convolution according to the first down-sampled image and the second down-sampled image to obtain a first image feature map and a second image feature map;

[0036] Dense residual connection according to the first image feature map and the second image feature map to obtain a first fusion feature map and a second fusion feature map;

[0037] Convolution and up-sampling according to the first fusion feature map and the second fusion feature map to obtain a first reconstructed image and a second reconstructed image.

[0038] In an embodiment, the step of acquiring a first blurred image, a first event stream, a second blurred image and a second event stream, and converting the first event stream and the second event stream into a first event bilinear feature and a second event bilinear feature respectively comprises:

[0039] Acquiring a first blurred image and a second blurred image collected by an event camera in a continuous acquisition period;

[0040] Acquiring a corresponding first event stream and a second event stream according to the first blurred image and the second blurred image respectively;

[0041] Event stream preprocessing is performed on the first event stream and the second event stream to convert the first event stream and the second event stream into a first event bilinear feature and a second event bilinear feature respectively, and the event stream preprocessing comprises convolution processing and activation function processing.

[0042] In addition, to achieve the above-mentioned purposes, the application further provides a blurred image super-resolution reconstruction device based on an event stream, which comprises:

[0043] A data acquisition module is configured to acquire a first blurred image, a first event stream, a second blurred image and a second event stream, and convert the first event stream and the second event stream into a first event bilinear feature and a second event bilinear feature respectively;

[0044] an image reconstruction module, configured to generate a first reconstructed image according to the first blurred image and the first event double integral feature, and generate a second reconstructed image according to the second blurred image and the second event double integral feature;

[0045] an image up-sampling module, configured to up-sample the first reconstructed image and the second reconstructed image respectively to obtain a first clear image and a second clear image, and adaptively select a target clear image according to the first clear image and the second clear image;

[0046] a loss function construction module, configured to construct a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image;

[0047] an image optimization module, configured to optimize network parameters of the target clear image according to the loss function to obtain a super-resolution image.

[0048] In addition, to achieve the above object, the present application further provides a blurred image super-resolution reconstruction device based on an event stream, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned blurred image super-resolution reconstruction method based on an event stream.

[0049] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned blurred image super-resolution reconstruction method based on an event stream.

[0050] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned blurred image super-resolution reconstruction method based on an event stream.

[0051] The one or more technical solutions provided by the present application have at least the following technical effects:

[0052] By acquiring two blurred images and their corresponding event streams, and constructing event double-integral features using the event streams, time information is introduced into the clear restoration process of the blurred images, effective fusion of blurred images and dynamic information is realized, and the problem that dynamic blurred regions cannot be clearly restored due to the dependence on a single blurred image in existing blurred image reconstruction methods is solved. Further, by constructing a first reconstructed image and a second reconstructed image and respectively performing upsampling, two candidate clear images are generated, and then based on the two clear images, an optimal result is adaptively selected as a target clear image, so that the model can dynamically adjust the image output path according to the image content and time sequence features, and the precision loss caused by the fixed processing flow is avoided. In addition, a target loss function is constructed using multi-source input, and the network parameters of the target clear image are optimized in the end-to-end structure, thereby forming unified optimization for super-resolution reconstruction, effectively improving the resolution and clarity of the finally generated image. Compared with the prior art, the high spatiotemporal resolution advantage of the event stream is fully utilized, a multi-frame complementary and adaptive selection mechanism is introduced, the image super-resolution effect in the high-speed motion or severe blur scene is significantly improved, and clearer and more accurate image reconstruction in a real dynamic blur environment is realized. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0055] Figure 1 A flowchart provided by the blurred image super-resolution reconstruction method based on event stream of the present application embodiment one;

[0056] Figure 2 A learnable division mapping diagram provided by the blurred image super-resolution reconstruction method based on event stream of the present application embodiment one;

[0057] Figure 3 A flowchart provided by the blurred image super-resolution reconstruction method based on event stream of the present application embodiment two;

[0058] Figure 4 A learnable double-integral module diagram provided by the blurred image super-resolution reconstruction method based on event stream of the present application embodiment two;

[0059] Figure 5A simplified flowchart illustrating the event-flow-based super-resolution reconstruction method for blurred images provided in Embodiment 2 of this application;

[0060] Figure 6 This is a schematic diagram of the module structure of the event-stream-based super-resolution reconstruction device for blurred images according to an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the event-flow-based fuzzy image super-resolution reconstruction method in the embodiments of this application.

[0062] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0064] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0065] The main solution of this application embodiment is as follows: Acquire a first blurred image, a first event stream, a second blurred image, and a second event stream; convert the first event stream and the second event stream into first event double integral features and second event double integral features, respectively; generate a first reconstructed image based on the first blurred image and the first event double integral features, and generate a second reconstructed image based on the second blurred image and the second event double integral features; upsample the first reconstructed image and the second reconstructed image to obtain a first clear image and a second clear image, and adaptively select a target clear image based on the first clear image and the second clear image; construct a target loss function based on the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image, and the second clear image; optimize the network parameters of the target clear image based on the loss function to obtain a super-resolution image.

[0066] In this embodiment, for ease of description, the following description will focus on identifying an event-stream-based fuzzy image super-resolution reconstruction device.

[0067] Since the existing image super-resolution reconstruction has poor generalization ability and poor reconstruction effect in real complex scenes, the present application provides a solution. By obtaining two blurred images and their corresponding event streams, and using the event streams to construct event double integral features, the temporal information is introduced into the clear restoration process of the blurred images, effectively fusing the blurred images and dynamic information, solving the problem that the dynamic blurred area cannot be clearly restored due to the dependence on single blurred image in the existing blurred image reconstruction method. Further, by constructing the first and second reconstructed images and performing upsampling on them respectively, two candidate clear images are generated, and then the optimal result is adaptively selected as the target clear image based on the two clear images, so that the model can dynamically adjust the image output path according to the image content and temporal characteristics, avoiding the precision loss caused by the fixed processing flow. In addition, a target loss function is constructed using multi-source input, and the network parameters of the target clear image are optimized in the end-to-end structure, thereby forming a unified optimization for super-resolution reconstruction, effectively improving the resolution and clarity of the finally generated image. Compared with the prior art, the high spatiotemporal resolution advantage of the event stream is fully utilized, and a multi-frame complementary and adaptive selection mechanism is introduced, which significantly improves the image super-resolution effect in high-speed motion or severe blur scenes, and realizes clearer and more accurate image reconstruction in real dynamic blur environment.

[0068] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a blurred image super-resolution reconstruction device based on event stream, etc. In the following, the blurred image super-resolution reconstruction device based on event stream is taken as an example to illustrate the present embodiment and each of the following embodiments.

[0069] Based on this, the present embodiment provides a blurred image super-resolution reconstruction method based on event stream, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the present application is shown in the following.

[0070] In the present embodiment, the blurred image super-resolution reconstruction method based on event stream includes steps S10-S50:

[0071] Step S10, obtaining a first blurred image, a first event stream, a second blurred image and a second event stream, and converting the first event stream and the second event stream into first event double integral features and second event double integral features respectively;

[0072] It should be noted that the first blurred image and the second blurred image are images captured by the event camera at two consecutive time points. The blurred image is obviously blurred due to the movement of the target in the dynamic scene or the shaking of the camera equipment, and it is usually difficult to be directly used for accurate identification or reconstruction. The first blurred image and the second blurred image have continuity between them, and can reflect the target motion process in a short time.

[0073] In addition, the event stream is a sequence of events output by the event camera in a continuous time period, each event containing pixel position, timestamp and brightness change polarity information. The first event stream and the second event stream correspond to the first blurred image and the second blurred image, respectively, and can reflect the dynamic change information of the image in the two time segments.

[0074] Further, the event double-integral feature is obtained by integrating the event stream in space and time dimensions. The event double-integral feature integrates the brightness information of each pixel in the event stream over time, which helps to extract visual features with time consistency at different resolution stages. The first event double-integral feature and the second event double-integral feature correspond to the first event stream and the second event stream, respectively.

[0075] It can be understood that the first blurred image and the first event stream are a pair of time-consistent data pairs, which are used to depict the image state and motion change at a certain moment, and the second blurred image and the second event stream correspond to another time point. The first event stream and the second event stream are double-integrated respectively, that is, the time information of the event is normalized and accumulated, and combined with the spatial position information to form a double-channel strong space-time structure feature. This step lays the foundation for subsequent reconstruction of clear images using blurred images and event data cooperatively.

[0076] Step S20, generating a first reconstructed image according to the first blurred image and the first event double-integral feature, and generating a second reconstructed image according to the second blurred image and the second event double-integral feature;

[0077] It should be noted that the first reconstructed image and the second reconstructed image are potential clear images preliminarily generated by the neural network after the blurred image and the event double-integral feature are fused. The reconstructed image is partially restored in texture edge details compared with the original blurred image, but has not been optimized in final resolution and global consistency.

[0078] It can be understood that the first reconstructed image is generated by combining the first blurred image and the first event double integral feature, and the second reconstructed image is generated by combining the second blurred image and the second event double integral feature. The generation process can be completed by a neural network backbone, which has a double-branch architecture in this embodiment, and can extract image features and event features at two time points in parallel and output a preliminary image restoration result after fusion. By means of event guidance to realize structure detail recovery, the authenticity and integrity of the preliminary clear image can be improved.

[0079] In a possible implementation, the step S20 can include: performing downsampling on the first blurred image and the second blurred image to obtain a first downsampled image and a second downsampled image; performing convolution on the first downsampled image and the second downsampled image to obtain a first image feature map and a second image feature map; performing dense residual connection on the first image feature map and the second image feature map to obtain a first fusion feature map and a second fusion feature map; and performing convolution and upsampling on the first fusion feature map and the second fusion feature map to obtain a first reconstructed image and a second reconstructed image.

[0080] It should be noted that downsampling refers to an operation of reducing the resolution of a high-resolution image in the spatial dimension, and common methods include max pooling, average pooling or step convolution. Downsampling can reduce the computational complexity, enhance the stability of local features, and facilitate subsequent feature extraction in a unified scale.

[0081] Further, the first downsampled image and the second downsampled image are the results of downsampling the first blurred image and the second blurred image respectively, and have a relatively low spatial resolution, mainly retaining the approximate texture and structure information in the original image.

[0082] In addition, convolution is a common feature extraction method in deep learning. By sliding a set of learnable convolution kernels over the input image, spatial texture, edge, direction and other information in the local region are extracted. Convolution operation has strong expression ability for the shape of the image and the outline of the object.

[0083] Further, the first image feature map and the second image feature map are deep representations extracted by performing convolution operation on the first downsampled image and the second downsampled image respectively. The structure features, blurred edges and texture patterns of the image at different spatial positions can be extracted, so that the features of the blurred area can be more clearly encoded and represented. Convolution operation strengthens the ability to distinguish the properties such as blur degree and direction in the image.

[0084] It can be understood that the dense residual connection is a network structure design method, which enhances feature reuse and alleviates the gradient vanishing problem by connecting feature maps between different layers. Among them, the dense means stacking the outputs of multiple previous layers as input, and the residual means preserving the original input to assist in learning residual information. Dense residual connection helps to keep low-level information in deep network and improve model stability.

[0085] Further, the first fusion feature map and the second fusion feature map are output results after the dense residual connection operation is performed on the first image feature map and the second image feature map respectively. The fusion feature map has both the structural contour of the original image and the higher-level semantic expression, thereby improving the ability of the network to identify and restore the blurred area. The dense residual connection also strengthens the context association between different resolutions, effectively alleviating the training instability problem of the deep model.

[0086] It can be understood that by using the learnable division mapping (LDR), the low-resolution clear image can be estimated to obtain the first reconstructed image and the second reconstructed image. The calculation formula of the first reconstructed image and the second reconstructed image is as follows:

[0087]

[0088] In the formula, B i is the first blurred image, is the first event double integral feature, is the first reconstructed image; B i+1 is the second blurred image, is the second event double integral feature, is the second reconstructed image, and LDR is the learnable division mapping.

[0089] Referring to Figure 2 , Figure 2 is the learnable division mapping of the first embodiment of the event stream-based blurred image super-resolution reconstruction method of the present application.

[0090] As Figure 2 shown, the input feature first enters the left pixel grouping module (Pixel Shuffle, PS) for down-sampling resolution reduction processing, and then sequentially passes through the convolution layer (Conv) for basic feature extraction. The convolution layer is followed by a nonlinear activation function, and then the feature flows into the deep feature mining area composed of multiple dense residual connection modules (RDB 1, RDB D, etc.), which further learns high-order information using the dense residual structure; after completing the deep feature extraction, the data flows to the right pixel grouping module (PS) for up-sampling resolution increase processing, and then passes through the convolution layer (Conv) to restore the channel dimension and refine the feature, and finally outputs the low-resolution clear image

[0091] Step S30, up-sampling the first reconstructed image and the second reconstructed image respectively to obtain a first clear image and a second clear image, and adaptively selecting a target clear image according to the first clear image and the second clear image;

[0092] It can be understood that up-sampling is an operation of converting a low-resolution image into a high-resolution image, and common methods include deconvolution, nearest neighbor interpolation or pixel remapping, etc. In the embodiment, up-sampling is used to improve the resolution of the reconstructed image to provide high-quality clear images for subsequent fusion and output.

[0093] Further, the first clear image and the second clear image are respectively obtained by up-sampling the first reconstructed image and the second reconstructed image, and the resolution is higher than that of the original reconstructed image, and the target is to approach the structural standard of the final super-resolution image.

[0094] In addition, the target clear image is adaptively selected from the first clear image and the second clear image. In the embodiment, the adaptive strategy is based on a clearness evaluation function or a feature fusion module to automatically select a frame of image with more complete structural information and clearer texture for subsequent optimization.

[0095] In addition, it can be understood that the first clear image and the second clear image are generated by up-sampling the first reconstructed image and the second reconstructed image respectively. After obtaining the two clear images, the target clear image is adaptively selected by the evaluation mechanism obtained by training according to the image quality, motion residual and texture details and other indicators.

[0096] In a possible implementation, step S30 can include: obtaining a current time, a first exposure period and a second exposure period; determining a first adaptive value according to the current time and the first exposure period; determining a second adaptive value according to the current time and the second exposure period; when the first adaptive value is less than the second adaptive value, selecting the first clear image as the target clear image; and when the first adaptive value is greater than or equal to the second adaptive value, selecting the second clear image as the target clear image.

[0097] It should be noted that the current time refers to the global time node corresponding to the current processing process, which is used to unify and synchronize the time base of different modal data.

[0098] In addition, the first exposure period is the exposure time period of the image sensor in the process of obtaining the first blurred image, and the second exposure period is the exposure time period of obtaining the second blurred image. The length of the exposure period directly affects the degree of blurring of the image, and the longer the period, the more obvious the blurring.

[0099] In addition, it should be noted that the first adaptive value is a quantitative result obtained by comparing and analyzing the current time and the first exposure period, and is used to represent the blur influence degree of the first blurred image at the current time point. The second adaptive value is a quantitative index calculated based on the relationship between the current time and the second exposure period, and is used to measure the time proximity and blur possibility of the second blurred image.

[0100] It can be understood that for any time m, two high-resolution clear images, i.e., the first clear image and the second clear image, can be obtained through the double-path parallel calculation branch structure, and the final blurred image super-resolution result can be obtained through the time adaptive selection strategy. The target clear image calculation formula is as follows:

[0101]

[0102] In the formula, is the target clear image, m is the current time, is the first exposure period, is the second exposure period, is the first adaptive value, is the second adaptive value, is the first clear image, is the second clear image. In addition, and respectively represent the intermediate time of the first exposure period and the second exposure period.

[0103] Specifically, if the first adaptive value is less than the second adaptive value, it means that the blur degree of the first clear image is relatively small, and therefore the image is preferentially selected as the target clear image; if the first adaptive value is greater than or equal to the second adaptive value, it means that the blur degree of the second clear image is small or comparable, and therefore the second clear image is selected as the target clear image, which is used to represent the high-quality image output at the current time.

[0104] Step S40, constructing a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image;

[0105] It should be noted that the target loss function is a function for measuring the difference between the current network prediction output and the expected result, and is commonly used to guide the model training direction. In the embodiment, the loss function comprehensively considers the alignment relationship between the blurred image, the event stream, the reconstructed image and the clear image, and embodies the multi-level feature consistency.

[0106] It can be understood that a target loss function for subsequent training is constructed. The loss function comprehensively uses the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image, and the second clear image, and other inputs and intermediate features. The consistency of the blurred image and the clear image can be maintained in the static area, the event-guided constraint weight is enhanced in the dynamic area, and balance between structure restoration and detail enhancement is ensured.

[0107] In an implementable embodiment, step S40 can include steps S41-S46:

[0108] In step S41, a preset image sample set and a downsampling operator are acquired, and a super-resolution consistency loss function is constructed according to the preset image sample set and the downsampling operator.

[0109] It should be noted that the preset image sample set refers to a set of high-resolution clear images contained in an existing public data set. In this embodiment, the preset image sample set is randomly extracted to provide basic reference data for constructing the super-resolution consistency loss function, for constraining the performance of the super-resolution network in terms of color and details.

[0110] In addition, the downsampling operator is an operation function for mapping a high-resolution image to a low-resolution image, and is commonly used to simulate the acquisition process of an actual blurred image. In this embodiment, the downsampling operator acts on the image data in the public image sample set to compress the image data in terms of spatial resolution, for establishing a training relationship between the clear image and the blurred image.

[0111] It can be understood that the super-resolution consistency loss function is a loss evaluation function constructed using the preset image sample set and the downsampling operator, for simultaneously constraining the color fidelity of the super-resolution network and the detail information in the spatial scale. The super-resolution consistency loss function is expressed as follows:

[0112]

[0113] In the formula, USR is an upsampling reconstruction module, is a downsampling operator, Y is a preset image sample set, and LSR is a super-resolution consistency loss function.

[0114] In step S42, a first resolution latent clear image consistency loss function is constructed according to the first reconstructed image and the second reconstructed image, where the first resolution latent clear image consistency loss function is used to constrain the consistency of the brightness and texture of the initial clear image.

[0115] It should be noted that the first resolution latent clear image consistency loss function is based on the assumption that the brightness and texture at the same moment m should be consistent, and the method constructs a low resolution latent clear image consistency loss function. The first resolution latent clear image consistency loss function is expressed as follows:

[0116]

[0117] In the formula, is the first reconstructed image, is the second reconstructed image, is the first resolution latent clear image consistency loss function.

[0118] Step S43, constructing a re-blur consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image and the second blurred image;

[0119] It can be understood that the re-blur consistency loss function is used to measure the image restoration effect between the blurred image and the corresponding clear image in the static area. By judging the distribution of static and dynamic areas in the image through the event stream, the brightness and texture consistency of the reconstructed image and the original blurred image is only constrained in the static area, thereby improving the discriminability of the model.

[0120] In a feasible implementation, step S43 can include the steps of: mapping according to the first reconstructed image and the first event stream to determine a first re-blurred image; mapping according to the second reconstructed image and the second event stream to determine a second re-blurred image; and constructing a re-blur consistency loss function according to the first re-blurred image, the first blurred image, the second re-blurred image and the second blurred image.

[0121] It can be understood that mapping refers to a conversion process through the inverse operation of a learnable division mapping module. The inverse operation of the division mapping can utilize the event double integral inverse feature converted by the event stream to process the reconstructed image, simulate the generation process of the blurred image, convert the low resolution clear image into a blurred image, and realize the reverse mapping from clear to blurred.

[0122] Further, the first deblurred image is a blurred image obtained through the mapping process described above, based on the first reconstructed image, combined with the motion and brightness change information contained in the first event stream, simulating the generation process of the first blurred image, and finally obtaining a deblurred result corresponding to the first blurred image in exposure period and scene content, for subsequent consistency comparison with the original first blurred image. The second deblurred image is a blurred image obtained by mapping the second reconstructed image, based on the second reconstructed image and the second event stream, simulating the generation process of the second blurred image, corresponding to the second blurred image in exposure period and scene content, for subsequent consistency verification with the original second blurred image.

[0123] In addition, the deblurred image can be calculated through the learnable division mapping, and the calculation formula of the first deblurred image and the second blurred image is as follows:

[0124]

[0125] In the formula, is the first reconstructed image, is the second reconstructed image, is the first event double integral feature, is the second event double integral feature, and LDR is the learnable division mapping, is the first deblurred image, is the second blurred image. Wherein, and are used to reconstruct and The method uses a cross-supervised strategy to cleverly use the consistency of and , on the one hand, avoiding the identity mapping, and on the other hand, effectively using the sharp complementary information between adjacent blurred images.

[0126] It can be understood that the reconstructed blurred images and can be constructed with the original blurred images B i and B i+1 A deblurring consistency loss function is constructed, and the expression of the deblurring consistency loss function is as follows:

[0127]

[0128] In the formula, is the first deblurred image, is the second deblurred image, B i is the first blurred image, and B i+1 is the second blurred image.

[0129] Step S44, constructing a second resolution latent clear image consistency loss function according to the first clear image and the second clear image, wherein the second resolution latent clear image consistency loss function is used to constrain the consistency of brightness and texture of the optimized clear image;

[0130] It can be understood that the second resolution latent clear image consistency loss function is based on the assumption that the brightness and texture at the same time m should be consistent, and the method constructs a high resolution latent clear image consistency loss function. The second resolution latent clear image consistency loss function is expressed as follows:

[0131]

[0132] In the formula, is the first clear image, is the second clear image, is the second resolution latent clear image consistency loss function.

[0133] Step S45, constructing a first blurred-clear consistency loss function and a second blurred-clear consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream and the down-sampling operator;

[0134] It can be understood that the first blurred-clear consistency loss function and the second blurred-clear consistency loss function are respectively used to constrain the brightness and texture consistency of the low resolution latent clear image and the low resolution blurred image in the static area, and the motion information contained in the event stream can distinguish the static and dynamic areas in the image. Due to the brightness and texture consistency of the high resolution latent clear image and the low resolution blurred image in the static area, and the motion information contained in the event stream can distinguish the static and dynamic areas in the image, the blurred-clear consistency constraint is constructed accordingly.

[0135] In a feasible implementation, step S45 can include steps S451-S453:

[0136] Step S451, obtaining a mask matrix;

[0137] It should be noted that the mask matrix is a matrix constructed according to the event stream, which is used to distinguish the static area and the dynamic area in the image. In this embodiment, the event stream contains the brightness change information of all pixels in the scene: when an event is triggered in a certain area, i.e. there is object motion or rapid brightness change in the area, it belongs to the dynamic area, and the value of the corresponding position in the mask matrix is 0; when there is no event triggered in a certain area, i.e. there is no obvious motion or brightness change in the area, it belongs to the static area, and the value of the corresponding position in the mask matrix is 1.

[0138] Step S452, constructing a first blur-sharp consistency loss function according to the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream;

[0139] It can be understood that, since the low-resolution potential sharp image and the low-resolution blurred image maintain the consistency of brightness and texture in the static region, and the motion information contained in the event stream can distinguish the static and dynamic regions in the image, a blur-sharp consistency constraint is constructed accordingly. The expression of the first blur-sharp consistency loss function is as follows:

[0140]

[0141] In the formula, is the first reconstructed image, is the second reconstructed image, B i is the first blurred image, B i+1 is the second blurred image, is the mask matrix, is the first event stream, is the second event stream.

[0142] Step S453, constructing a second blur-sharp consistency loss function according to the down-sampling operator, the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream.

[0143] It can be understood that, since the high-resolution potential sharp image and the low-resolution blurred image maintain the consistency of brightness and texture in the static region, and the motion information contained in the event stream can distinguish the static and dynamic regions in the image, a blur-sharp consistency constraint is constructed accordingly. The expression of the second blur-sharp consistency loss function is as follows:

[0144]

[0145] In the formula, is the first reconstructed image, is the second reconstructed image, B i is the first blurred image, B i+1 is the second blurred image, is the mask matrix, is the first event stream, is the second event stream, is the down-sampling operator.

[0146] Step S46, according to the super-resolution consistency loss function, the first resolution latent clear image consistency loss function, the re-blur consistency loss function, the first blur clear consistency loss function, the second resolution latent clear image consistency loss function, and the second blur clear consistency loss function, and the target loss function is constructed.

[0147] It can be understood that all the aforementioned sub-loss functions are integrated to construct a complete target loss function. The loss function includes image consistency, structure fidelity, static region restoration, and super-resolution reconstruction, etc. multiple evaluation dimensions are used to guide network learning and finally output high-quality super-resolution images. Through the target loss function, the advantages of each loss function can be complementary, so that the final model still has excellent image restoration ability in complex scenes. The target loss function calculation formula is as follows:

[0148]

[0149] In the formula, α1 to α5 are the balance coefficients of each loss term, and the parameter optimization can use the back propagation, gradient descent algorithm, and use the corresponding related function AdamOptimizer in Pytorch to realize; is the first resolution latent clear image consistency loss function, is the re-blur consistency loss function, is the first blur clear consistency loss function, is the second resolution latent clear image consistency loss function, is the second blur clear consistency loss function, is the super-resolution consistency loss, is the target loss function.

[0150] Step S50, according to the loss function, the network parameters of the target clear image are optimized to obtain a super-resolution image.

[0151] It should be noted that the network parameters refer to the weights and bias values of each layer in the neural network, which are important carriers for model learning data features. Optimizing network parameters can continuously improve the output quality of the model, making the output image more consistent with the target image.

[0152] Further, the super-resolution image refers to the result of restoring a blurred low-resolution image to a clear high-resolution image. In this embodiment, the final super-resolution image is a clear image generated by a multi-stage network and a multi-loss function optimization, which has higher detail restoration degree and structural integrity.

[0153] It can be understood that, by back propagation through the constructed target loss function, the network parameters for generating a target clear image are optimized, and an output super-resolution image is output. By adjusting the parameters by using loss feedback, the output image continuously approaches the ideal state in terms of clarity, texture fidelity, motion consistency, etc. The resolution and visual quality of the finally generated image are improved, which can meet the image reconstruction requirements in real scenes and enhance the application applicability and reconstruction ability of the model.

[0154] It should be understood that the performance of the embodiments is tested on the deblurring and super-resolution field standard public data sets GoPro and REDS. These data sets cover real complex scenes indoors and outdoors, and contain physical imaging level blur caused by camera jitter and high-speed motion of objects. The experiment strictly follows the field benchmark method, and the low-resolution blurred image is synthesized by downsampling the original image frame by 2 and 4 times, and averaging 11 consecutive frames.

[0155] Experiments show that the present technology has achieved good technical effects in the high-quality reconstruction task of the GoPro and REDS data sets. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as objective evaluation indexes for performance evaluation, and the specific experimental results are shown in Table 1 of the experimental results table.

[0156] Table 1

[0157]

[0158] As can be seen from the performance comparison in the table, the proposed method (SE-SRB) achieves good performance on both 4-fold and 2-fold reconstruction on the two data sets, far superior to other comparison methods, effectively demonstrating the effectiveness of the proposed method.

[0159] The embodiment provides a blurred image super-resolution reconstruction method based on event flow, which introduces event flow guided self-supervised constraints and constructs a lightweight dual parallel network framework, solves the technical problems of weak generalization ability of existing supervised methods in real scenes and low inference calculation efficiency, and achieves the beneficial effects of improving the generalization performance and model inference efficiency of blurred image super-resolution reconstruction in real complex environments. The clear image generated by the multi-stage network and the multi-loss function optimization has higher detail restoration degree and structural integrity, effectively enhancing the application applicability and reconstruction ability of the model.

[0160] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 3The step S10 of the event stream based blurred image super-resolution reconstruction method comprises steps S11-S13:

[0161] Step S11, acquiring a first blurred image and a second blurred image captured by the event camera in a continuous acquisition period;

[0162] It should be noted that the continuous acquisition period refers to two exposure periods that are consecutive, i.e., a first exposure period and a second exposure period immediately following the first exposure period. In this embodiment, each exposure period corresponds to a time interval in which a blurred image is captured by a traditional camera, and the continuous acquisition period ensures that the acquired images can reflect the changes in the scene over a continuous period of time.

[0163] In addition, the event camera is a new type of visual sensor. The event camera outputs an event when a pixel detects a change in brightness that exceeds a preset threshold. Each event contains the exact time of occurrence, the pixel position, and the direction of the brightness change, and can capture high-speed moving scenes with a time resolution of microseconds, effectively avoiding the blur produced by traditional cameras when moving quickly, while the amount of data output is smaller and suitable for real-time processing.

[0164] It can be understood that through the continuous work of the event camera in the continuous acquisition period, two adjacent frames of blurred images in time are acquired, which are respectively denoted as a first blurred image and a second blurred image. The first blurred image and the second blurred image are the basis for subsequent event stream alignment, clear image reconstruction, and consistency constraint.

[0165] Step S12, acquiring a first event stream and a second event stream corresponding to the first blurred image and the second blurred image, respectively;

[0166] It can be understood that the first event stream and the second event stream are extracted from the event camera according to the exposure time period corresponding to the first blurred image and the second blurred image. This process ensures that the time range of the event stream strictly corresponds to the blurred image, so that the event data accurately reflects the dynamic changes during the generation of the image frame, thereby providing an accurate spatiotemporal information basis for subsequent construction of event features and auxiliary image reconstruction.

[0167] Step S13, performing event stream preprocessing on the first event stream and the second event stream, converting the first event stream and the second event stream into first event bilinear features and second event bilinear features, respectively, wherein the event stream preprocessing comprises convolution processing and activation function processing.

[0168] It can be understood that, in combination with the physical generation model of the event camera and the generation model of the blurred image, the physical relationship between the blurred image B, the clear image I, and the event stream can be obtained, i.e., the time bilinear physical relationship expression is as follows:

[0169]

[0170] wherein, is the exposure period of the blurred image B, c is the threshold value of the event stream trigger, B is the blurred image, I is the sharp image, and m is the current time. However, due to the defects of the event camera sensor hardware, the estimation of the threshold value c is usually disturbed by long-time noise accumulation and global instability, so that the direct calculation of by the physical formula will produce errors. Therefore, a learning calculation with network parameters is performed by a learnable double integration module (LDI), and the first event double integration feature and the second event double integration feature are calculated as follows:

[0171]

[0172] wherein, is the first event double integration feature, E(m, T i+1 ) is the second event double integration feature, and are the preprocessing operations of the first event stream and the second event stream, respectively, and LDI is the learnable double integration. Refer to Figure 4 , Figure 4 for a schematic diagram of the learnable double integration module of the first embodiment of the blurred image super-resolution reconstruction method based on the event stream according to the present application.

[0173] As shown in Figure 4 , the learnable double integration module (LDI) is composed of three cascaded encoding modules; the first two modules each contain a 3x3 convolution layer and a ReLU activation function immediately after the convolution layer, and the last module additionally adds a convolution layer as an output layer on the basis of the foregoing structure, and the output channel numbers of all the convolution layers are 32, 64, 32, and 1 in sequence.

[0174] The embodiment provides a blurred image super-resolution reconstruction method based on an event stream, which acquires a blurred image and its corresponding event stream by using an event camera in a continuous acquisition period, and performs an event stream preprocessing and a double integration feature conversion technical means, solves the technical problem that a traditional camera is prone to blurring in a fast motion scene and the data volume is large and difficult to process in real time, and achieves the beneficial effects of improving the time resolution of image acquisition, reducing the data volume, enhancing the capture capability of a high-speed motion scene, and providing an accurate spatiotemporal information basis for subsequent image reconstruction and consistency constraints.

[0175] By way of example, in order to facilitate understanding of the implementation process of the blurred image super-resolution reconstruction method based on the event stream obtained after the above-mentioned embodiment one, please refer to Figure 5 , Figure 5 for a schematic diagram of a brief process of a blurred image super-resolution reconstruction method based on an event stream, in particular:

[0176] are divided into two branches, respectively processing two consecutive low-resolution blurred images B i and B i+1 and the corresponding event stream and In branch 1, the low-resolution blurred image B i and the event stream are first input into a learnable double integration module, the data processed by the module is transmitted to a learnable division mapping module, and then the data enters an up-sampling reconstruction module, and finally a high-resolution clear image L i is output. In branch 2, the low-resolution blurred image B i+1 and the event stream are also first input into a learnable double integration module, and then transmitted to a learnable division mapping module, and then enter an up-sampling reconstruction module, and output a high-resolution clear image L i+1 . The output high-resolution clear images L i and L i+1 of the two branches are connected through a spatial domain super-resolution reconstruction constraint, while the input low-resolution blurred image B i and the event stream of branch 1 and the input low-resolution blurred image B i+1 and the event stream of branch 2 are connected through a temporal domain deblurring constraint, realizing the mapping of different low-resolution blurred frames to a low-resolution clear image and a high-resolution clear image at the same potential time, and constructing a deblurring consistency self-supervised loss in the time dimension and a super-resolution consistency loss in the spatial dimension, and optimizing the overall self-supervised network framework.

[0177] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the event stream-based blurred image super-resolution reconstruction method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0178] The present application also provides an event stream-based blurred image super-resolution reconstruction device, please refer to Figure 6 , the event stream-based blurred image super-resolution reconstruction device comprises:

[0179] A data acquisition module 10 is configured to acquire a first blurred image, a first event stream, a second blurred image, and a second event stream, and convert the first event stream and the second event stream into first event double integration features and second event double integration features, respectively.

[0180] The image reconstruction module 20 is configured to generate a first reconstructed image according to the first blurred image and the first event double integral feature, and generate a second reconstructed image according to the second blurred image and the second event double integral feature.

[0181] The image up-sampling module 30 is configured to up-sample the first reconstructed image and the second reconstructed image respectively to obtain a first clear image and a second clear image, and adaptively select a target clear image according to the first clear image and the second clear image.

[0182] The loss function construction module 40 is configured to construct a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image.

[0183] The image optimization module 50 is configured to optimize the network parameters of the target clear image according to the loss function to obtain a super-resolution image.

[0184] The event stream-based blurred image super-resolution reconstruction device provided in the present application adopts the event stream-based blurred image super-resolution reconstruction method in the above embodiments, and can solve the technical problems of insufficient generalization ability and poor reconstruction effect of image super-resolution reconstruction in real complex scenes. Compared with the prior art, the event stream-based blurred image super-resolution reconstruction device provided in the present application has the same beneficial effects as the event stream-based blurred image super-resolution reconstruction method provided in the above embodiments, and other technical features in the event stream-based blurred image super-resolution reconstruction device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0185] In an embodiment, the loss function construction module 40 is further configured to obtain a preset image sample set and a down-sampling operator, and construct a super-resolution consistency loss function according to the preset image sample set and the down-sampling operator; construct a first resolution latent clear image consistency loss function according to the first reconstructed image and the second reconstructed image, wherein the first resolution latent clear image consistency loss function is configured to constrain the consistency of brightness and texture of an initial clear image; construct a re-blur consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image, and the second blurred image; construct a second resolution latent clear image consistency loss function according to the first clear image and the second clear image, wherein the second resolution latent clear image consistency loss function is configured to constrain the consistency of brightness and texture of an optimized clear image; construct a first blurred clear consistency loss function and a second blurred clear consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream, and the down-sampling operator; and construct a target loss function according to the super-resolution consistency loss function, the first resolution latent clear image consistency loss function, the re-blur consistency loss function, the first blurred clear consistency loss function, the second resolution latent clear image consistency loss function, and the second blurred clear consistency loss function.

[0186] In an embodiment, the loss function construction module 40 is further configured to map the first reconstructed image and the first event stream to determine a first re-blurred image; map the second reconstructed image and the second event stream to determine a second re-blurred image; and construct a re-blur consistency loss function according to the first re-blurred image, the first blurred image, the second re-blurred image, and the second blurred image.

[0187] In an embodiment, the loss function construction module 40 is further configured to obtain a mask matrix; construct a first blurred clear consistency loss function according to the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream; and construct a second blurred clear consistency loss function according to the down-sampling operator, the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image, and the second event stream.

[0188] In an embodiment, the image upsampling module 30 is further configured to obtain a current time, a first exposure period and a second exposure period; determine a first adaptive value according to the current time and the first exposure period; determine a second adaptive value according to the current time and the second exposure period; select the first sharp image as a target sharp image when the first adaptive value is less than the second adaptive value; and select the second sharp image as the target sharp image when the first adaptive value is greater than or equal to the second adaptive value.

[0189] In an embodiment, the image reconstruction module 20 is further configured to perform downsampling according to the first blurred image and the second blurred image to obtain a first downsampled image and a second downsampled image; perform convolution according to the first downsampled image and the second downsampled image to obtain a first image feature map and a second image feature map; perform dense residual connection according to the first image feature map and the second image feature map to obtain a first fused feature map and a second fused feature map; and perform convolution and upsampling according to the first fused feature map and the second fused feature map to obtain a first reconstructed image and a second reconstructed image.

[0190] In an embodiment, the data acquisition module 10 is further configured to acquire a first blurred image and a second blurred image collected by an event camera in a continuous acquisition period; acquire a first event stream and a second event stream according to the first blurred image and the second blurred image, respectively; perform event stream preprocessing on the first event stream and the second event stream, and convert the first event stream and the second event stream into first event double-integral features and second event double-integral features, respectively, the event stream preprocessing including convolution processing and activation function processing.

[0191] The present application provides an event stream-based blurred image super-resolution reconstruction device, which comprises at least one processor and a memory in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the event stream-based blurred image super-resolution reconstruction method in Embodiment I.

[0192] Reference will be made to the following Figure 7This document illustrates a structural schematic diagram of an event-stream-based super-resolution reconstruction device suitable for implementing embodiments of this application. The event-stream-based super-resolution reconstruction device for blurred images in this application can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The event-stream-based super-resolution reconstruction device for blurred images shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0193] like Figure 7 As shown, the event-stream-based blurred image super-resolution reconstruction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the event-stream-based blurred image super-resolution reconstruction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the event-stream-based blurred image super-resolution reconstruction device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows an event-stream-based blurred image super-resolution reconstruction device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0194] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0195] The event stream-based fuzzy image super-resolution reconstruction device provided in the present application adopts the event stream-based fuzzy image super-resolution reconstruction method in the above-mentioned embodiments, and can solve the technical problems of insufficient generalization ability and poor reconstruction effect of image super-resolution reconstruction in real complex scenes. Compared with the prior art, the event stream-based fuzzy image super-resolution reconstruction device provided in the present application has the same beneficial effects as the event stream-based fuzzy image super-resolution reconstruction method provided in the above-mentioned embodiments, and other technical features in the event stream-based fuzzy image super-resolution reconstruction device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0196] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0197] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0198] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the event stream-based fuzzy image super-resolution reconstruction method in the above-mentioned embodiments.

[0199] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a RAM (Random Access Memory), a ROM (Read Only Memory), an erasable programmable read-only memory (EPROM), an optical fiber, a CD-ROM (CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0200] The above computer readable storage medium may be included in the event stream based fuzzy image super-resolution device, or may exist separately without being assembled into the event stream based fuzzy image super-resolution device.

[0201] The above computer readable storage medium carries one or more programs, which, when executed by the event stream based fuzzy image super-resolution device, cause the event stream based fuzzy image super-resolution device to: acquire a first fuzzy image, a first event stream, a second fuzzy image, and a second event stream, and convert the first event stream and the second event stream into first event double integral features and second event double integral features, respectively; generate a first reconstructed image according to the first fuzzy image and the first event double integral features, and generate a second reconstructed image according to the second fuzzy image and the second event double integral features; upsample the first reconstructed image and the second reconstructed image to obtain a first clear image and a second clear image, and adaptively select a target clear image according to the first clear image and the second clear image; construct a target loss function according to the first fuzzy image, the second fuzzy image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image, and the second clear image; optimize the network parameters of the target clear image according to the loss function to obtain a super-resolution image.

[0202] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0203] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0204] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0205] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the above-mentioned event stream-based fuzzy image super-resolution reconstruction method, and can solve the technical problems of insufficient generalization ability and poor reconstruction effect of image super-resolution reconstruction in real complex scenes. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the event stream-based fuzzy image super-resolution reconstruction method provided by the above-mentioned embodiments, and will not be described here.

[0206] The application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the event stream based fuzzy image super-resolution reconstruction method as described above.

[0207] The computer program product provided by the application can solve the technical problems of insufficient generalization ability and poor reconstruction effect of image super-resolution reconstruction in real complex scenes. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the event stream based fuzzy image super-resolution reconstruction method provided by the above-mentioned embodiments, and are not described here.

[0208] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the application, and the contents of the specification and drawings are included in the patent protection scope of the application.

Claims

1. A method for reconstructing a blurred image based on event stream, characterized in that, The method comprises: obtaining a first blurred image, a first event stream, a second blurred image and a second event stream, and converting the first event stream and the second event stream into first event bilinear features and second event bilinear features respectively; generating a first reconstructed image according to the first blurred image and the first event bilinear features, and generating a second reconstructed image according to the second blurred image and the second event bilinear features; up-sampling the first reconstructed image and the second reconstructed image to obtain a first clear image and a second clear image, and adaptively selecting a target clear image according to the first clear image and the second clear image; constructing a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image; optimizing network parameters of the target clear image according to the loss function to obtain a super-resolution image.

2. The method of claim 1, wherein, The step of constructing a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image and the second clear image comprises: obtaining a preset image sample set and a down-sampling operator, and constructing a super-resolution consistency loss function according to the preset image sample set and the down-sampling operator; constructing a first resolution latent clear image consistency loss function according to the first reconstructed image and the second reconstructed image, wherein the first resolution latent clear image consistency loss function is used to constrain the consistency of brightness and texture of an initial clear image; constructing a remodified blur consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image and the second blurred image; constructing a second resolution latent clear image consistency loss function according to the first clear image and the second clear image, wherein the second resolution latent clear image consistency loss function is used to constrain the consistency of brightness and texture of an optimized clear image; constructing a first blurred clear consistency loss function and a second blurred clear consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream and the down-sampling operator; constructing a target loss function according to the super-resolution consistency loss function, the first resolution latent clear image consistency loss function, the remodified blur consistency loss function, the first blurred clear consistency loss function, the second resolution latent clear image consistency loss function and the second blurred clear consistency loss function.

3. The method of claim 2, wherein, The step of constructing a remodified blur consistency loss function according to the first reconstructed image, the second reconstructed image, the first event stream, the second event stream, the first blurred image and the second blurred image comprises: mapping according to the first reconstructed image and the first event stream, determine a first deblurred image; mapping according to the second reconstructed image and the second event stream, determine a second deblurred image; construct a deblurred consistency loss function according to the first deblurred image, the first blurred image, the second deblurred image and the second blurred image.

4. The method of claim 2, wherein, The step of constructing the first blurred-sharp consistency loss function and the second blurred-sharp consistency loss function according to the first reconstructed image, the first blurred image, the first event stream, the second reconstructed image, the second blurred image, the second event stream and the down-sampling operator comprises: obtain a mask matrix; construct a first blurred-sharp consistency loss function according to the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image and the second event stream; construct a second blurred-sharp consistency loss function according to the down-sampling operator, the first reconstructed image, the first blurred image, the mask matrix, the first event stream, the second reconstructed image, the second blurred image and the second event stream.

5. The method of claim 1, wherein, The step of adaptively selecting a target sharp image according to the first sharp image and the second sharp image comprises: obtain a current time, a first exposure period and a second exposure period; determine a first adaptive value according to the current time and the first exposure period; determine a second adaptive value according to the current time and the second exposure period; select the first sharp image as the target sharp image when the first adaptive value is less than the second adaptive value; select the second sharp image as the target sharp image when the first adaptive value is greater than or equal to the second adaptive value.

6. The method of claim 1, wherein, The step of generating the first reconstructed image according to the first blurred image and the first event double-integral feature, and generating the second reconstructed image according to the second blurred image and the second event double-integral feature comprises: down-sample according to the first blurred image and the second blurred image to obtain a first down-sampled image and a second down-sampled image; convolve according to the first down-sampled image and the second down-sampled image to obtain a first image feature map and a second image feature map; perform dense residual connection according to the first image feature map and the second image feature map to obtain a first fusion feature map and a second fusion feature map; convolve and up-sample according to the first fusion feature map and the second fusion feature map to obtain a first reconstructed image and a second reconstructed image.

7. The method of claim 1, wherein, The step of obtaining the first blurred image, the first event stream, the second blurred image and the second event stream, and converting the first event stream and the second event stream into the first event double-integral feature and the second event double-integral feature respectively comprises: obtain the first blurred image and the second blurred image collected by the event camera in a continuous collection period; obtain the corresponding first event stream and the second event stream according to the first blurred image and the second blurred image respectively; The first event stream and the second event stream are subjected to event stream preprocessing, and the first event stream and the second event stream are converted into first event double integral features and second event double integral features respectively, and the event stream preprocessing includes convolution processing and activation function processing.

8. A super-resolution reconstruction device for blurred images based on event flow, characterized in that, The device comprises: The data acquisition module is configured to acquire a first blurred image, a first event stream, a second blurred image, and a second event stream, and convert the first event stream and the second event stream into first event double integral features and second event double integral features respectively. The image reconstruction module is configured to generate a first reconstructed image according to the first blurred image and the first event double integral features, and generate a second reconstructed image according to the second blurred image and the second event double integral features. The image up-sampling module is configured to up-sample the first reconstructed image and the second reconstructed image respectively to obtain a first clear image and a second clear image, and adaptively select a target clear image according to the first clear image and the second clear image. The loss function construction module is configured to construct a target loss function according to the first blurred image, the second blurred image, the first event stream, the second event stream, the first reconstructed image, the second reconstructed image, the first clear image, and the second clear image. The image optimization module is configured to optimize network parameters of the target clear image according to the loss function to obtain a super-resolution image.

9. An event stream based device for super-resolution of a blurred image, characterized in that, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the event stream-based blurred image super-resolution reconstruction method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the event stream-based blurred image super-resolution reconstruction method according to any one of claims 1 to 7.

Images