Image processing method and system, storage medium and electronic device

Abstract

The invention discloses an image processing method and system, a storage medium and an electronic device, which are applied to an image processing system, and the method comprises the steps: obtaining a mosaic color event signal and a blurred image from an image collection module; inputting the mosaic color event signal and the blurred image into an event demosaicing module of the image processing model to obtain a demosaicing full-color event signal output by the event demosaicing module; inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module; and inputting the demosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model, and obtaining a deblurred image output by the fusion module.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and system, a storage medium and an electronic device. Background Art

[0002] At present, event cameras, relying on the advantages of their imaging principles, have achieved better imaging effects than traditional images under difficult imaging conditions such as high speed and high dynamic range, and are suitable for collaborative imaging deblurring with traditional cameras.

[0003] However, the current collaborative imaging devices and algorithms are all based on monochrome event cameras, and the color performance in color imaging scenarios is not excellent. Although there are currently color event cameras equipped with color filter arrays, when using the collaborative imaging deblurring method based on monochrome events to deblur images, not only is the deblurring effect poor, but for images with complex colors, color artifacts such as color overflow and edge anomalies will also appear. In other words, the related art lacks an imaging algorithm that can reasonably utilize color events, and color events cannot be used for collaborative imaging. In addition, there are differences in data modalities, and the de-mosaicing algorithm for traditional color images is not suitable for direct use on the mosaic original signal of color events, and it is impossible to achieve de-mosaicing for mosaic color events. Therefore, in the related art, in color imaging scenarios, there is a problem that color events cannot be used for collaborative imaging, and de-mosaicing cannot be achieved for mosaic color events.

[0004] However, no effective solution has been proposed to this problem yet. Summary of the invention

[0005] Embodiments of the present invention provide an image processing method and system, a storage medium and an electronic device.

[0006] According to one aspect of an embodiment of the present invention, there is provided an image processing method, which is applied to an image processing system, wherein a data transmission channel is provided between the image processing system and an image acquisition module, and the image processing system is pre-installed with at least an image processing model, including: acquiring a mosaic color event signal and a blurred image from the image acquisition module; inputting the mosaic color event signal and the blurred image into an event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module; and inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module; and inputting the demosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model to obtain a deblurred image output by the fusion module.

[0007] Furthermore, in an exemplary embodiment, the mosaic color event signal and the blurred image are input into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module, including: inputting the mosaic color event signal into the feature extraction layer of the event demosaicing module to obtain a first image feature, and inputting the blurred image into the feature extraction layer of the event demosaicing module to obtain a second image feature; sending the first image feature and the second image feature to the feature fusion layer of the event demosaicing module to obtain a demosaiced full-color event signal output by the feature fusion layer.

[0008] Furthermore, in an exemplary embodiment, the method also includes: using historical mosaic color event signals and historical blurred images as input samples and supervised full-color event signals as output samples to train the event demosaicing module, wherein a first supervised attention layer is provided after the feature fusion layer, and the first supervised attention layer is used to learn the data association relationship between the output result of the feature fusion layer and the supervised full-color event signal.

[0009] Further, in an exemplary embodiment, the mosaic color event signal and the blurred image are input into the image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module, including: sending the mosaic color event signal and the blurred image to the image feature encoding layer of the image deblurring module; performing feature extraction on the mosaic color event signal at the image feature encoding layer to obtain a third image feature, and performing feature extraction on the blurred image to obtain a fourth image feature; after sending the third image feature and the fourth image feature to the cross-attention layer connected to the image feature encoder, sending a fusion result output by the cross-attention layer based on the third image feature and the fourth image feature to the image feature decoding layer, and determining the decoding result of the image feature decoding layer as the deblurred intermediate image.

[0010] Furthermore, in an exemplary embodiment, the method also includes: using historical mosaic color event signals and historical blurred images as input samples and using supervised clear images as output samples to train the image deblurring module, wherein a second supervised attention layer is provided after the image feature decoding layer, and the second supervised attention layer is used to learn the data association relationship between the output result of the image feature decoding layer and the supervised clear image.

[0011] Furthermore, in an exemplary embodiment, the historical blurred image is obtained in the following manner: using the image acquisition module to acquire images of a preset video played in an image acquisition area to obtain a clear image sequence, wherein the clear image sequence includes multiple frames of clear images; performing image blurring processing on each frame of the clear image to obtain a blurred processed image corresponding to each frame of the clear image, and determining the blurred processed image as the historical blurred image.

[0012] Furthermore, in an exemplary embodiment, the mosaic color event signal is acquired in the following manner: using the image acquisition module to perform image acquisition on the current acquisition area to obtain image signals in different color channels; performing signal calibration on the image signals in the different color channels in the time dimension and the space dimension respectively to obtain a full-size color event signal, and the pixels of the image signals in the different color channels have the same pixel position at the same moment; and selectively downsampling the image signals of the full-size color event signal in different color channels to obtain the mosaic color event signal.

[0013] Further, in an exemplary embodiment, the image acquisition module is used to capture the preset video played in the image acquisition area to obtain image signals in different color channels, including: using image acquisition modules corresponding to N different filters to capture the preset video played in the image acquisition area to obtain N image signals, wherein the filter types of the N different filters are different, and each filter type corresponds to a color channel; determining the image signals in the different color channels based on the N image signals, wherein the N image signals include at least one of the following: a first image signal captured through a red channel, a second image signal captured through a green channel, a third image signal captured through a blue channel, and a fourth image signal captured through a monochrome channel.

[0014] Furthermore, in an exemplary embodiment, the demosaiced full-color event signal and the deblurred intermediate image are input into a fusion module of the image processing model to obtain a deblurred image output by the fusion module, including: using the fusion module to extract the target event signal feature of the demosaiced color event signal, and using the fusion module to extract the fifth image feature of the deblurred intermediate image; obtaining the similarity between the fifth image feature and the target event signal feature; normalizing the similarity to obtain a normalized similarity, and setting the normalized similarity as a weight coefficient of the target image feature; using the product of the fifth image feature and the weight coefficient to update the fifth image feature to obtain a sixth image feature, and obtaining the deblurred image output by the fusion module after feature reconstruction of the sixth image feature.

[0015] Furthermore, in an exemplary embodiment, the method also includes: using historical de-mosaiced color event signals and historical blurred intermediate images as input samples and using supervised clear images as output samples to train the fusion module, wherein a third supervised attention layer is provided in the fusion module, and the third supervised attention layer is used to learn the data association relationship between the sixth image feature and the supervised clear image.

[0016] According to another aspect of an embodiment of the present invention, an image processing system is further provided. The image processing system is pre-installed with at least an image processing model, and the image processing system is used to execute the above-mentioned image processing method.

[0017] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned image processing method when running.

[0018] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image processing method through the computer program.

[0019] Through the present invention, an image processing model for collaborative imaging using color events is designed in an image processing system. On the basis of realizing de-mosaicing, full-size color event signals are used and color event signal fusion imaging is used to perform motion blur removal operations on color traditional images. Specifically, a mosaic color event signal and a blurred image are obtained from the image acquisition module; the mosaic color event signal and the blurred image are input into the event de-mosaic module of the image processing model to obtain a de-mosaiced full-color event signal output by the event de-mosaic module; and the mosaic color event signal and the blurred image are input into the image de-blurring module of the image processing model to obtain a de-blurred intermediate image output by the image de-blurring module; the de-mosaiced full-color event signal and the de-blurred intermediate image are input into the fusion module of the image processing model to obtain a de-blurred image output by the fusion module. Therefore, the technical solution of the present invention solves the problem that in a color imaging scenario, color events cannot be used for collaborative imaging, and that mosaic color events cannot be demosaiced. It not only realizes collaborative imaging using color events, but also can realize demosaicing for mosaic color events, thereby achieving color-fidelity image deblurring and improving the restoration effect of traditional color-fidelity images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 is a hardware structure block diagram of a server of the image processing method according to an embodiment of the present invention;

[0022] Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of the architecture of an image processing model according to an embodiment of the present invention (I);

[0024] Figure 4 is a schematic diagram of the architecture of an image processing model according to an embodiment of the present invention (II);

[0025] Figure 5 is a schematic diagram of the architecture of an event demosaicing module according to an embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of a generation principle of an imaging data set according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of the principle of constructing a mosaic full-color event according to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of constructing a blurred degraded frame according to an embodiment of the present invention;

[0029] Fig. 9 is a structural block diagram of an image processing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms and "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The method embodiments provided in the embodiments of the present invention can be executed in a server, a mobile terminal or a similar computing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0033] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, which is equivalent to implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] In this embodiment, an image processing method is provided, which is applied to an image processing system. A data transmission channel is provided between the image processing system and an image acquisition module. The image processing system is pre-installed with at least an image processing model, which includes an event demosaicing module, an image deblurring module and a fusion module.

[0036] Optionally, the image processing method of this embodiment can also be applied to operating systems such as vehicle-mounted systems, mobile phone systems, and computer systems.

[0037] It is understandable that the image processing method of this embodiment involves the process of analyzing, modifying, enhancing and interpreting image data, including but not limited to the following usage scenarios:

[0038] Medical Imaging: In the medical field, image processing is used to improve the quality of imaging technologies such as X-rays, CT scans, MRIs, etc., to help doctors make diagnoses.

[0039] Security Monitoring: In security monitoring systems, image processing technology is used for automatic detection of abnormal behavior, facial recognition, and license plate recognition.

[0040] Photography and Image Editing: In the field of art and media, image processing is used for retouching photos, color correction, filter application, etc.

[0041] Computer Vision: In the field of machine learning and artificial intelligence, image processing is the basis of computer vision algorithms, which are used for object recognition, scene understanding, etc.

[0042] Self-driving cars: Self-driving technology relies on image processing to analyze road conditions and identify traffic signs, pedestrians, and other vehicles.

[0043] Satellite image analysis: In Geographic Information System (GIS), image processing is used to analyze satellite images for land use classification, environmental monitoring, etc.

[0044] Industrial Inspection: In the manufacturing industry, image processing is used to automatically detect product defects and improve production efficiency and product quality.

[0045] Augmented Reality (AR) and Virtual Reality (VR): In AR and VR applications, image processing is used to create an immersive experience that merges virtual objects with the real world.

[0046] Social Media and Content Creation: On social media platforms, image manipulation is used to beautify pictures and videos uploaded by users.

[0047] Document scanning and optical character recognition (OCR): Image processing is used to improve the clarity of document scans and assist OCR technology in extracting text from images.

[0048] Film and Video Production: In film and video production, image processing is used in the creation and post-production of visual effects.

[0049] Biometrics: In identity verification systems, image processing is used for fingerprint, iris, and facial recognition.

[0050] Agricultural Monitoring: In precision agriculture, image processing is used for monitoring crop health, pest and disease detection, and yield assessment.

[0051] Cultural Relics Protection: In the field of cultural heritage, image processing is used for the digitization, restoration and protection of cultural relics.

[0052] Figure 2 : is a flowchart of an image processing method according to an embodiment of the present invention, and the process includes the following steps:

[0053] Step S202, acquiring a mosaic color event signal and a blurred image from the image acquisition module;

[0054] Step S204, inputting the mosaic color event signal and the blurred image into an event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module;

[0055] Step S206, inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module;

[0056] Step S208, inputting the demosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model to obtain a deblurred image output by the fusion module.

[0057] Optional, such as Figure 3 As shown, for the fusion module (i.e. Figure 3 The fusion process of the refined fusion module of the previous two modules is explained. The refined fusion module inherits the structural design of the previous two modules. The difference is that in order to strike a balance between performance and computational complexity, the refined fusion module has fewer feature extraction layers than before. The refined fusion module further fuses the deblurred intermediate result image (deblurred intermediate image) output by the image deblurring module and the demosaiced full-color event, optimizes and improves the deblurring result, focuses on the color performance, and makes up for the problem of the initial lack of three-channel color event signals in the previous image deblurring module. Among them, in order to better utilize the feature information between channels at a single pixel position and improve color accuracy during the refinement process, the fusion module also introduces a pixel attention mechanism (PA), which further enhances the color restoration quality of the image deblurring task and avoids artifacts such as color bleeding.

[0058] Through the above steps, an image processing model for collaborative imaging using color events is designed in the image processing system. On the basis of realizing de-mosaicing, full-size color event signals are used and color event signal fusion imaging is used to perform motion blur removal operations on color traditional images. Specifically, the mosaic color event signal and blurred image from the image acquisition module are obtained; the mosaic color event signal and the blurred image are input into the event de-mosaic module of the image processing model to obtain a de-mosaic full-color event signal output by the event de-mosaic module; and the mosaic color event signal and the blurred image are input into the image de-blurring module of the image processing model to obtain a de-blurred intermediate image output by the image de-blurring module; the de-mosaic full-color event signal and the de-blurred intermediate image are input into the fusion module of the image processing model to obtain a de-blurred image output by the fusion module. Therefore, the technical solution of the present invention solves the problem that in a color imaging scenario, color events cannot be used for collaborative imaging, and that mosaic color events cannot be demosaiced. It not only realizes collaborative imaging using color events, but also can realize demosaicing for mosaic color events, thereby achieving color-fidelity image deblurring and improving the restoration effect of traditional color-fidelity images.

[0059] Furthermore, in an exemplary embodiment, the implementation process of inputting the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain the demosaiced full-color event signal output by the event demosaicing module includes: inputting the mosaic color event signal into the feature extraction layer of the event demosaicing module to obtain the first image feature, and inputting the blurred image into the feature extraction layer of the event demosaicing module to obtain the second image feature; sending the first image feature and the second image feature to the feature fusion layer of the event demosaicing module, and obtaining the demosaiced full-color event signal output by the feature fusion layer. In this embodiment, reference is made to Figure 4 It can be seen that the mosaic color event signal (i.e. Figure 4 After the mosaic event) is input into the event demosaicing module, the event demosaicing module uses the output demosaiced full-color event signal as the input of the refined fusion module.

[0060] Furthermore, in an exemplary embodiment, the event demosaicing module is obtained by training with historical mosaic color event signals and historical blurred images as input samples and supervised full-color event signals as output samples, wherein a first supervised attention layer is provided after the feature fusion layer, and the first supervised attention layer is used to learn the data association relationship between the output result of the feature fusion layer and the supervised full-color event signal. In the training process of the event demosaicing module, this embodiment uses the supervised attention module (Supervised Attention Module, SAM) provided after the feature fusion layer to build data associations between this module and subsequent modules, thereby enhancing the quality of the final output content.

[0061] Further, combined with Figure 5 The process of training the event demosaicing module in this embodiment is explained. In the event demosaicing module, the input discrete events are first preprocessed to form an event voxel grid over a space-time neighborhood so as to be sent to the convolutional neural network model for processing. Each event voxel grid is initially processed by pixel unshuffle to convert the original single-plane mosaic event data into four color planes, namely the four channels of R / G / G / B. Subsequently, through interpolation processing, the event voxel grid is spatially upsampled to match the original resolution size. After that, the event voxel grid is formally sent to the network structure for computational processing. Figure 5 As shown in the figure, the main structure of the event demosaicing module is a backbone network built on the U-Net structure. Unlike the general U-Net, the event demosaicing model has two encoder structures, corresponding to the feature extraction of mosaic color events and traditional images respectively, and then uses the cross-attention mechanism to fuse the two modal features extracted by the two encoders to obtain the demosaiced full-color event.

[0062] Through the above embodiment, an event demosaicing module for mosaic color events is used to construct a mapping relationship between "mosaic color events-full-size color events", repair the resolution reduction of each color channel caused by spatial multiplexing, and realize demosaicing of color event signals. At the same time, the low noise and full-size resolution characteristics of traditional color images are utilized to improve the quality of color event signals realized by fusion imaging.

[0063] Further, in an exemplary embodiment, the technical solution for inputting the mosaic color event signal and the blurred image into the image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module may also include: sending the mosaic color event signal and the blurred image to the image feature encoding layer of the image deblurring module; performing feature extraction on the mosaic color event signal at the image feature encoding layer to obtain a third image feature, and performing feature extraction on the blurred image to obtain a fourth image feature; after sending the third image feature and the fourth image feature to the cross-attention layer connected to the image feature encoder, sending a fusion result output by the cross-attention layer based on the third image feature and the fourth image feature to the image feature decoding layer, and determining the decoding result of the image feature decoding layer as the deblurred intermediate image.

[0064] Optionally, in one embodiment, the working principle of the image deblurring module is further described. In order to facilitate the network to extract event signal features on different time scales, and match the event double integral model, referring to the above process, the input mosaic event can be pre-processed using Symmetric Cumulative Event Representation (SCER) in the event de-mosaicing model to form an event voxel grid on a space-time neighborhood. The event voxel grid is subjected to pixel rearrangement processing and up-sampled to the original spatial resolution by bilinear interpolation.

[0065] Next, the blurred image (for example, represented as B) is sent to the image feature encoder (corresponding to the above-mentioned image feature encoding layer), and each feature layer is connected to the corresponding layer of the image feature encoder through a cross-attention block. Then, the supervised attention module (Supervised Attention Module, SAM) set at the end of the decoder is used to better build the data association between this module and subsequent modules during the training process to enhance the quality of the final output content. The image deblurring module can learn to map the input blurred image B to the deblurred clear image (for example, represented as I), and the output of the image deblurring module is a deblurred image.

[0066] However, for the entire neural network model, the deblurred image output by the image deblurring module is only an intermediate result of deblurring. The reason why the output of the image deblurring module is called an intermediate result is that the event will not be processed by pixel rearrangement, so the event here is a single channel, and the three channels of the blurred image are deblurred using the event on this plane. In fact, the events in the image deblurring module are mainly used to provide clear edge information and texture information for image deblurring, and color restoration is not concerned for the time being. The reconstruction of color information will be the focus of the subsequent fusion module.

[0067] Furthermore, in an exemplary embodiment, the image deblurring module is obtained by training with historical mosaic color event signals and historical blurred images as input samples and supervised clear images as output samples, wherein a second supervised attention layer is provided after the image feature decoding layer, and the second supervised attention layer is used to learn the data association relationship between the output result of the image feature decoding layer and the supervised clear image. In the training process of the image deblurring module of this embodiment, the supervised attention module provided after the image feature decoding layer is used to establish the data association relationship between this module and subsequent modules, thereby improving the quality of the output result and further improving the prediction ability of the model. Figure 4 As shown, the image deblurring module has a similar network structure to the event demosaicing module, and mosaic events (corresponding to mosaic color event signals) are also input into the encoder to guide the image deblurring process.

[0068] Furthermore, in an exemplary embodiment, the historical blurred image can also be obtained in the following manner: using the image acquisition module to acquire images of a preset video played in the image acquisition area to obtain a clear image sequence, wherein the clear image sequence includes multiple frames of clear images; performing image blurring processing on each frame of the clear image to obtain a blurred processed image corresponding to each frame of the clear image, and determining the blurred processed image as the historical blurred image.

[0069] Among them, Figure 8 As shown, the process of performing image blur processing on each frame of clear image includes: interpolating and averaging the clear image frames by using the cumulative exposure averaging method to generate corresponding blurred degraded frames (corresponding blurred images) affected by motion blur.

[0070] In this embodiment, specifically, first configure Figure 6The display shown is ready for the source video content to be captured. Configure the event camera and install the filter; start the event camera signal acquisition program to make it ready; make the registration so that the display is parallel to the camera sensor and the display is located in a suitable position in the camera field of view. Then adjust the filter to red, start the event camera acquisition process, and the display starts to play the video. After the display content is played, stop the acquisition and save the red color channel event content. Then adjust the filter to green, start the event camera acquisition process, and the display starts to play the video. After the display content is played, stop the acquisition and save the green color channel event content. Then adjust the filter to blue, start the event camera acquisition process, and the display starts to play the video. After the display content is played, stop the acquisition and save the blue color channel event content. Finally, adjust the filter to the transparent glass, or remove the filter, start the event camera acquisition process, and the display starts to play the video. After the display content is played, stop the acquisition and save the monochrome color channel event content.

[0071] Then, the collected four color channel event contents are time-aligned and spatially registered so that when the contents of the four color channel events are played back simultaneously, the contents on the screen keep the pixel positions consistent at all times until the end of the playback, thereby obtaining the required full-size R / G / B color events and monochrome events.

[0072] By downsampling the full-size R / G / B color events in the Bayer array format and integrating the extraction results, the desired mosaic color events can be obtained. The content of the source video stream is split into frame images to obtain the desired clear image, and then the clear image is optionally frame-rate-enhanced, and then the adjacent frames are accumulated and averaged to achieve the purpose of extending the image exposure time. The desired blurred image can be obtained through this image blurring process. In this way, the imaging data set for subsequent training of the image processing model can be obtained based on the full-size R / G / B color events, monochrome events, clear images and blurred images.

[0073] Among them, for the process of obtaining the above-mentioned clear image, in addition to the consideration of simplicity, as mentioned above, directly selecting the video source file (i.e., source video stream) played on the display as the image content, you can also use an ordinary traditional camera to shoot the screen to obtain a traditional image (to simulate the actual imaging process).

[0074] The above-mentioned blurred image can also be obtained by processing a clear image using a multi-frame synthesized image degradation model.

[0075] It should be noted that for the same original scene, the event camera needs to shoot four times, respectively shooting the corresponding red, green, blue and monochrome events. The requirement for the same original scene is that the scene content remains exactly the same during the shooting process, and the relative motion between the camera and the scene also remains exactly the same.

[0076] Of course, it is quite difficult to shoot the same original scene multiple times, so the present invention uses convenient and controllable scene content, see Figure 6 As shown, the external display should have a higher refresh rate as much as possible to reduce the difference in time accuracy between the display content and the real world. That is, the ready-made video screen played on the display, while using the display to play the video instead of the real world scene, so as to achieve controllable collection of effective event data.

[0077] Based on the above embodiments, the user can use a sensor with a Bayer array filter to collect event signals and traditional images, use the event demosaicing module of the present invention to implement event demosaicing, and then use the full-size color events obtained after demosaicing to implement color-fidelity traditional image collaborative imaging deblurring, thereby performing image deblurring processing on images based on color event collaborative imaging, and achieving color-fidelity image deblurring.

[0078] Furthermore, in an exemplary embodiment, the mosaic color event signal is obtained in the following manner: the image acquisition module is used to perform image acquisition on the current acquisition area to obtain image signals in different color channels; the image signals in the different color channels are calibrated in the time dimension and the space dimension to obtain a full-size color event signal, and the pixels of the image signals in the different color channels have the same pixel position at the same time; the image signals of the full-size color event signal in different color channels are selectively downsampled to obtain the mosaic color event signal. Among them, the current acquisition area corresponds to the acquisition area in the real scene. This embodiment acquires a full-color image frame by using the image acquisition module to acquire the real environment, and then continues to perform de-mosaicing and de-blurring processing on the image frame to achieve a de-blurring operation on the full-color event image, thereby improving the de-blurring efficiency of the full-color event image.

[0079] Further, in an exemplary embodiment, the historical mosaic color event signal is obtained in the following manner: using the image acquisition module to perform image acquisition on the preset video played in the image acquisition area to obtain image signals in different color channels, wherein the image acquisition area represents the display area of ​​the display device; performing signal calibration on the image signals in the different color channels in the time dimension and the space dimension respectively to obtain full-size color event signals and monochrome event signals, and the pixels of the image signals in the different color channels are consistent in pixel position at the same time; selectively downsampling the image signals of the full-size color event signal in different color channels to obtain the historical mosaic color event signal. This embodiment acquires a full-color image frame by using the image acquisition module to acquire the preset video, and then continues to perform de-mosaicing and de-blurring processing on the image frame to achieve de-blurring operation on the full-color event image, thereby improving the de-blurring efficiency of the full-color event image.

[0080] In this embodiment, if Figure 7 As shown, the signal in each color channel can be selectively downsampled according to the pixel color characteristics of the color filter array (Bayer array). Specifically, the output signals of different pixel positions are extracted in different color channels to form the original signals output by the Bayer array sensor, forming a data set of the original signal of "mosaic color signal + full color signal" (i.e., the above-mentioned imaging data set).

[0081] The key to constructing this data set suitable for training image processing models is to obtain paired data such as full-size color events (or full-color events) and mosaic color events corresponding to full-size color events, so as to construct the corresponding relationship between the neural network input data (representing the above-mentioned mosaic color event signal) and the expected output result (representing the de-mosaiced full-color event signal), so that the neural network model can learn the mapping relationship between the two. However, the commonly used color event camera can only collect mosaic color event signals, and the image quality is poor (the number of pixels occupied by each color channel is only one-fourth of the original), so the present invention uses a de-mosaic algorithm to restore it to complete three-channel event information, solves the de-mosaicing problem of the color event camera, and obtains high-quality imaging results.

[0082] The present invention considers using a hardware combination of a color filter + a monochrome event camera to capture color event data. By controlling the color of the light reaching the sensor, it is possible to obtain complete event information in space for each color channel (i.e., red, green, blue, and monochrome color channels). This complete information naturally constitutes a standard supervisory signal for neural network training. Based on this complete information, Figure 7The format of the Bayer array pattern shown in the figure extracts the events of each color channel at the corresponding position, integrates the extracted event signals, and obtains the mosaic color event signals after downsampling and degradation. Using these signals as input data, we can get the corresponding mosaic events and full-color events, which meets the requirements of the task of training the demosaicing model.

[0083] On the other hand, in addition to event signals, in order to achieve the collaborative imaging task of RGB traditional images + events, the dataset also needs to include traditional images. Similarly, traditional images must also include standard signals as supervision and degraded images as input to build a relationship between the two. Moreover, traditional images must have a corresponding relationship with the aforementioned event content.

[0084] Therefore, the imaging data set provided in this embodiment has the function of providing input data for the image processing model, aiming to provide data support for the data-driven image processing model (including the de-mosaic module, the image deblurring module and the fusion module), which may include but is not limited to: (1) mosaic color event signals, (2) full-size color event signals, (3) monochrome event signals, (4) blurred images, (5) clear images (i.e. the clear full-color images mentioned above). But in fact, on the other hand, the imaging data set can also provide supervision signals to the neural network structure in the image processing model. Figure 3 As shown in the figure, by using the Supervised Attention Module (SAM) and taking clear full-color images as supervisory signals, the data association between this module and subsequent modules can be better constructed during the training process, thereby enhancing the quality of the final output content.

[0085] Through the above embodiments, an imaging system of "display-filter-event camera" is proposed, which uses a time multiplexing method to record full-size color events, and also uses a space multiplexing method to construct mosaic color events, generating an imaging data set corresponding to the color event signal, which not only reduces the loss of image signals in the spatial dimension, but also reduces the gap between the simulator output data and the real data, realizes the synchronous recording of mosaic events and full-size color events, and provides support for the data-driven method.

[0086] Further, in an exemplary embodiment, the process of using the image acquisition module to acquire an image of a preset video played in an image acquisition area to obtain image signals in different color channels is implemented by the following steps: using image acquisition modules corresponding to N different filters to acquire an image of the preset video played in the image acquisition area to obtain N image signals, wherein the filter types of the N different filters are different, and each filter type corresponds to a color channel; determining the image signals in the different color channels based on the N image signals, wherein the N image signals include at least one of the following: a first image signal acquired through a red channel, a second image signal acquired through a green channel, a third image signal acquired through a blue channel, and a fourth image signal acquired through a monochrome channel.

[0087] Optionally, in this embodiment, a "display-event camera" imaging device is proposed to collect images. The imaging device can be understood as the above-mentioned image acquisition module. The imaging device includes a high refresh rate display, a filter holder equipped with RGB three-color filters, and a monochrome event camera.

[0088] Among them, the high refresh rate display is used to repeatedly play high frame rate videos to reproduce real-world scenes. The present invention uses ASUS ROG PG259QNR, a high refresh rate display with a refresh rate of 360Hz and a spatial resolution of 1920×1080 pixels, to obtain red, green, blue, and monochrome event signals with exactly the same viewing angle and content by looping video content.

[0089] Among them, the color filter is responsible for encoding the color information in the scene into the incident light. The present invention selects a filter fixing wheel as the RGB three-color filter and its fixing device, and changes the color of the acquired event signal by switching the filter type. Since the filter is installed in front of the lens, all the light reaching the sensor will be filtered, and a full-size color event signal can be obtained.

[0090] The monochrome event camera (abbreviated as monochrome camera) collects the light intensity changes of the scene after filtering and generates corresponding events. Among them, for the images taken by the monochrome camera (black and white camera), the image can be converted into a specific monochrome effect by post-processing. The present invention uses the DAVIS 346 model monochrome event camera, which can be used to actually shoot the event content. The monochrome event camera senses the changes in light intensity within the entire visible light wavelength range. Therefore, the type of light color collected by the monochrome camera is determined by the filter set in front of it.

[0091] This embodiment can select different filter combinations according to different needs. By using filters of different wavelengths, spectral information that is invisible to the human eye can be captured, thereby obtaining richer color data for imaging, which helps to improve image quality and improve the processing capabilities of image processing such as image segmentation, target recognition and feature extraction.

[0092] Among them, the filter type corresponding to the red channel is a red filter, the filter type corresponding to the green channel is a green filter signal, and the filter type corresponding to the blue channel is a blue filter.

[0093] Optionally, in this embodiment, if Figure 7 As shown, the filter type may include, for example, a red filter, a green filter, a blue filter, etc. In the present invention, when shooting, first switch a type of filter for the camera, then play the video content on the display, and the acquisition begins. After the content is played, stop the acquisition and switch to another filter. Repeat the above process until all events of the four color types of red, green, blue, and monochrome are acquired.

[0094] It should be noted that a monochrome image usually refers to an image containing only one color, and red, green, and blue are the three primary colors in the RGB (Red, Green, Blue) color model.

[0095] The process of using filters to capture monochrome images and red, green, and blue images is further explained:

[0096] For monochrome images, you can add transparent glass instead of filters to produce monochrome black and white images.

[0097] It should be noted that the "monochromatic" involved in the present invention is a direct translation of the English word "monochromatic", and its actual meaning is a "black and white" image or a "grayscale" image. Therefore, for the shooting of a monochrome image, the color of the light cannot be changed, otherwise it will not be a "monochromatic" image.

[0098] For red images, a red filter is used to capture a red image. This filter only allows red light to pass through and absorbs light of other colors.

[0099] Similarly, a green filter allows you to capture a green image. This filter only allows green light to pass through. And a blue filter allows you to capture a blue image. This filter only allows blue light to pass through.

[0100] In certain photography techniques such as High Dynamic Range (HDR) or multiple exposures, RGB spectroscopic filters can also be used to capture images of the red, green, and blue channels separately, and then the final color image is obtained through post-synthesis.

[0101] Optionally, for a color image, it can be converted into a black and white image by post-processing, retaining only brightness information and not retaining color information. The post-processing may include adjusting the color channel in the image editing software to retain only one color channel, thereby obtaining a monochrome image.

[0102] Furthermore, in an exemplary embodiment, a technical solution for inputting the demosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model to obtain a deblurred image output by the fusion module is described, specifically including: using the fusion module to extract the target event signal feature of the demosaiced color event signal, and using the fusion module to extract the fifth image feature of the deblurred intermediate image; obtaining the similarity between the fifth image feature and the target event signal feature; normalizing the similarity to obtain a normalized similarity, and setting the normalized similarity as a weight coefficient of the target image feature; using the product of the fifth image feature and the weight coefficient to update the fifth image feature to obtain a sixth image feature, and obtaining the deblurred image output by the fusion module after feature reconstruction of the sixth image feature.

[0103] Optionally, the fusion module includes, for example, a convolution layer, and using the fusion module to extract the target event signal feature of the demosaiced color event signal can be understood as using the convolution layer of the fusion module to extract the feature of the demosaiced color event signal to obtain the target event signal feature. Using the fusion module to extract the fifth image feature of the deblurred intermediate image can be understood as using the convolution layer of the fusion module to extract the feature of the deblurred intermediate image to obtain the fifth image feature.

[0104] In this embodiment, the process of obtaining the similarity between the fifth image feature and the target event signal feature may be implemented by a similarity measurement algorithm, and the similarity measurement algorithm includes but is not limited to a dot product, a scaled dot product, and the like.

[0105] In this embodiment, the process of reconstructing the sixth image feature may be implemented by, for example, using inverse transformation techniques such as inverse convolution or generative adversarial networks. Alternatively, the image corresponding to the sixth image feature may be reconstructed by autoencoders or other generative models.

[0106] Optionally, the sixth image feature may be projected back to the dimension of the original image using a mapping algorithm such as interpolation or resampling, and the difference before and after mapping may be compared to facilitate subsequent analysis.

[0107] Optionally, heat maps or feature maps may be used to display the sixth image feature.

[0108] Optionally, dimensionality reduction techniques such as principal component analysis (PCA) or t-SNE may be used to reduce the dimensionality of the sixth image features for easier visualization and understanding.

[0109] Optionally, filtering, sharpening or other image enhancement techniques may be used to enhance key information of the sixth image feature.

[0110] Optionally, the sixth image feature may be subjected to denoising, sharpening, smoothing and normalization operations to remove noise or unimportant information in the sixth image feature, and the most useful feature may be selected, thereby improving the utilization rate of the feature.

[0111] Furthermore, in an exemplary embodiment, the fusion module is obtained by training the historical demosaiced color event signal and the historical blurred intermediate image as input samples and the supervised clear image as output samples, wherein a third supervised attention layer is provided in the fusion module, and the third supervised attention layer is used to learn the data association relationship between the sixth image feature and the supervised clear image. It should be noted that the fusion module also needs to use the cross attention mechanism to realize feature fusion, so the third supervised attention layer is provided. Figure 3 The cross attention module shown in the figure is different from the event demosaicing module in that the attention calculation here is based on the image feature as the query and the event feature as the key and value. In addition, the supervised attention module (corresponding to the third supervised attention layer) is also set in the fusion module.

[0112] Among them, the third supervised attention layer is implemented based on the Cross-Attention (CA) mechanism. The Cross-Attention mechanism allows a model to refer to the information of another input (such as event signal features) when processing one input (such as image features). The fusion module uses the Cross-Attention mechanism to calculate attention with image features as query and event signal features as key and value, so that the process of feature fusion can be understood as follows:

[0113] The query represents the target input for obtaining information, and the key and value represent the source of information that you want to refer to. Therefore, the image features are defined as queries, and the event signal features are defined as key-value pairs. The similarity between the query and the key is calculated, and then this similarity is used to obtain the weighted value.

[0114] Specifically, the similarity score between the query and each key is calculated, and the similarity score is normalized by the softmax function so that the sum of all similarity scores is 1. Among them, each similarity score represents the importance of the query relative to each key.

[0115] Then, the normalized similarity scores are used as weights to perform weighted summation on the corresponding values. In this way, the contribution of each value is proportional to the similarity between its corresponding key and the query. Finally, the result is weighted based on the original value according to the similarity between the query and the key. This result can be regarded as the fusion result of image features and event features. By reconstructing the features of this fusion result, the image can be reconstructed.

[0116] Alternatively, the fusion result can be used as the input of the next layer of the network for further application in tasks such as classification and detection in image processing, computer vision or machine learning.

[0117] Through the above process, the attention level of event features is dynamically adjusted according to the image content based on the cross-attention mechanism, thereby achieving more effective feature fusion. This mechanism enables the fusion module to better understand the correlation between different modalities and improves the feature fusion effect of the fusion module.

[0118] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. In order to better understand the above-mentioned image processing method, the above-mentioned process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present invention. Specifically:

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0120] In this embodiment, an image processing system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0121] Fig. 9 is a structural block diagram of an image processing system according to an embodiment of the present invention, wherein the image processing system and the image acquisition module ( Fig. 9 There is a data transmission channel between the image acquisition module and the image processing system (not shown), and the data transmission channel is used to transmit image data between the image acquisition module and the image processing system. The image processing system is at least pre-installed with an image processing model 92, and the image processing model 92 includes an event demosaicing module and an image deblurring module.

[0122] The image processing system is used to obtain a mosaic color event signal and a blurred image from the image acquisition module; input the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module; and input the mosaic color event signal and the blurred image into the image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module; input the demosaiced full-color event signal and the deblurred intermediate image into the fusion module of the image processing model to obtain a deblurred image output by the fusion module.

[0123] Through the present invention, an image processing model for collaborative imaging using color events is designed in an image processing system. On the basis of realizing de-mosaicing, full-size color event signals are used and color event signal fusion imaging is used to perform motion blur removal operations on color traditional images. Specifically, a mosaic color event signal and a blurred image are obtained from the image acquisition module; the mosaic color event signal and the blurred image are input into the event de-mosaic module of the image processing model to obtain a de-mosaiced full-color event signal output by the event de-mosaic module; and the mosaic color event signal and the blurred image are input into the image de-blurring module of the image processing model to obtain a de-blurred intermediate image output by the image de-blurring module; the de-mosaiced full-color event signal and the de-blurred intermediate image are input into the fusion module of the image processing model to obtain a de-blurred image output by the fusion module. Therefore, the technical solution of the present invention solves the problem that in a color imaging scenario, color events cannot be used for collaborative imaging, and that mosaic color events cannot be demosaiced. It not only realizes collaborative imaging using color events, but also can realize demosaicing for mosaic color events, thereby achieving color-fidelity image deblurring and improving the restoration effect of traditional color-fidelity images.

[0124] Further, in an exemplary embodiment, the image processing system is also used for: Further, in an exemplary embodiment, the mosaic color event signal and the blurred image are input into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module, including: inputting the mosaic color event signal into the feature extraction layer of the event demosaicing module to obtain a first image feature, and inputting the blurred image into the feature extraction layer of the event demosaicing module to obtain a second image feature; sending the first image feature and the second image feature to the feature fusion layer of the event demosaicing module to obtain a demosaiced full-color event signal output by the feature fusion layer.

[0125] Furthermore, in an exemplary embodiment, the image processing system is also used to: use historical mosaic color event signals and historical blurred images as input samples and supervised full-color event signals as output samples to train the event demosaicing module, wherein a first supervised attention layer is provided after the feature fusion layer, and the first supervised attention layer is used to learn the data association relationship between the output result of the feature fusion layer and the supervised full-color event signal.

[0126] Furthermore, in an exemplary embodiment, the image processing system is also used to: send the mosaic color event signal and the blurred image to the image feature encoding layer of the image deblurring module; perform feature extraction on the mosaic color event signal at the image feature encoding layer to obtain a third image feature, and perform feature extraction on the blurred image to obtain a fourth image feature; after sending the third image feature and the fourth image feature to the cross-attention layer connected to the image feature encoder, send a fusion result output by the cross-attention layer based on the third image feature and the fourth image feature to the image feature decoding layer, and determine the decoding result of the image feature decoding layer as the deblurred intermediate image.

[0127] Furthermore, in an exemplary embodiment, the image processing system is also used to: use historical mosaic color event signals and historical blurred images as input samples and supervised clear images as output samples to train the image deblurring module, wherein a second supervised attention layer is provided after the image feature decoding layer, and the second supervised attention layer is used to learn the data association relationship between the output result of the image feature decoding layer and the supervised clear image.

[0128] Furthermore, in an exemplary embodiment, the image processing system is also used to: obtain the historical blurred image in the following manner: use the image acquisition module to perform image acquisition on a preset video played in an image acquisition area to obtain a clear image sequence, wherein the clear image sequence includes multiple frames of clear images; perform image blurring on each frame of the clear image to obtain a blurred processed image corresponding to each frame of the clear image, and determine the blurred processed image as the historical blurred image.

[0129] Furthermore, in an exemplary embodiment, the image processing system is also used to: obtain the mosaic color event signal in the following manner: use the image acquisition module to perform image acquisition on the current acquisition area to obtain image signals in different color channels; perform signal calibration on the image signals in different color channels in the time dimension and the space dimension respectively to obtain a full-size color event signal, and the pixels of the image signals in the different color channels have the same pixel position at the same time; selectively downsample the image signals of the full-size color event signal in different color channels to obtain the mosaic color event signal.

[0130] Furthermore, in an exemplary embodiment, the image processing system is also used to: use image acquisition modules corresponding to N different filters to respectively acquire images of a preset video played in the image acquisition area to obtain N image signals, wherein the filter types of the N different filters are different, and each filter type corresponds to a color channel; determine the image signals in the different color channels based on the N image signals, wherein the N image signals include at least one of the following: a first image signal acquired through a red channel, a second image signal acquired through a green channel, a third image signal acquired through a blue channel, and a fourth image signal acquired through a monochrome channel.

[0131] Furthermore, in an exemplary embodiment, the image processing system is also used to: use the fusion module to extract the target event signal feature of the de-mosaiced color event signal, and use the fusion module to extract the fifth image feature of the deblurred intermediate image; obtain the similarity between the fifth image feature and the target event signal feature; normalize the similarity to obtain a normalized similarity, and set the normalized similarity as a weight coefficient of the target image feature; use the product of the fifth image feature and the weight coefficient to update the fifth image feature to obtain a sixth image feature, and obtain the deblurred image output after the fusion module performs feature reconstruction on the sixth image feature.

[0132] Furthermore, in an exemplary embodiment, the image processing system is also used to: use historical de-mosaiced color event signals and historical blurred intermediate images as input samples and supervised clear images as output samples to train the fusion module, wherein a third supervised attention layer is provided in the fusion module, and the third supervised attention layer is used to learn the data association relationship between the sixth image feature and the supervised clear image.

[0133] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0134] S1, acquiring a mosaic color event signal and a blurred image from the image acquisition module;

[0135] S2, inputting the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module;

[0136] S3, and inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module;

[0137] S4, inputting the de-mosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model, and obtaining a deblurred image output by the fusion module.

[0138] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0139] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0140] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0141] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0142] S1, acquiring a mosaic color event signal and a blurred image from the image acquisition module;

[0143] S2, inputting the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module;

[0144] S3, and inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module;

[0145] S4, inputting the de-mosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model, and obtaining a deblurred image output by the fusion module.

[0146] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0147] An embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0148] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0149] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0150] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0151] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: Applied to an image processing system, a data transmission channel is provided between the image processing system and the image acquisition module, and the image processing system is at least pre-installed with an image processing model, including: Acquiring mosaic color event signals and blurred images from the image acquisition module; Inputting the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain a demosaiced full-color event signal output by the event demosaicing module; and inputting the mosaic color event signal and the blurred image into an image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module; The de-mosaiced full-color event signal and the deblurred intermediate image are input into a fusion module of the image processing model to obtain a deblurred image output by the fusion module.

2. The image processing method according to claim 1, characterized in that: Inputting the mosaic color event signal and the blurred image into the event demosaicing module of the image processing model to obtain a demosaic full-color event signal output by the event demosaicing module, comprising: Inputting the mosaic color event signal into the feature extraction layer of the event demosaicing module to obtain a first image feature, and inputting the blurred image into the feature extraction layer of the event demosaicing module to obtain a second image feature; The first image feature and the second image feature are sent to the feature fusion layer of the event demosaicing module, and a demosaiced full-color event signal output by the feature fusion layer is obtained.

3. The image processing method according to claim 2, characterized in that: The method further comprises: The event demosaicing module is obtained by training with historical mosaic color event signals and historical blurred images as input samples and supervised full-color event signals as output samples, wherein a first supervised attention layer is arranged after the feature fusion layer, and the first supervised attention layer is used to learn the data association relationship between the output result of the feature fusion layer and the supervised full-color event signal.

4. The image processing method according to claim 1, characterized in that: Inputting the mosaic color event signal and the blurred image into the image deblurring module of the image processing model to obtain a deblurred intermediate image output by the image deblurring module, comprising: sending the mosaic color event signal and the blurred image to an image feature coding layer of the image deblurring module; Performing feature extraction on the mosaic color event signal at the image feature coding layer to obtain a third image feature, and performing feature extraction on the blurred image to obtain a fourth image feature; After the third image feature and the fourth image feature are sent to the cross-attention layer connected to the image feature encoder, a fusion result output by the cross-attention layer based on the third image feature and the fourth image feature is sent to an image feature decoding layer, and a decoding result of the image feature decoding layer is determined as the deblurred intermediate image.

5. The image processing method according to claim 2, characterized in that: The method further comprises: The image deblurring module is obtained by training with historical mosaic color event signals and historical blurred images as input samples and supervised clear images as output samples, wherein a second supervised attention layer is arranged after the image feature decoding layer, and the second supervised attention layer is used to learn the data association relationship between the output result of the image feature decoding layer and the supervised clear image.

6. The image processing method according to any one of claims 2 to 5, characterized in that: The historical blurred image is obtained by: The image acquisition module is used to acquire images of a preset video played in an image acquisition area to obtain a clear image sequence, wherein the clear image sequence includes multiple frames of clear images; image blurring is performed on each frame of the clear image to obtain a blurred processed image corresponding to each frame of the clear image, and the blurred processed image is determined as the historical blurred image.

7. The image processing method according to any one of claims 2 to 5, characterized in that: The mosaic color event signal is obtained by: Use the image acquisition module to acquire images of the current acquisition area to obtain image signals in different color channels; Performing signal calibration on the image signals in the different color channels in the time dimension and the space dimension respectively to obtain full-size color event signals, wherein the pixels of the image signals in the different color channels have the same pixel position at the same time; The image signals of the full-size color event signal in different color channels are selectively down-sampled to obtain the mosaic color event signal.

8. The image processing method according to claim 7, characterized in that: The image acquisition module is used to acquire images of a preset video played in the image acquisition area to obtain image signals in different color channels, including: Using image acquisition modules corresponding to N different filters to acquire images of a preset video played in the image acquisition area, respectively, to obtain N image signals, wherein the filter types of the N different filters are different, and each filter type corresponds to a color channel; The image signals in the different color channels are determined based on the N image signals, wherein the N image signals include at least one of the following: a first image signal acquired through a red channel, a second image signal acquired through a green channel, a third image signal acquired through a blue channel, and a fourth image signal acquired through a monochrome channel.

9. The image processing method according to claim 1, characterized in that: Inputting the de-mosaiced full-color event signal and the deblurred intermediate image into a fusion module of the image processing model, and obtaining a deblurred image output by the fusion module, comprising: extracting a target event signal feature of the demosaiced color event signal using the fusion module, and extracting a fifth image feature of the deblurred intermediate image using the fusion module; Acquire the similarity between the fifth image feature and the target event signal feature; Normalizing the similarity to obtain a normalized similarity, and setting the normalized similarity as a weight coefficient of the target image feature; The fifth image feature is updated using the product of the fifth image feature and the weight coefficient to obtain a sixth image feature, and a deblurred image output after the fusion module performs feature reconstruction on the sixth image feature is obtained.

10. The image processing method according to claim 9, characterized in that: The method further comprises: The fusion module is obtained by training with historical de-mosaiced color event signals and historical blurred intermediate images as input samples and supervised clear images as output samples, wherein a third supervised attention layer is provided in the fusion module, and the third supervised attention layer is used to learn the data association relationship between the sixth image feature and the supervised clear image.

11. An image processing system, characterized in that: include: The image processing system is pre-installed with at least an image processing model, and the image processing system is used to execute the method described in any one of claims 1 to 10 above.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 10 when executed.

13. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method described in any one of claims 1 to 10 is implemented.

Images