Image demosaicing method and system

Through the two-stage demosaic model of deep neural network, the images of Quad Bayer CFA and hybrid event cameras are processed, and the problems of complex arrangement and event pixel color deficiency information are solved, achieving high-quality RGB image reconstruction and optimization of computing resources.

CN118488322BActive Publication Date: 2024-11-01HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410951152.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-01
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Quad Bayer CFA faces the problem of complex arrangement methods and lack of color information during the demosaic reconstruction process, which leads to a degradation of reconstruction quality and traditional methods are difficult to effectively deal with the mosaic and noise of mixed event cameras.

Method used

Using a two-stage demosaic model based on deep neural networks, the first stage network denoising and repairing missing event color information, the second stage network performs demosaic processing, combined with the Quad Bayer arrangement position map and event position mask map, and use double Branch encoder and multimodal attention mechanism optimize the encoding of color and position features.

Benefits of technology

It realizes high-quality RGB image reconstruction, reduces network parameters and computing volume, is suitable for mobile devices, and significantly improves image quality and computing resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118488322B_ABST
    Figure CN118488322B_ABST
Patent Text Reader

Abstract

The present invention relates to an image demosaicing method and system. The method comprises: obtaining a mosaic image including RGB color information collected by a hybrid event image sensor; obtaining a Quad Bayer arrangement position map and an event position mask map according to the mosaic image; and obtaining a demosaiced reconstructed image by processing the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage demosaicing model based on a deep neural network. In the above manner, color information and position information can be considered at the same time, and a high-quality demosaicing effect can be achieved by jointly encoding the two, so as to reconstruct a high-quality demosaiced RGB image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an image de-mosaicing method and system, and belongs to the field of image processing and signal transmission. Background Art

[0002] De-mosaicing is a technique in the field of image processing that involves converting the raw data collected by the image sensor into a high-quality color image. These raw data are usually collected by the color filter array (CFA) on the image sensor, and the CFA can only capture the brightness or monochrome information of the image, not the complete color information. Since the image data sampled by the CFA is incomplete in color, the purpose of the de-mosaicing algorithm is to restore the complete RGB color information of each pixel through interpolation technology.

[0003] The application of traditional Bayer CFA sensors on mobile devices is limited by their design, and it is difficult to obtain high-quality images under low-light conditions. To solve this problem, Non-Bayer CFA sensors came into being, and the imaging performance in low-light environments was improved through a specially designed CFA layout. Among them, Quad Bayer CFA has become one of the mainstream technologies in the field of mobile photography in recent years because it can obtain high-resolution images under normal lighting conditions and enhance low-light imaging through pixel merging technology. Although Quad Bayer CFA technology has made progress in improving imaging quality, it still faces challenges in the demosaicing and reconstruction process. Compared with Bayer CFA, Quad Bayer CFA has a more complex arrangement and lacks color information at the event pixel position, which brings difficulties to the design of demosaicing algorithms. When dealing with such a complex arrangement, traditional demosaicing methods have difficulty in effectively extracting image patterns, resulting in a decrease in reconstruction quality. At the same time, with the rapid development of image sensor technology, event cameras (Event Cameras), as a new type of image sensor, have shown great potential in the field of dynamic visual capture. Compared with traditional digital cameras, event cameras capture event information by sensing pixel intensity changes, thereby accurately recording the movement trajectory of objects. This unique way of capturing visual data makes event cameras have broad application prospects in multiple high-speed dynamic scenes such as robot vision, self-driving cars, and drone monitoring.

[0004] A hybrid event camera uses a Quad Bayer CFA on the camera sensor and designates certain pixels as event pixels to capture motion data instead of RGB color information. This design introduces a camera with improved low-light imaging capabilities and the ability to effectively capture high-speed objects. However, the non-traditional four-pixel Bayer arrangement and the lack of color information at the event pixel position pose challenges to the demosaicing reconstruction process. Specifically: i) how to mitigate the degradation of reconstruction quality caused by the lack of color information at the event pixel position; ii) how to achieve joint processing of demosaicing and denoising for hybrid event cameras; iii) how to reduce the parameters of the model to make it valuable in practical applications of edge computing. Summary of the invention

[0005] The present invention provides an image de-mosaicing method and system, aiming to solve at least one of the technical problems existing in the prior art.

[0006] The technical solution of the present invention relates to an image demosaicing method. The method according to the present invention includes: obtaining a mosaic image including RGB color information collected by a hybrid event image sensor; obtaining a Quad Bayer arrangement position map and an event position mask map according to the mosaic image; and obtaining a demosaiced reconstructed image by processing the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage demosaicing model based on a deep neural network.

[0007] Furthermore, the two-stage demosaicing model includes a first-stage network and a second-stage network. The mosaic image, the Quad Bayer arrangement position map and the event position mask map are processed by the two-stage demosaicing model based on a deep neural network to obtain a demosaiced reconstructed image, including: applying the first-stage network to denoise and repair missing event color information of the mosaic image, the Quad Bayer arrangement position map and the event position mask map according to the input basis to obtain a pure mosaic image; applying the second-stage network to perform demosaicing according to the Quad Bayer arrangement position map and the pure mosaic image to obtain a demosaiced reconstructed image.

[0008] Further, the first-stage network is applied to denoise and repair missing event color information of the mosaic image, the QuadBayer arrangement position map and the event position mask map according to the input, so as to obtain a pure mosaic image, including: applying a repair layer to roughly repair missing event color information of the input mosaic image; applying a first Fourier feature layer and a first-stage position extraction layer group to extract a first position feature according to the Quad Bayer arrangement position map and the event position mask map; applying multiple first-stage color extraction layer groups to extract a first color feature and a first fusion feature according to the output of the repair layer and the first position feature; applying multiple first-stage intermediate providing layer groups and multiple first-stage reconstruction layer groups to process according to the first color feature and the first fusion feature to obtain a pure mosaic image.

[0009] Furthermore, the second-stage network is applied to perform de-mosaicing processing according to the Quad Bayer arrangement position map and the pure mosaic image to obtain a reconstructed image after de-mosaicing, including: applying a second Fourier feature layer and a two-stage position extraction layer group to extract a second position feature according to the Quad Bayer arrangement position map and the event position mask map; applying multiple two-stage color extraction layer groups to extract a second color information feature and a second fusion feature according to the pure mosaic image and the second position feature; applying multiple two-stage intermediate providing layer groups and multiple two-stage reconstruction layer groups to perform processing according to the second color information feature and the second fusion feature to obtain a reconstructed image after de-mosaicing.

[0010] Furthermore, according to applying the two-stage demosaicing model based on a deep neural network to process the mosaic image, the Quad Bayer arrangement position map and the event position mask map, before obtaining the reconstructed image after demosaicing, the two-stage demosaicing model is trained according to the training data including the historical mosaic image and the original image, including: generating a historical pure mosaic image without noise and missing event color information according to the original image; obtaining a historical Quad Bayer arrangement position map and a historical event position mask map according to the historical mosaic image; using the image pair consisting of the historical mosaic image and the historical pure mosaic image, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as the position branch input, to train the first stage network; using the image pair consisting of the historical pure mosaic image and the original image, combined with the historical Quad Bayer arrangement position map as the position branch input, to train the second stage network; using the image pair consisting of the historical mosaic image and the original image, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as the position branch input, to train the second stage network; Bayer arrangement position map, jointly training the two-stage demosaicing model composed of the first-stage network and the second-stage network.

[0011] Furthermore, the image pair consisting of the historical mosaic image and the historical pure mosaic image is used, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as position branch inputs, to train the first-stage network, including: applying the first-stage network to denoise and repair missing event color information according to the historical mosaic image, the historical event position mask map and the historical Quad Bayer arrangement position map to obtain a reconstructed mosaic image; and training the first-stage network according to the error between the reconstructed mosaic image and the historical pure mosaic image.

[0012] Further, the image pair consisting of the historical pure mosaic image and the original image is used in combination with the historical Quad Bayer arrangement position map as the position branch input to train the second stage network, including: applying the second stage network to perform de-mosaicing according to the historical pure mosaic image and the Quad Bayer arrangement position map to obtain a de-mosaiced historical reconstructed image; training the second stage network according to the error between the historical reconstructed image and the original image

[0013] The technical solution of the present invention also relates to an image demosaicing system, comprising: an image acquisition unit, used to acquire a mosaic image including RGB color information collected by a hybrid event image sensor; a position branch acquisition unit, used to acquire a Quad Bayer arrangement position map and an event position mask map according to the mosaic image; and a demosaicing unit, used to process the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage demosaicing model based on a deep neural network to acquire a demosaiced reconstructed image.

[0014] The technical solution of the present invention also relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the program.

[0015] The technical solution of the present invention also relates to a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute the aforementioned method.

[0016] The beneficial effects of the present invention are as follows:

[0017] The present invention discloses an image demosaicing method and system, which proposes to use a two-stage method to demosaic a mixed event camera sensor, and achieves excellent performance while reducing computing resources through customized subtask allocation and dual-branch encoders. The model simultaneously considers color features and position features, achieves high-quality demosaicing effects through the joint encoding of the two, reconstructs high-quality RGB images, adopts the design of a dual-branch encoder, and uses a multimodal attention mechanism to enhance the detail color restoration effect, and has a local-global attention acquisition mechanism. Targeted networks can effectively reduce network parameters and computational complexity, greatly save computing resources, and achieve the effect of lightweight deployment on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a flow chart of an image demosaicing method in an embodiment of the present invention is shown;

[0019] Figure 2 A processing schematic diagram of a two-stage demosaicing model in an embodiment of the present invention is shown;

[0020] Figure 3 A schematic diagram of the structure of a two-stage demosaicing model in an embodiment of the present invention is shown;

[0021] Figure 4 A schematic diagram of the structure of the Fourier feature layer in the two-stage demosaicing model in an embodiment of the present invention is shown;

[0022] Figure 5A schematic diagram showing the structure of a spatial position attention module in a two-stage demosaicing model in an embodiment of the present invention is shown;

[0023] Figure 6 A schematic diagram of the structure of a cross-window attention module in a two-stage demosaicing model in an embodiment of the present invention is shown;

[0024] Figure 7 A schematic diagram of the structure of the first dual-head feature extraction layer in the two-stage demosaicing model in an embodiment of the present invention is shown;

[0025] Figure 8 A schematic diagram of the structure of the second double-head feature extraction layer in the two-stage demosaicing model in an embodiment of the present invention is shown;

[0026] Fig. 9 A schematic diagram of the process of progressive training of a two-stage demosaicing model in an embodiment of the present invention is shown;

[0027] Fig.10 A schematic structural diagram of an image demosaicing system in an embodiment of the present invention is shown;

[0028] Fig.11 A schematic diagram of an electronic device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0030] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0031] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0032] In some embodiments, the present invention provides an image demosaicing method, such as Figure 1 As shown, at least the following steps are included:

[0033] S11: Acquire a mosaic image including RGB color information collected by a mixed event image sensor.

[0034] The hybrid event image sensor is used to take pictures to obtain a mosaic image, which includes RGB color information collected by the hybrid event image sensor.

[0035] S12: Acquire a Quad Bayer arrangement position map and an event position mask map according to the mosaic image.

[0036] Optionally, get the original color data Generate the corresponding Quad Bayer arrangement position map Its mathematical expression is:

[0037]

[0038] Where (x, y) is the position of each pixel in the image, M Qr (x,y),M Qg (x,y),M Qb (x, y) are the masks of the three channels R, G, and B respectively. Concatenate is to merge the three sub-mask maps into the final Quad Bayer arrangement position map along the channel dimension C. And the event position mask map Its mathematical expression is:

[0039]

[0040] In this way, the event position mask map M is obtained E and Quad Bayer arrangement position diagram M Q .

[0041] S13: Processing the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage de-mosaicing model based on a deep neural network to obtain a demosaiced reconstructed image.

[0042] See also Figure 2 This application uses the Quad Bayer arrangement position map and the event position mask map as the position branch input, applies a two-stage demosaicing model based on a deep neural network to demosaic the mosaic image, and obtains the reconstructed image after demosaicing. Figure 2 In the above, the R mask, G mask and B mask are combined into a Quad Bayer arrangement position map, and the original color data is the mosaic image. After being processed by the two-stage demosaicing model, the demosaiced RGB image is output. The two-stage demosaicing model based on the deep neural network considers both color information and position information at the same time, and achieves high-quality demosaicing effect through the joint encoding of the two, and reconstructs the high-quality demosaiced RGB image.

[0043] In the embodiment of the present application, the two-stage de-mosaicing model includes a first-stage network and a second-stage network. In step S13, the first-stage network is first applied to denoise and repair missing event color information of the mosaic image, the QuadBayer arrangement position map and the event position mask map according to the input, so as to obtain a pure mosaic image. Optionally, see Figure 3 , apply a repair layer to roughly repair the missing event color information of the input mosaic image; apply a first Fourier feature layer and a one-stage position extraction layer group to extract a first position feature according to the Quad Bayer arrangement position map and the event position mask map; apply multiple one-stage color extraction layer groups to extract a first color feature and a first fusion feature according to the output of the repair layer and the first position feature; apply multiple one-stage intermediate providing layer groups and multiple one-stage reconstruction layer groups to process according to the first color feature and the first fusion feature to obtain a pure mosaic image.

[0044] The first stage network includes: a patching layer for roughly patching the missing event color information, a first Fourier feature layer for position information extraction and a one-stage position extraction layer group, and an N 1 One-stage color extraction layer groups, N for image reconstruction 1 a first-stage reconstruction layer group, and N first-stage color information extraction layer groups disposed between the first-stage reconstruction layer group and the first-stage reconstruction layer group; 1 The repair layer is calculated as follows:

[0045]

[0046] Where I is the input mosaic image, c is the channel index, i and j are the row and column indices respectively, and I′ c,i,j is the pixel value of the event pixel, and neighbors is I′ c,i,j The surrounding pixels, including the pixels above, to the left, and in the upper left corner, d i,k and d j,k are the row and column offsets, respectively. For example, for the pixel above, d i,k = -1 and d j,k =0.

[0047] Fourier feature layer Figure 4 As shown, the calculation is as follows:

[0048]

[0049] Where v is the input vector, b is the Fourier basis frequency, and m is the sampling frequency.

[0050] The first-stage color extraction layer group includes the second convolution layer, the first-stage information fusion module and the downsampling layer. The first-stage information fusion module is connected to the first-stage position extraction layer group, including the third convolution layer and the first cross attention layer. The first-stage information fusion module is called the spatial position attention module (SPA), referring to Figure 5 ,The position feature is first standardized and then input into the convolution layer, and then activated by the ReLu function. The color feature is first standardized and then passed through the convolution layer. The processed position feature and color feature use element-by-element multiplication, and the result is input into the convolution layer. The final result is residually connected and added to the input color feature, and finally the fusion feature is output. The first-stage reconstruction layer group includes the upsampling layer, the fourth convolution layer, and the first-stage intermediate extraction layer group includes the fifth convolution layer.

[0051] Then, the second-stage network is applied to perform demosaicing according to the Quad Bayer arrangement position map and the pure mosaic image to obtain a demosaiced reconstructed image. Figure 3 , applying a second Fourier feature layer and a two-stage position extraction layer group to extract a second position feature according to the Quad Bayer arrangement position map and the event position mask map; applying multiple two-stage color extraction layer groups to extract a second color information feature and a second fusion feature according to the pure mosaic image and the second position feature; applying multiple two-stage intermediate providing layer groups and multiple two-stage reconstruction layer groups to process according to the second color information feature and the second fusion feature to obtain a demosaiced reconstructed image.

[0052] The second stage network includes: a second Fourier feature layer for position information extraction and a two-stage position extraction layer group, N 2 Two-stage color extraction layer groups, N for image reconstruction 2 A two-stage reconstruction layer group, and N two-stage color information extraction layer groups disposed between the two-stage reconstruction layer group and the two-stage reconstruction layer group 2 The second-stage intermediate extraction layer group includes the sixth convolution layer and the downsampling layer. The second-stage color extraction layer group includes the first dual-head feature extraction layer and the downsampling layer. The first dual-head feature extraction layer contains the second-stage information fusion module and the seventh convolution layer. The second-stage information fusion module is called Cross Swin Attention (CSA), refer to Figure 6 First, there is the color feature and location features The two features are split into non-overlapping M×M windows, resulting in a window of the form Then, for each feature from F I Features Features from FP Perform the following calculations:

[0053] Q=YP Q , K = XP K , V = XP V ,

[0054] In the formula, are the corresponding linear layer weights, and then, for the obtained Perform attention calculation:

[0055]

[0056] In the formula, B is the relative position code within the window. After the above attention mechanism is completed, a residual connection is performed, and the result is then normalized and a two-layer fully connected network is connected. Finally, a residual connection is performed again and the output is performed. It is worth noting that when dividing the window, the effect of sliding window can be achieved through different division methods.

[0057] The structure of the first dual-head feature extraction layer refers to Figure 7 ,The color feature first passes through the convolution layer, and the result is split into two parts. The first part and the position feature are used as input to pass through the two-stage information fusion module, and the second part passes through the convolution layer. The results of the two parts are concatenated and passed through the convolution layer and residual connection to the final fusion feature.

[0058] The structure reference of the second dual-head feature extraction layer Figure 8, the color feature first passes through the convolution layer, and the result is split into two parts. The first part is input into the window self-attention layer, and the second part is input into the convolution layer. The results of the two parts are concatenated, and the reconstructed features are obtained through the convolution layer and residual connection. Among them, the window self-attention is calculated as follows:

[0059] First, there is the color feature Split into non-overlapping M×M windows, and get the form Then, for each feature from F I Features Perform the following calculations:

[0060] Q = XP Q , K = XP K , V = XP V ,

[0061] In the formula, are the corresponding linear layer weights, and then, for the obtained Perform attention calculation:

[0062]

[0063] Where B is the relative position code within the window. After the above attention mechanism is completed, a residual connection is performed, and then the result is normalized and a two-layer fully connected network is constructed.

[0064] In the embodiment of the present application, before step S13, the two-stage demosaicing model is trained based on training data including historical mosaic images and original images. Fig. 9 As shown, a historical pure mosaic image without noise and missing event color information is generated according to the original image; a historical Quad Bayer arrangement position map and a historical event position mask map are obtained according to the historical mosaic image; an image pair consisting of the historical mosaic image and the historical pure mosaic image is used, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as position branch input to train the first stage network; an image pair consisting of the historical pure mosaic image and the original image is used, combined with the historical Quad Bayer arrangement position map as position branch input to train the second stage network; an image pair consisting of the historical mosaic image and the original image is used, combined with the historical event position mask map and the historical Quad Bayer arrangement position map, to jointly train the two-stage de-mosaicing model composed of the first stage network and the second stage network.

[0065] That is, when the first-stage network is first trained end-to-end, the first-stage network is applied to denoise and repair missing event color information according to the historical mosaic image, the historical event position mask map and the historical Quad Bayer arrangement position map to obtain a reconstructed mosaic image; the first-stage network is trained according to the error between the reconstructed mosaic image and the historical pure mosaic image. Given the original image training set The model can use the minimum L 1 Loss for training:

[0066]

[0067] In the formula, x k represents the reconstructed mosaic image, x p is the corresponding pure mosaic image, N represents the number of elements in the training set, |x k -x p | represents the difference between the reconstructed mosaic image and the pure mosaic image.

[0068] Then, when the second-stage network is trained end-to-end, the second-stage network is applied to perform de-mosaicing according to the historical pure mosaic image and the Quad Bayer arrangement position map to obtain the de-mosaiced historical reconstructed image; the second-stage network is trained according to the error between the historical reconstructed image and the original image. Given the original image training set The model can use the minimum L 1 Loss for training:

[0069]

[0070] In the formula, x k represents the historical reconstructed image, x is the corresponding original image, N represents the number of elements in the training set, |x k -x| represents the difference between the historical reconstructed image and the original image. After pre-training the two-stage network, the two-stage demosaicing model is jointly trained, and the historical mosaic image and the original image are input for training. Given the original image training set The model can use the minimum L 1 Loss for training:

[0071]

[0072] In the formula, x k represents the reconstructed image output by the two-stage demosaicing model, x is the corresponding original image, N represents the number of elements in the training set, |x k -x| represents the difference between the reconstructed image output by the two-stage demosaicing model and the original image.

[0073] This application uses a progressive training method when iteratively training the two-stage demosaicing model, that is, first training the networks in two different stages separately, and then merging the two into the same network for joint training, which can improve the accuracy of the two-stage demosaicing model training. This application uses an innovative algorithm framework to decompose complex tasks and design corresponding algorithm modules, effectively improving the demosaicing effect while reducing the number of model parameters, reducing computing resource requirements, optimizing computing efficiency, and ensuring efficient deployment of the algorithm in a resource-constrained environment. It can solve the problems of artifacts and moiré caused by the lack of event pixel color information in the prior art. Through targeted optimization, the image quality is significantly improved, and the model size is reduced while maintaining the algorithm performance. The deployment difficulty is simplified, providing an efficient and practical technical solution for the field of hybrid event camera image processing.

[0074] This application proposes to use a two-stage demosaicing model to demosaic the mixed event camera sensor. Through customized subtask allocation and dual-branch encoder, it achieves excellent performance while reducing computing resources. The two-stage demosaicing model considers color information and position information at the same time, achieves high-quality demosaicing effect through the joint encoding of the two, reconstructs high-quality RGB images, adopts the design of a dual-branch encoder, and uses a multimodal attention mechanism to enhance the detail color restoration effect, with a local-global attention acquisition mechanism. Targeted networks can effectively reduce network parameters and computational complexity, greatly save computing resources, and achieve the effect of lightweight deployment on mobile devices.

[0075] The embodiment of the present application obtains a mosaic image including RGB color information collected by a hybrid event image sensor; obtains a Quad Bayer arrangement position map and an event position mask map based on the mosaic image; and obtains a demosaiced reconstructed image by processing the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage de-mosaicing model based on a deep neural network. The color information and the position information can be considered at the same time, and a high-quality de-mosaicing effect can be achieved through the joint encoding of the two, thereby reconstructing a de-mosaiced high-quality RGB image.

[0076] Based on the same concept, an embodiment of the present invention also provides an image demosaicing system, such as Fig.10 As shown, it includes: an image acquisition unit, a position branch acquisition unit and a de-mosaicing unit.

[0077] An image acquisition unit, used to acquire a mosaic image including RGB color information acquired by using a mixed event image sensor;

[0078] A position branch acquisition unit, used for acquiring a Quad Bayer arrangement position map and an event position mask map according to the mosaic image;

[0079] The demosaicing unit is used to process the mosaic image, the Quad Bayer arrangement position map and the event position mask map using a two-stage demosaicing model based on a deep neural network to obtain a demosaiced reconstructed image.

[0080] For the convenience of description, the above system is described by dividing it into various modules according to its functions. Of course, when implementing the embodiment of the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0081] The system of the above embodiment is applied to the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0082] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the above embodiments is implemented.

[0083] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the method described in any of the above embodiments.

[0084] Fig.11 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1101, a memory 1102, an input / output interface 1103, a communication interface 1104, and a bus 1105. The processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 are connected to each other through the bus 1105 in the device.

[0085] The processor 1101 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided by the method embodiment of the present invention.

[0086] The memory 1102 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1102 may store an operating system and other application programs. When the technical solution provided by the method embodiment of the present invention is implemented by software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101.

[0087] The input / output interface 1103 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0088] The communication interface 1104 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0089] The bus 1105 includes a path that transmits information between the various components of the device (eg, the processor 1101 , the memory 1102 , the input / output interface 1103 , and the communication interface 1104 ).

[0090] It should be noted that, although the above device only shows the processor 1101, the memory 1102, the input / output interface 1103, the communication interface 1104 and the bus 1105, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0091] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0092] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0093] Further, the method can be implemented in and operably connected to any type of suitable computing platform, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0094] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0095] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. An image demosaicing method, characterized in that: The method comprises: Acquire a mosaic image including RGB color information acquired using a hybrid event image sensor; Acquire a Quad Bayer arrangement position map and an event position mask map according to the mosaic image; The mosaic image, the Quad Bayer arrangement position map and the event position mask map are processed by a two-stage demosaicing model based on a deep neural network to obtain a demosaiced reconstructed image; The processing of the mosaic image, the Quad Bayer arrangement position map and the event position mask map by a two-stage demosaicing model based on a deep neural network, and training the two-stage demosaicing model according to training data including historical mosaic images and original images before obtaining the demosaiced reconstructed image, comprises: Generate a historical pure mosaic image without noise and missing event color information based on the original image; Acquire a historical Quad Bayer arrangement position map and a historical event position mask map according to the historical mosaic image; Using the image pair consisting of the historical mosaic image and the historical pure mosaic image, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as position branch input, the first stage network is trained; Using the image pair consisting of the historical pure mosaic image and the original image, combined with the historical QuadBayer arrangement position map as position branch input, to train the second stage network; Using the image pair consisting of the historical mosaic image and the original image, combined with the historical event position mask map and the historical Quad Bayer arrangement position map, a two-stage de-mosaic model consisting of the first-stage network and the second-stage network is jointly trained.

2. The method according to claim 1, characterized in that The two-stage demosaicing model includes a first-stage network and a second-stage network, and the mosaic image, the Quad Bayer arrangement position map, and the event position mask map are processed by the two-stage demosaicing model based on a deep neural network to obtain a demosaiced reconstructed image, including: Applying the first-stage network to perform denoising and repair missing event color information according to the input mosaic image, the Quad Bayer arrangement position map and the event position mask map to obtain a pure mosaic image; The second-stage network is applied to perform a demosaicing process according to the Quad Bayer arrangement position map and the pure mosaic image to obtain a demosaiced reconstructed image.

3. The method according to claim 2, characterized in that The applying of the first-stage network to perform denoising and repairing of missing event color information according to the input mosaic image, the Quad Bayer arrangement position map and the event position mask map to obtain a pure mosaic image includes: Applying a repair layer to roughly repair missing event color information on the input mosaic image; Applying a first Fourier feature layer and a first-stage position extraction layer group to extract a first position feature according to the Quad Bayer arrangement position map and the event position mask map; Applying a plurality of one-stage color extraction layer groups to extract a first color feature and a first fusion feature according to the output of the repair layer and the first position feature; A plurality of first-stage intermediate provided layer groups and a plurality of first-stage reconstruction layer groups are applied to perform processing according to the first color feature and the first fusion feature to obtain a pure mosaic image.

4. The method according to claim 2, characterized in that: The applying the second-stage network to perform de-mosaicing according to the QuadBayer arrangement position map and the pure mosaic image to obtain a reconstructed image after de-mosaicing includes: Applying a second Fourier feature layer and a two-stage position extraction layer group to extract a second position feature according to the Quad Bayer arrangement position map and the event position mask map; Applying a plurality of two-stage color extraction layer groups to extract a second color feature and a second fusion feature according to the clean mosaic image and the second position feature; A plurality of two-stage intermediate provided layer groups and a plurality of two-stage reconstruction layer groups are applied to perform processing according to the second color feature and the second fusion feature to obtain a demosaiced reconstructed image.

5. The method according to claim 1, characterized in that The image pair consisting of the historical mosaic image and the historical pure mosaic image is used, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as position branch inputs, to train the first stage network, including: Applying the first-stage network to denoise and repair missing event color information according to the historical mosaic image, the historical event position mask map, and the historical Quad Bayer arrangement position map to obtain a reconstructed mosaic image; The first-stage network is trained according to the error between the reconstructed mosaic image and the historical pure mosaic image.

6. The method according to claim 1, characterized in that The image pair consisting of the historical pure mosaic image and the original image is used in combination with the historical Quad Bayer arrangement position map as a position branch input to train the second stage network, including: Applying the second-stage network to perform a demosaicing process according to the historical pure mosaic image and the Quad Bayer arrangement position map to obtain a demosaiced historical reconstructed image; The second-stage network is trained according to the error between the historical reconstructed image and the original image.

7. An image demosaicing system, characterized in that: include: An image acquisition unit, used to acquire a mosaic image including RGB color information acquired by using a mixed event image sensor; A position branch acquisition unit, used for acquiring a Quad Bayer arrangement position map and an event position mask map according to the mosaic image; A demosaicing unit is used to process the mosaic image, the Quad Bayer arrangement position map and the event position mask map by applying a two-stage demosaicing model based on a deep neural network to obtain a demosaiced reconstructed image; wherein, before the mosaic image, the Quad Bayer arrangement position map and the event position mask map are processed by applying a two-stage demosaicing model based on a deep neural network to obtain the demosaiced reconstructed image, the two-stage demosaicing model is trained according to training data including historical mosaic images and original images, including: generating a historical pure mosaic image without noise and missing event color information according to the original image; obtaining a historical Quad Bayer arrangement position map and a historical event position mask map according to the historical mosaic image; using an image pair consisting of the historical mosaic image and the historical pure mosaic image, combined with the historical event position mask map and the historical Quad Bayer arrangement position map as position branch inputs, to train the first stage network; using an image pair consisting of the historical pure mosaic image and the original image, combined with the historical Quad The Bayer arrangement position map is used as the position branch input to train the second stage network; the image pair consisting of the historical mosaic image and the original image is used, combined with the historical event position mask map and the historical Quad Bayer arrangement position map, to jointly train the two-stage de-mosaic model composed of the first stage network and the second stage network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the method as described in any one of claims 1-6.