Image imaging method, device, apparatus and storage medium

Abstract

The application discloses an image imaging method and device, equipment and storage medium, and relates to the technical field of imaging. The method comprises the following steps: acquiring a mosaic image; performing synchronous denoising and spectral reconstruction on the mosaic image by using a U-Net image imaging model to obtain a multispectral image; the U-Net image imaging model comprises an encoder network and a decoder network; the encoder network comprises a plurality of encoders, each encoder comprises a down-sampling layer, at least one of the encoders further comprises a first feature aggregation feedforward module configured to perform aggregation processing on the features in the neighborhood space of the image input into the first feature aggregation feedforward module; the decoder network comprises a plurality of decoders, each decoder comprises an up-sampling layer, and at least one of the decoders further comprises a second feature aggregation feedforward module configured to perform aggregation processing on the features in the neighborhood space of the image input into the second feature aggregation feedforward module. The application can improve the imaging quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of imaging technology, in particular to an image imaging method, device, equipment and storage medium. BACKGROUND

[0002] In the related art, a color camera is configured to add a CFA (color filter array) in front of a detector array to acquire an image with incomplete spatial information of a preset number of channels, i.e., a mosaic image, and then to supplement the information of the preset number of channels by using a corresponding algorithm to reconstruct the information of the preset number of channels to achieve the purpose of color imaging.

[0003] However, the noise removal is performed in the preprocessing process before the spectral reconstruction, which inevitably leads to the loss of spectral information and thus causes the color distortion of the final image. SUMMARY

[0004] The main purpose of the present application is to provide an image imaging method, device, equipment and storage medium, which aims to solve the technical problem of low imaging quality caused by noise.

[0005] To achieve the above purpose, in a first aspect, the present application provides an image imaging method, comprising:

[0006] acquiring a mosaic image;

[0007] synchronously processing the mosaic image by using a U-Net image imaging model to remove noise and reconstruct the spectrum to obtain a multispectral image; the U-Net image imaging model comprises an encoder network and a decoder network;

[0008] The encoder network comprises a plurality of encoders, each encoder comprises a down-sampling layer, and at least one encoder further comprises a first feature aggregation feedforward module configured to aggregate the features in the neighborhood space of the image input into the first feature aggregation feedforward module.

[0009] The decoder network comprises a plurality of decoders, each decoder comprises a down-sampling layer, and at least one decoder further comprises a second feature aggregation feedforward module configured to aggregate the features in the neighborhood space of the image input into the second feature aggregation feedforward module.

[0010] In an embodiment of the present application, at least one encoder further comprises a first multi-head attention module configured to extract the sparse spectral features of the image input into the first multi-head attention module to obtain a spectral feature map, and the first feature aggregation feedforward module is configured to aggregate the features in the neighborhood space of the spectral feature map.

[0011] The at least one decoder further comprises a second multi-head attention module configured to perform sparse feature recovery processing on the image input into the second multi-head attention module by using a multi-head attention mechanism to obtain a first information recovery image, and the second feature aggregation feedforward module is configured to perform aggregation processing on the features in the neighborhood space of the first information recovery image.

[0012] In a possible embodiment of the present application, the U-Net image imaging model further comprises a bottleneck layer, and the bottleneck layer comprises a third multi-head attention module and a third feature aggregation feedforward module.

[0013] The third multi-head attention module is configured to perform sparse feature recovery processing on the image output by the last encoding layer of the encoder network by using a multi-head attention mechanism to obtain a second information recovery image.

[0014] The third feature aggregation feedforward module is configured to perform aggregation processing on the features in the neighborhood space of the second information recovery image to obtain a spectral feature aggregation image, and input the spectral feature aggregation image into the first decoder of the decoder network.

[0015] In a possible embodiment of the present application, the U-Net image imaging model further comprises a channel expansion layer, a position encoding layer and a position decoding layer.

[0016] The channel expansion layer is used to expand the mosaic image into a channel image with a preset number of channels.

[0017] The position encoding layer is configured to perform spatial position encoding processing on the channel image to obtain spatial position features of the channel image.

[0018] The position decoding layer is configured to perform fusion processing on the image with the preset number of channels output by the last decoder in the decoder network based on the spatial position features to obtain a multispectral image.

[0019] In a possible embodiment of the present application, in the at least one encoder, the first multi-head attention module is configured to extract sparse spectral features of the image input into the encoder to obtain a spectral feature map, the first feature aggregation feedforward module is configured to perform aggregation processing on the features in the neighborhood space of the spectral feature map to obtain an aggregated feature map, and the down-sampling layer is configured to down-sample the aggregated feature map to obtain a feature map; wherein the aggregated feature map is also input into the corresponding decoder.

[0020] In at least one decoder, the up-sampling layer is configured to perform up-sampling processing on an image input to the up-sampling layer to obtain a restored image, the second multi-head attention module is configured to perform sparse feature restoration processing on a spliced image obtained by splicing the restored image and a corresponding aggregated feature map using a multi-head attention mechanism to obtain a first information restored image, and the second feature aggregation feed-forward module is configured to perform aggregation processing on features in a neighborhood space of the first information restored image to obtain a spectral restored image.

[0021] In a possible embodiment of the present application, all encoders in the encoder network include the first multi-head attention module and the first feature aggregation feed-forward module; and / or

[0022] All decoders in the decoder network include the second multi-head attention module and the second feature aggregation feed-forward module.

[0023] In a possible embodiment of the present application, the encoder network includes 3 layers of encoders, and the decoder network includes 3 layers of decoders.

[0024] In a second aspect, the present application further provides an image imaging device, comprising:

[0025] An image acquisition module configured to acquire a mosaic image;

[0026] A spectral reconstruction module configured to perform denoising and spectral reconstruction synchronous processing on the mosaic image by using a U-Net image imaging model to obtain a multi-spectral image; the U-Net image imaging model includes an encoder network and a decoder network;

[0027] The encoder network includes a plurality of encoders, each encoder includes a down-sampling layer, and at least one encoder further includes a first feature aggregation feed-forward module configured to perform aggregation processing on features in a neighborhood space of an image input to the first feature aggregation feed-forward module.

[0028] The decoder network includes a plurality of decoders, each decoder includes a down-sampling layer, and at least one decoder further includes a second feature aggregation feed-forward module configured to perform aggregation processing on features in a neighborhood space of an image input to the second feature aggregation feed-forward module.

[0029] In a third aspect, the present application further provides an image signal processing device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the image imaging method of the first aspect.

[0030] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image imaging method of the first aspect.

[0031] The image imaging method provided by the technical scheme of the present application utilizes a U-Net image imaging model, which comprises a channel expansion layer, an encoder network, a decoder network and a channel fusion layer; wherein at least one of the encoders further comprises a first feature aggregation feedforward module, which is configured to perform aggregation processing on the features in the neighborhood space of the image input to the first feature aggregation feedforward layer; and at least one of the decoders further comprises a second feature aggregation feedforward module, which is configured to perform aggregation processing on the features in the neighborhood space of the image input to the second feature aggregation feedforward layer.

[0032] As can be seen, the U-Net image imaging model provided by the technical scheme of the present application has a feature aggregation feedforward module in each of the encoder network and the decoder network, so that the feature selection technology is utilized to perform feature aggregation in the neighborhood space of the processed image, so as to strengthen the effective features and simultaneously suppress the noise. Thus, the present application provides a synchronous processing network for denoising and spectral reconstruction, so as to improve the imaging quality. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical schemes in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the structures shown in the drawings.

[0034] Figure 1 FIG. 1 is a structural schematic diagram of an image imaging device of the present application;

[0035] Figure 2 FIG. 2 is a flowchart of a first embodiment of an image imaging method of the present application;

[0036] Figure 3 FIG. 3 is a schematic diagram of a U-Net image imaging model of the first embodiment of the image imaging method of the present application;

[0037] Figure 4 FIG. 4 is a schematic diagram of a U-Net image imaging model of a second embodiment of the image imaging method of the present application;

[0038] Figure 5 FIG. 5 is a schematic diagram of a U-Net image imaging model of a third embodiment of the image imaging method of the present application;

[0039] Figure 6 FIG. 6 is a schematic diagram of a U-Net image imaging model of a fourth embodiment of the image imaging method of the present application;

[0040] Figure 7 FIG. 7 is a schematic diagram of a light filtering micro-nano structure of an embodiment of the light filtering micro-nano structure in the present application;

[0041] Figure 8 An arrangement of filters in a macro-pixel in an example of the present application;

[0042] Figure 9 An arrangement of filters in a macro-pixel in an example of the present application;

[0043] Figure 10 A schematic diagram of a module of an image forming device of the present application.

[0044] BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Label name Label name 10 first transparent substrate 20 Light filtering micro-nanostructures 21 The first micro-nano modulation layer 22 The second micro-nano modulation layer 23 First transparent protective layer

[0046] The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work under the premise of the present application, all belong to the scope of protection of the present application.

[0048] It should be noted that all the directionality indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between the components in a certain specific posture (as shown in the drawings), if the specific posture changes, the directionality indications will also change accordingly.

[0049] In the present application, unless otherwise explicitly specified and limited, the terms "connection", "fixation" and the like should be understood in a broad sense, for example, "fixation" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection; can be direct connection, or indirect connection through an intermediate medium; can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0050] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. For example, "A and / or B" includes A scheme, or B scheme, or A and B scheme. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0051] In the related art, a general color camera generally adds a Bayer CFA (color filter array) in front of the detector array to collect an image of incomplete spatial information of three channels of red, green and blue (R, G, B), that is, a mosaic image, and then the missing information of the three spectral channels is supplemented through the corresponding ISP (Image Signal Processing) algorithm to achieve the purpose of color imaging.

[0052] However, in the related imaging processing process, noise removal is performed in the preprocessing process, which inevitably leads to loss of spectral information and error accumulation problem, thereby affecting the imaging quality. Or in the related spectral reconstruction processing network, the denoising network and the spectral reconstruction network are independent modules, and the denoising network is before the spectral reconstruction network, so there is also a problem of loss of spectral information and error accumulation in the imaging processing process based on deep learning technology, thereby affecting the imaging quality.

[0053] Therefore, the present application provides a solution, in which the encoder network and the decoder network of the U-Net image imaging model used in the solution both have a feature aggregation feedforward module, so as to use the feature selection technology to perform feature aggregation in the neighborhood space of the processed image, so as to strengthen the effective features and suppress the noise at the same time. Therefore, the present application provides a denoising and spectral reconstruction synchronous processing network, which synchronously performs denoising and spectral reconstruction processing on the complete spatial resolution image, so as to avoid the loss of spectral information caused by denoising before spectral reconstruction, so as to improve the imaging quality.

[0054] The inventive concept of the present application will be further described below in conjunction with some specific embodiments.

[0055] Reference Figure 1 , Figure 1 The structural schematic diagram of the image imaging device related to the hardware running environment of the embodiments of the present application.

[0056] As shown in Figure 1 The image imaging device can include a processor 1001, such as a CPU, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection communication between the components. The user interface 1003 can include a voice pickup module, such as a microphone array, and the like. The optional user interface 1003 can also be a display, an input unit such as a keyboard, and the like. The memory 1005 can be a high-speed RAM memory, or a stable memory such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0057] It can be understood that the image imaging device can also include a network interface 1004, which can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). Optionally, the image imaging device can also include RF (Radio Frequency, radio frequency) circuit, sensor, audio circuit, WiFi module, and the like.

[0058] Those skilled in the art can understand that Figure 1 The structure of the image imaging device shown in the above description does not constitute a limitation on the image imaging device, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0059] Based on the above structure but not limited to the above structure, the first embodiment of the image imaging method of the present application is proposed. Referring to Figure 2 , Figure 2 The flowchart of the first embodiment of the image imaging method of the present application is shown.

[0060] It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown.

[0061] In this embodiment, the method comprises:

[0062] Step S100, acquiring a mosaic image.

[0063] The execution subject of the image imaging method provided in this embodiment is an image signal processor of an image imaging device, and the image imaging device further includes an image sensor. The image signal processor is used to perform ISP processing on the mosaic image output by the image sensor, and finally output a color image and / or a near-infrared image in RGB format.

[0064] It can be understood that the image sensor collects a spectral signal for the target to be imaged. Of course, the spectral signal also includes spatial distribution information corresponding to the target to be imaged. Then the collected spectral signal is spectrally modulated by the light filtering array in the image sensor, and the pixel array photoelectrically senses the modulated spectral signal, thereby outputting a mosaic raw data, i.e. a mosaic image. The image signal processor receives the mosaic image through the port connected with the image sensor.

[0065] Step S200, the mosaic image is simultaneously denoised and spectrally reconstructed by the U-Net image imaging model to obtain a multispectral image.

[0066] Please refer to Figure 3 , the U-Net image imaging model includes an encoder network and a decoder network.

[0067] Among them, the encoder network includes a plurality of encoders, the encoder includes a down-sampling layer, and at least one encoder further includes a first feature aggregation feedforward module, the first feature aggregation feedforward module is configured to aggregate the features in the neighborhood space of the image input to the first feature aggregation feedforward layer; the decoder network includes a plurality of decoders, the decoder includes a down-sampling layer, and at least one decoder further includes a second feature aggregation feedforward module, the second feature aggregation feedforward module is configured to aggregate the features in the neighborhood space of the image input to the first feature aggregation feedforward layer.

[0068] It can be understood that the overall structure of the U-Net neural network model is similar to the U shape, including an encoder network and a decoder network, wherein the encoder network includes K encoders, and K is a positive integer greater than 1. The image is sequentially input to the K encoders for multiple down-sampling processing, thereby extracting the deep features in the image, and it can be understood that the size of the K encoders decreases in turn, thereby the feature map obtained is smaller and smaller. The image output by the last encoder is input to the decoder network after passing through the bottleneck layer (generally including 2 3*3 convolution layers). The decoder network includes K decoders. The K decoders sequentially perform up-sampling processing on the input image to perform image recovery processing. It is worth mentioning that each decoder is concatenated with the corresponding encoder in feature mapping, that is, the image input to each decoder is the splicing image of the image output by the bottleneck layer and the feature map output by the corresponding encoder, or the splicing image of the image output by the last decoder and the feature map output by the corresponding encoder.

[0069] In the embodiment, the at least one encoder in the encoder network and the at least one decoder in the decoder network each include a feature aggregation feedforward module. Since the noise features are not closely related to other features in their neighborhood space, and the effective spectral features are more closely related to the spectral features in their neighborhood space, the feature aggregation feedforward module can use a feature selection technique to aggregate the features in the feature layer neighborhood space of the processed image, thereby strengthening the effective spectral features and suppressing the noise features. Specifically, when performing feature extraction in the encoder, the feature map output by the encoder is more focused on expressing the effective spectral features after the first feature aggregation feedforward module, and the noise features are suppressed at the same time. When performing image restoration in the decoder, the effective spectral features are strengthened in the spectral image output by the decoder after the second feature aggregation feedforward module, and the noise features are also suppressed at the same time.

[0070] It can be understood that the aggregation processing in the feature aggregation feedforward module can be weighted aggregation, which is not limited in the embodiment.

[0071] In addition, the encoder in the encoder network will perform down-sampling processing, and the encoder network includes multiple layers of encoders. Therefore, the first feature aggregation feedforward module can perform feature aggregation processing in the feature layer neighborhood space of the image before down-sampling processing, or perform feature aggregation processing in the feature layer neighborhood space of the image after down-sampling processing, which is not limited in the embodiment.

[0072] As can be seen, in the image imaging method provided by the embodiment, a U-Net image imaging model is used, and the feature aggregation feedforward modules in the encoder network and the decoder network perform aggregation processing on the spectral features, so as to complete the suppression of noise information in the spectral reconstruction process. Therefore, the U-Net image imaging model in the embodiment is constructed as a synchronous processing network for denoising and spectral reconstruction, and can perform synchronous denoising and spectral reconstruction processing on the full spatial resolution image, so as to avoid the loss of spectral information caused by denoising before spectral reconstruction, thereby improving the imaging quality.

[0073] It can be understood that, since the filter array of the image sensor has a preset number of channels, in order to complete the spectral reconstruction, the U-Net image imaging model further includes a channel expansion layer, a position encoding layer, and a position decoding layer.

[0074] The channel expansion layer is configured to expand the mosaic image to a channel image with a preset number of channels. The position encoding layer is configured to perform spatial position encoding processing on the channel image to obtain spatial position features of the channel image. The position decoding layer is configured to perform fusion processing on the image with a preset number of channels output by the last decoder in the decoder network based on the spatial position features, to obtain a multispectral image.

[0075] Specifically, the channel expansion layer is configured to perform channel conversion on the mosaic image to obtain a plurality of channel images with a preset number of channels, and the position encoder layer is configured to perform spatial position encoding processing on the channel images to retain spatial position features of the channels, so that correct spatial information recovery can be achieved when the plurality of channel images with the preset number of channels output by the last decoder are fused through the spatial position features in the position decoder layer.

[0076] Based on the above embodiment, a second embodiment of the image imaging method is provided.

[0077] It can be understood that in the related art, the light flux of the color filter array is only about 1 / 3 in the low-illumination scene at night, and most of the light energy is absorbed by the dye, resulting in a significant decrease in imaging quality and serious color distortion. In addition, during the imaging processing, noise removal is performed in the preprocessing process, which inevitably leads to loss of spectral information. Therefore, the above two factors are superimposed on each other, resulting in serious color distortion of the final image in the low-illumination scene.

[0078] Therefore, referring to Figure 4 In this embodiment, the at least one encoder of the U-Net image imaging model further includes a first multi-head attention module configured to extract sparse spectral features of an image input to the first multi-head attention module to obtain a spectral feature map, and a first feature aggregation feedforward module configured to aggregate features in a neighborhood space of the spectral feature map.

[0079] Specifically, the first multi-head attention module can use the multi-head attention mechanism to pay more attention to the sparse spectral features in the image when extracting deep features. Therefore, in the encoder, the first multi-head attention module extracts sparse spectral features from the input image to make the sparse spectral features more expressive, and obtains a spectral feature map. Then, the spectral feature map with enhanced expression of sparse spectral features is transmitted to the first feature aggregation feedforward module, which aggregates features in the neighborhood space of the spectral feature map. In this process, the sparse spectral features are further aggregated and enhanced.

[0080] The at least one decoder further includes a second multi-head attention module configured to use the multi-head attention mechanism to perform sparse feature recovery processing on an image input to the second multi-head attention module to obtain a first information recovery image, and a second feature aggregation feedforward module configured to aggregate features in a neighborhood space of the first information recovery image.

[0081] Specifically, the second multi-head attention module can utilize the multi-head attention mechanism to pay more attention to the sparse spectral features in the image during image restoration. In this way, in the decoder, the second multi-head attention module can better restore the sparse spectral features in the spectral signal to obtain a first information restoration image. The first information restoration image in which the sparse spectral features are enhanced and restored is then fed to the second feature aggregation feedforward module, which aggregates the features in the neighborhood space of the first information restoration image. In this process, the sparse spectral features in the spectral signal are further aggregated and enhanced.

[0082] It can be understood that the advantage of the deep learning-based spectral reconstruction is that deep features are used to represent the intrinsic sparsity of the spectral signal, and the sparsity expression capability is stronger than that of the sparse optimization and sparse representation methods. The U-Net image imaging model in the embodiment performs better in extracting sparse spectral features by utilizing the multi-head attention mechanism, thereby performing better in imaging quality in the face of images in low-illumination scenes. In combination with the feature aggregation feedforward module, not only can the spectral features in the low-illumination scene be better extracted, but noise features can also be simultaneously suppressed when facing images in low-illumination scenes, thereby avoiding loss of spectral information and error accumulation, so as to jointly realize high-quality spectral reconstruction in low-illumination.

[0083] In the embodiment, the image sensor for collecting the mosaic image can include a pixel array and a filter array corresponding to the pixel array, and the filter array includes a macro-pixel, the macro-pixel includes at least four filters arranged in an array, the at least four filters include at least two micro-nano structure filters to form at least three channels, the micro-nano structure filter includes a first transparent substrate and a plurality of filter micro-nano structures arranged on one side of the first transparent substrate in an array, the filter micro-nano structure includes a first micro-nano modulation layer, a second micro-nano modulation layer and a first transparent protective layer, the first micro-nano modulation layer is arranged on the first transparent substrate, and the material of the first micro-nano modulation layer is one of a transparent material and a light loss material, the refractive index of the light loss material is higher than the refractive index of the transparent material, the second micro-nano modulation layer is arranged on the side of the first micro-nano modulation layer away from the first transparent substrate, the material of the second micro-nano modulation layer is the other of the transparent material and the light loss material, the first transparent protective layer covers the first micro-nano modulation layer and the second micro-nano modulation layer, and is connected with the first transparent substrate, and the refractive index of the transparent material is higher than the refractive index of the material of the first transparent substrate and the first transparent protective layer.

[0084] The filter array is configured to correspond to the pixel array, and each filter in the filter array corresponds to a single pixel, and a plurality of filters can be combined to form a macro-pixel. In color reconstruction, the macro-pixel is taken as the object, and the color of the macro-pixel as a whole can be reconstructed according to the electrical signals output by the pixels corresponding to each filter. The macro-pixel includes at least 4 filters arranged in an array, that is, the macro-pixel can be a 2x2, 3x3, or 4x4 filter square array. Subsequently, a 2x2 filter array is taken as an example to be described in detail.

[0085] The filter array includes a transparent medium layer opposite the pixel array, and the transparent medium layer can be divided into a plurality of block-shaped first transparent substrates 10 according to the pixel array. Each first transparent substrate 10 forms a filter in combination with the filter structure arranged thereon. The filter structure can be configured as a filter micro-nano structure array, thereby forming the micro-nano structure filter in the embodiment. The filter micro-nano structure is used to filter and modulate incident light. Specifically, the filter micro-nano structure includes a first micro-nano modulation layer 21, a second micro-nano modulation layer 22, and a first transparent protective layer 23.

[0086] Referring to Figure 7 The first transparent substrate 10 is the main part of the micro-nano structure filter, which is made of a transparent material with a low refractive index corresponding to the light wave band. The refractive index of the material of the first transparent substrate 10 is c, and c satisfies: 1.4 < c < 2. Specifically, the material can be various glass materials or silicon dioxide SiO2 used for filters, etc. It can be understood that the first transparent substrate 10 allows all light in the visible light wave band or near-infrared light to pass through. The first transparent substrate 10 and the pixel array are arranged opposite to each other, so that the light signal filtered and modulated by the filter is received by the corresponding pixel array to output the mosaic raw data, that is, the mosaic image. The first micro-nano modulation layer 21 is arranged on the side surface of the first transparent substrate 10 away from the pixel array, and the second micro-nano modulation layer 22 is arranged on the side surface of the first micro-nano modulation layer 21 away from the first transparent substrate 10.

[0087] The first micro-nano modulation layer 21 has a first orthographic projection on the first transparent substrate 10, and the second micro-nano modulation layer 22 has a second orthographic projection on the first transparent substrate 10. The first orthographic projection and the second orthographic projection coincide with each other, that is, the outer circumferential surfaces of the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 are coplanar and perpendicular to the first transparent substrate 10, so as to facilitate the incident light to pass through the double-layer modulation structure and enter the first transparent substrate 10.

[0088] It is worth mentioning that the material of the first micro-nano modulation layer 21 is one of transparent material and light loss material, and the material of the second micro-nano modulation layer 22 is the other one of transparent material and light loss material. That is, when the material of the first micro-nano modulation layer 21 is transparent material, the material of the second micro-nano modulation layer 22 is light loss material. Or, when the material of the second micro-nano modulation layer 22 is transparent material, the material of the first micro-nano modulation layer 21 is light loss material. It can be understood that, in the embodiment, the high refractive index and the low refractive index of the material are only for the visible light band, or for the visible light band and the near-infrared light band. Specifically, the refractive index of the transparent material is a, and a satisfies: 2

[0089] The first transparent protection layer 23 is arranged on the side of the first transparent substrate 10 away from the pixel array, and covers the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22. Optionally, the first transparent protection layer 23 is formed by photoresist filled around the micro-nano structure, for protecting the micro-nano structure, and for flattening the filter surface to facilitate the preparation of other structural devices such as microlenses thereon. The refractive index of the material of the first transparent protection layer 23 is d, and d satisfies: 1.4

[0090] In the embodiment, the modulation layer made of light loss material with high refractive index is used to increase the refractive index contrast between the micro-nano structure as a whole and the surrounding medium such as photoresist and filter substrate, thereby improving the coupling strength of the incident light field of the micro-nano structure and the waveguide light field in the micro-nano structure, and enhancing the filtering modulation ability of the micro-nano structure when the number is small. In addition, the light loss of the light loss material will make the quality factor Q of the resonance in the first transparent substrate 10 decrease, thereby facilitating the accurate reconstruction of subsequent colors after wideband modulation.

[0091] It can be understood that when light enters from one medium to another medium, if the refractive index difference between the two media decreases, the energy of the reflected light decreases, and the energy of the transmitted light increases. In the embodiment, since the refractive index of the transparent material is higher than that of the first transparent substrate 10 and the first transparent protection layer 23, the refractive index difference between adjacent media decreases from the incident end to the transmission end of the filtering micro-nano structure, so as to improve the transmittance in the embodiment.

[0092] As can be seen, the embodiment designs the filter micro-nano structure as a double-layer modulation layer structure formed by laminating the modulation layer made of the light loss material with high refractive index and the transparent material with low refractive index, and the double-layer modulation layer structure simultaneously has the high transmittance and high modulation characteristics of the light loss material with high refractive index and the transparent material with low refractive index, thereby improving the transmittance of the incident light, collecting more spectral information, ensuring that the filter micro-nano structure still has strong broadband spectral modulation capability under the small-size pixel arrangement of several wavelengths, and realizing the color imaging function under low illumination, and improving the broadband spectral modulation capability of the filter as a whole under the condition of the spectral modulation capability reduction caused by the refractive index contrast reduction of the small-size pixel with the first transparent protective layer 23, and ensuring that the filter still has strong broadband spectral modulation capability for color reconstruction.

[0093] In addition, compared with the existing complementary color filter array, the filter micro-nano structure provided in the embodiment does not need to additionally prepare a multi-layer coated filter, thereby reducing the manufacturing cost.

[0094] As an optional embodiment, the transparent material is silicon nitride Si3N4 or titanium dioxide TiO2. It can be understood that the refractive index of Si3N4 or TiO2 is higher than that of various glass materials or silicon dioxide SiO2 used for filters.

[0095] As an optional embodiment, the light loss material is polysilicon P-Si, amorphous silicon α-Si or germanium Ge. At this time, in the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22, the layer thickness of the modulation layer made of the light loss material is H2, and 10 nm≤H2≤100 nm, which can ensure that the micro-nano structure has a large light flux, such as greater than 60%, and enhance the broadband modulation capability of the filter under a small amount of micro-nano structure arrangement, and avoid the reduction of the broadband modulation capability.

[0096] Of course, the side of the first transparent substrate 10 away from the pixel array can also be not provided with the filter micro-nano structure, so that this part of the filter substantially allows all light in the visible light band to pass through, that is, allows white light to pass through, to form a visible light full-pass filter.

[0097] Specifically, the visible light full-pass filter only includes a second transparent substrate and a second transparent protective layer, and the second transparent protective layer is arranged on the side of the second transparent substrate away from the pixel array. The second transparent substrate is the transparent substrate 10 corresponding to a pixel in the transparent dielectric layer.

[0098] It can be understood that the visible light full-pass filter is approximately full-pass light energy, which can further improve the overall light transmittance of the filter array to collect more spectral information in a low-illumination scene.

[0099] In this embodiment, each macropixel includes at least two micro-nanostructure filters. This means that in this embodiment, a 2×2 array can contain two identical filters, or four different filters. It will be appreciated that different filters have different broadband spectral modulation curves, thereby forming three or four channels.

[0100] As an alternative to this embodiment, see Figure 8 The four micro-nanostructure filters in the macropixel are arranged in a 2×2 array; wherein, the four micro-nanostructure filters include a first filter M1, a second filter M2, a third filter M3 and a fourth filter M4, and the filtering micro-nanostructures of any two of the first filter M1, the second filter M2, the third filter M3 and the fourth filter M4 are different from each other.

[0101] Alternatively, see Figure 9 As another option of this embodiment, the four micro-nanostructure filters in the macropixel are arranged in a 2×2 array; wherein the four micro-nanostructure filters include two first filters M1, a second filter M2, and a third filter M3, the filtering micro-nanostructures of any two of the first filter M1, the second filter M2, and the third filter M3 are different from each other, and the two first filters are arranged along the diagonal direction of the macropixel.

[0102] Alternatively, as another option of this embodiment, the macropixel structure includes 2 micro-nanostructure filters and 2 visible light all-pass filters, the 2 micro-nanostructure filters and the 2 visible light all-pass filters are arranged in a 2×2 array, and the visible light all-pass filters are arranged along the diagonal direction of the macropixel.

[0103] The preferred embodiment provided in this embodiment has a better combination of arrangement period and shape size parameters, which can enable the filter array to still have the broadband modulation function of accurately reconstructing colors under low illumination with small-sized pixels.

[0104] For example, a 2×2 array containing two different micro-nanostructure filters and two visible light all-pass filters has a broadband modulation curve of M1 and a broadband modulation curve of M2. The energy values ​​received by the visible light all-pass filters are approximately the same, which can be expressed as C. The two different micro-nanostructure filters and the two visible light all-pass filters provide a total of n spectral modulation channels. At this time, the energy values ​​of the three colors of light received by the macro pixel, I1, I2, and I3, are expressed as:

[0105] Equation 1:

[0106] Where A is the modulation matrix composed of broadband modulation curves M1, M2 and C, φ nThe incident spectrum of the nth spectral modulation channel. By using the compressed sensing algorithm, the incident spectrum corresponding to the macro-pixel can be solved by equation one, and the color RGB value of the incident light at the macro-pixel position is as follows:

[0107]

[0108] It can be understood that the compressed sensing algorithm requires that the correlation between the wideband modulation curve M1 and the wideband modulation curve M2 is small, and as close as possible to the linear combination of R, G, and B curves, that is, at this time, M1 and M2 satisfy: minM i -(a·R+b·G+c·B)2, i = 1, 2. Where a, b, and c are constants. At the same time, the minimum normalized singular value of the modulation matrix A composed of M1, M2, and C, that is, α = min[SVD(A)] / sum[SVD(A)], generally should be greater than 0.05, where SVD(A) is the singular value decomposition of the modulation matrix A.

[0109] It is worth mentioning that in the present embodiment, the filter can not only perform wideband modulation on the visible light spectrum signal, but also perform wideband modulation on the visible light spectrum signal and the near-infrared light spectrum signal, so that the image signal processing device can realize high-quality visible light multispectral and color RGB imaging and near-infrared image imaging under low illumination.

[0110] In order to enable those skilled in the art to better understand the light filtering micro-nano structure in the present embodiment, the following specific implementation examples are explained and described. In the following examples, the combination of the structure height, arrangement period and shape size parameters of the modulation layer is optimal, which can enable the light filter array to still have the wideband modulation function of accurately reconstructing color under low illumination under small size pixels.

[0111] Structure one: the first transparent substrate 10 is made of silicon dioxide SiO2, and the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 are both structured as columnar structures with circular cross sections. The first micro-nano modulation layer 21 adopts light loss material amorphous silicon (α-Si), and the second micro-nano modulation layer 22 adopts transparent material titanium dioxide TiO2. It can be understood that the refractive index of TiO2 is greater than that of SiO2.

[0112] The height of the second micro-nano modulation layer 22 ranges between 200-1000nm, and the pitch ranges between 200-500nm, so as to ensure that there are several waveguide modes in the double-layer modulation structure for modulation. In order to ensure a large light flux and enhance the wideband modulation capability under a small number of micro-nano structures, the height of the first micro-nano modulation layer 21 is between 20-100nm. The overall duty cycle of the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 is between 0.2-0.8, so as to ensure that the structure is easy to etch and prepare.

[0113] Based on the above structure one, two specific examples are provided:

[0114] Example 1: the height of the first micro-nano modulation layer 21 (a-Si) is 62.8 nm, the height of the second micro-nano modulation layer 22 (TiO2) is 465.4 nm, the pitch is 265.2 nm, and the duty cycle is 0.31;

[0115] Example 2: the height of the first micro-nano modulation layer 21 (a-Si) is 62.8 nm, the height of the second micro-nano modulation layer 22 (TiO2) is 465.4 nm, the pitch is 377.3 nm, and the duty cycle is 0.57.

[0116] The light filtering micro-nano structure 20 provided in Example 1 forms a first filter 100, and the light filtering micro-nano structure 20 provided in Example 2 forms a second filter 200. In a 2x2 macro-pixel area, the first filter 100 is located in the upper left sub-area, the second filter 200 is located in the lower right sub-area, and two third filters 300 allowing white light to pass through are respectively located in the upper right and lower left sub-areas. The high transmittance broadband modulation curves of Example 1 and Example 2 reach 75.4% and 68.6% respectively. The average value of the color deviation ΔE of the reconstructed color image is as low as 0.87, and it can be seen that the filter array using Example 1 and Example 2 accurately reconstructs the image color.

[0117] Structure two: the first transparent substrate 10 is made of SiO2, and the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 are both configured as columnar structures with circular cross-sections. The first micro-nano modulation layer 21 is made of transparent material silicon nitride Si3N4, and the second micro-nano modulation layer 22 is made of light loss material polycrystalline silicon (p-Si).

[0118] The height of the first micro-nano modulation layer 21 is between 200-1500 nm, and the pitch P is between 200-500 nm to ensure that there are several waveguide modes in the double-layer modulation structure for modulation. In order to ensure a large light flux and modulation capacity, the height of the second micro-nano modulation layer 22 is between 30-200 nm. The overall duty cycle of the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 is between 0.2-0.8, which ensures that the structure is easy to etch and prepare.

[0119] Based on the above structure two, two specific examples are provided:

[0120] Example 3: the height of the first micro-nano modulation layer 21 (Si3N4) is 431.3 nm, the height of the second micro-nano modulation layer 22 (p-Si) is 76.5 nm, the pitch is 269.2 nm, and the duty cycle is 0.44;

[0121] Example 4: the height of the first micro-nano modulation layer 21 (Si3N4) is 431.3 nm, the height of the second micro-nano modulation layer 22 (p-Si) is 76.5 nm, the pitch is 347.9 nm, and the duty cycle is 0.54.

[0122] The light-filtering micro-nano structure 20 provided in Example 3 forms a first filter 100, and the light-filtering micro-nano structure 20 provided in Example 4 forms a second filter 200. In a 2x2 macro-pixel area, the first filter 100 is located in the upper left sub-area, the second filter 200 is located in the lower right sub-area, and two third filters 300 allowing white light to pass through are respectively located in the upper right sub-area and the lower left sub-area. The high-transmittance broadband modulation curves of Example 3 and Example 4 reach 65.7% and 72.7%, respectively. The average value of the color deviation ΔE of the reconstructed color image is as low as 1.13, and the filter array using Example 3 and Example 4 accurately reconstructs the image color.

[0123] Structure three: the first transparent substrate 10 is made of SiO2, and the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 are both configured as columnar structures with square cross sections. The first micro-nano modulation layer 21 is made of a light loss material, polycrystalline silicon (p-Si), and the second micro-nano modulation layer 22 is made of a transparent material, titanium dioxide (TiO2).

[0124] The height of the first micro-nano modulation layer 21 is between 20-100 nm, and the pitch P is between 200-500 nm to ensure that there are several waveguide modes in the double-layer modulation structure for modulation. In order to ensure a large light flux and modulation capacity, the height of the second micro-nano modulation layer 22 is between 200-1000 nm. The overall duty cycle of the first micro-nano modulation layer 21 and the second micro-nano modulation layer 22 is between 0.2-0.8, ensuring that the structure is easy to etch and prepare.

[0125] Based on the above structure three, two specific examples are provided:

[0126] Example 5: the height of the first micro-nano modulation layer 21 (p-Si) is 83.6 nm, the height of the second micro-nano modulation layer 22 (TiO2) is 313.2 nm, the pitch is 379.7 nm, and the duty cycle is 0.41;

[0127] Example 6: the height of the first micro-nano modulation layer 21 (p-Si) is 83.6 nm, the height of the second micro-nano modulation layer 22 (TiO2) is 313.2 nm, the pitch is 271.6 nm, and the duty cycle is 0.62.

[0128] The light filtering micro-nano structure 20 provided in Example 5 forms a first filter 100, and the light filtering micro-nano structure 20 provided in Example 6 forms a second filter 200. In a 2*2 macro-pixel, the first filter 100 is located in the upper left sub-region, the second filter 200 is located in the lower right sub-region, and two third filters 300 allowing white light to pass through are respectively located in the upper right sub-region and the lower left sub-region. The high transmittance broadband modulation curves of Examples 5 and 6 reach 77.9% and 63.8%, respectively. The average value of the color deviation AE of the reconstructed color image is as low as 1.21, and the filter array using Examples 5 and 6 can accurately reconstruct the image color.

[0129] To this end, in one aspect, the light filtering micro-nano structure is designed as a double-layer modulation structure composed of a modulation layer made of a light loss material with a high refractive index and a transparent material with a low refractive index, so that the double-layer modulation structure simultaneously has the high transmittance and high modulation characteristics of the two materials, thereby improving the transmittance of incident light and further enabling more spectral information to be collected. On the other hand, the complete spatial resolution image is simultaneously denoised and spectrally reconstructed by the U-Net image imaging model, which can avoid the loss of spectral information caused by denoising before spectral reconstruction. In this way, the present application improves from two dimensions of collecting more spectral information and avoiding the loss of spectral information caused by removing noise, and further realizes the function of high-quality color imaging under low illumination.

[0130] Based on the above embodiments, a third embodiment of the image imaging method of the present application is proposed.

[0131] Please refer to Figure 5 In this embodiment, the U-Net image imaging model further includes a bottleneck layer, and the bottleneck layer includes a third multi-head attention module and a third feature aggregation feedforward module.

[0132] The third multi-head attention module is configured to use a multi-head attention mechanism to perform sparse feature recovery processing on the image output by the last encoding layer of the encoder network, to obtain a second information recovery image.

[0133] The third feature aggregation feedforward module is configured to aggregate the features in the neighborhood space of the second information recovery image to obtain a spectral feature aggregation image, and input the spectral feature aggregation image to the first decoder of the decoder network.

[0134] It can be understood that in the related art, the bottleneck layer of the U-Net image imaging model is generally only constructed as two 3*3 convolution layers, i.e., a down-sampling layer and an up-sampling layer. In this embodiment, in order to improve the imaging quality, the bottleneck layer is constructed as the third multi-head attention module and the third feature aggregation feedforward module.

[0135] Specifically, the image output by the last layer of the encoder in the encoder network is input into the third multi-head attention module of the bottleneck layer, and the third multi-head attention module processes the sparse spectral features in the input image by using the multi-head attention of the spectral feature channel, so that the sparse spectral features can be better expressed, thereby obtaining a second information recovery image after sparse feature recovery processing. The second information recovery image is input into the third feature aggregation feedforward module, in which the sparse spectral features are aggregated and strengthened and noise features can be suppressed, so that the spectral features can be better expressed. Then the output image is input into the decoder network.

[0136] Based on the above embodiment, a fourth embodiment of the image imaging method is proposed.

[0137] Please refer to Figure 6 In this embodiment, the U-Net image imaging model includes:

[0138] The channel expansion layer is configured to expand the mosaic image to a channel image with a preset number of channels.

[0139] The position encoding layer is configured to perform spatial position encoding processing on the channel image to obtain spatial position features of the channel image.

[0140] The encoder network comprises three layers of encoders, the first layer of encoders comprising a first multi-head attention module, a first feature aggregation feedforward module and a down-sampling layer. The first multi-head attention module is configured to extract sparse spectral features of the channel image to obtain a spectral feature map of the first layer of encoders, the first feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the spectral feature map of the first layer of encoders to obtain an aggregated feature map of the first layer of encoders, and the down-sampling layer is configured to down-sample the aggregated feature map of the first layer of encoders to obtain a feature map of the first layer of encoders. The second layer of encoders in the three layers of encoders comprises a first multi-head attention module, a first feature aggregation feedforward module and a down-sampling layer. The first multi-head attention module in the second layer of encoders is configured to extract sparse spectral features of the feature map of the first layer of encoders to obtain a spectral feature map of the second layer of encoders, the first feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the spectral feature map of the second layer of encoders to obtain an aggregated feature map of the second layer of encoders, and the down-sampling layer is configured to down-sample the aggregated feature map of the second layer of encoders to obtain a feature map of the second layer of encoders. The third layer of encoders in the three layers of encoders comprises a first multi-head attention module, a first feature aggregation feedforward module and a down-sampling layer. The first multi-head attention module in the third layer of encoders is configured to extract sparse spectral features of the feature map of the second layer of encoders to obtain a spectral feature map of the third layer of encoders, the first feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the spectral feature map of the third layer of encoders to obtain an aggregated feature map of the third layer of encoders, and the down-sampling layer is configured to down-sample the aggregated feature map of the third layer of encoders to obtain a feature map of the third layer of encoders. It can be seen that the encoder network of the embodiment encodes and down-samples the image in sequence to realize feature extraction at different scales and depths.

[0141] The bottleneck layer comprises a third multi-head attention module and a third feature aggregation feedforward module. The third multi-head attention module is configured to use a multi-head attention mechanism to perform sparse feature recovery processing on the feature map output by the third layer of encoders to obtain a second information recovery image, and the third feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the second information recovery image to obtain a spectral feature aggregation image.

[0142] The decoder network comprises three layers of decoders, the first layer of decoders in the three-layer decoder network comprises an upsampling layer, a second multi-head attention module and a second feature aggregation feedforward module. The upsampling layer of the first layer of decoders is configured to perform upsampling processing on the spectral feature aggregation image to obtain a recovery image of the first layer of decoders, the second multi-head attention module is configured to perform sparse feature recovery processing on a spliced image obtained by splicing the recovery image of the first layer of decoders and the aggregated feature map of the third layer of encoders by using a multi-head attention mechanism to obtain a first information recovery image of the first layer of decoders, and the second feature aggregation feedforward module of the first layer of decoders is configured to perform aggregation processing on the features in the neighborhood space of the first information recovery image of the first layer of decoders to obtain a spectral recovery image of the first layer of decoders. The second layer of decoders in the three-layer decoder network comprises an upsampling layer, a second multi-head attention module and a second feature aggregation feedforward module. The third sampling layer of the second layer of decoders is configured to perform upsampling processing on the spectral recovery image of the first layer of decoders to obtain a recovery image of the second layer of decoders, the second multi-head attention module is configured to perform sparse feature recovery processing on a spliced image obtained by splicing the recovery image of the second layer of decoders and the aggregated feature map of the second layer of encoders by using a multi-head attention mechanism to obtain a first information recovery image of the second layer of decoders, and the second feature aggregation feedforward module of the second layer of decoders is configured to perform aggregation processing on the features in the neighborhood space of the first information recovery image of the second layer of decoders to obtain a spectral recovery image of the second layer of decoders. The third layer of decoders in the three-layer decoder network comprises an upsampling layer, a second multi-head attention module and a second feature aggregation feedforward module. The upsampling layer of the third layer of decoders is configured to perform upsampling processing on the spectral recovery image of the second layer of decoders to obtain a recovery image of the third layer of decoders, the second multi-head attention module is configured to perform sparse feature recovery processing on a spliced image obtained by splicing the recovery image of the third layer of decoders and the aggregated feature map of the first layer of encoders by using a multi-head attention mechanism to obtain a first information recovery image of the third layer of decoders, and the second feature aggregation feedforward module of the third layer of decoders is configured to perform aggregation processing on the features in the neighborhood space of the first information recovery image of the third layer of decoders to obtain a spectral recovery image of the third layer of decoders. It can be seen that the input image of the decoder in the embodiment is from the mixing of deep features and the same scale of encoded layer features, and the spectral information is processed for recovery, and the upsampling-decoding processing is also performed step by step to realize multi-scale feature fusion.

[0143] The position decoding layer is configured to perform fusion processing on the image of the preset channel number output by the last layer of decoders in the decoder network based on the spatial position features to obtain a multispectral image, so as to realize correct spatial information recovery.

[0144] As in an example, the channel image is an image of HxWxC, the aggregated feature map of the first layer encoder is an image of HxWxC, the feature map output after the down-sampling of the down-sampling layer of the first layer encoder is an image of Hx2Wx2C, the aggregated feature map of the second layer encoder is an image of Hx2Wx2C, the feature map output after the 2 times down-sampling of the down-sampling layer of the second layer encoder is an image of Hx4Wx4C, the aggregated feature map of the third layer encoder is an image of Hx4Wx4C, the feature map output after the 4 times down-sampling of the down-sampling layer of the third layer encoder is an image of Hx8Wx8C, and the bottleneck layer outputs a spectral feature aggregated image of Hx8Wx8C. The up-sampling layer of the first layer decoder first up-samples the spectral feature aggregated image of Hx8Wx8C to obtain a recovery image of Hx4Wx4C, then inputs the recovery image of Hx4Wx4C and the splicing image of the aggregated feature map of Hx4Wx4C of the third layer encoder into the second multi-head attention module of the first layer decoder to output a first information recovery image of Hx4Wx4C, and the second feature aggregation feedforward module of the first layer decoder processes the first information recovery image of Hx4Wx4C to output a spectral recovery image of Hx4Wx4C. The up-sampling layer of the second layer decoder first up-samples the spectral recovery image of Hx4Wx4C to obtain a recovery image of Hx2Wx2C, then inputs the recovery image of Hx2Wx2C and the splicing image of the aggregated feature map of Hx2Wx2C of the second layer encoder into the second multi-head attention module of the second layer decoder to output a first information recovery image of Hx2Wx2C, and the second feature aggregation feedforward module of the second layer decoder processes the first information recovery image of Hx2Wx2C to output a spectral recovery image of Hx2Wx2C. The up-sampling layer of the third layer decoder first up-samples the recovery image of Hx2Wx2C to obtain a recovery image of HxWxC, then inputs the recovery image of HxWxC and the splicing image of the aggregated feature map of HxWxC of the first layer encoder into the second multi-head attention module of the third layer decoder to output a first information recovery image of HxWxC. The second feature aggregation feedforward module of the third layer decoder processes the first information recovery image of HxWxC to output a spectral recovery image of HxWxC.

[0145] In addition, it is not difficult to see that, in the encoder network, the input image is first encoded (multi-head attention module and feature aggregation feedforward module), and then down-sampled to improve the feature loss problem in the down-sampling process. In the decoder network, the input image is first up-sampled, and then decoded (multi-head attention module and feature aggregation feedforward module) to improve the expression of the features.

[0146] Compared with the general noise removal processing step, the pre-processing is performed, that is, before the spectral demodulation reconstruction, but it is inevitable to cause the problem of spectral information loss of the complete spatial resolution image after noise removal, that is, it is not conducive to high-quality imaging in a low-illumination scene. Therefore, the U-Net image imaging model of the synchronous processing is adopted in the embodiment. The U-Net image imaging model uses the feature aggregation feedforward module to synchronously perform the noise removal and spectral reconstruction processing, that is, the noise suppression processing is performed while the spectral reconstruction is performed, so as to avoid the spectral information loss. The error accumulation problem caused by the noise removal processing and the spectral reconstruction processing in sequence can also be avoided.

[0147] In addition, the image imaging device can also obtain a color image based on the multispectral image.

[0148] Specifically, when the broadband spectral modulation capability of the light filtering array of the image sensor is only for the visible light band, the image imaging module projects the multispectral image output by the spectral reconstruction module onto the standard CIE (International Commission on Illumination)-RGB spectral response curve to generate a high-quality RGB color image.

[0149] As an optional implementation, when the broadband spectral modulation capability of the light filtering array of the image sensor is extended from the visible light band to the near-infrared band, the image imaging module can also directly obtain a near-infrared image from the multispectral image based on the wavelength information of the near-infrared band while projecting the multispectral image output by the spectral reconstruction module onto the standard CIE (International Commission on Illumination)-RGB spectral response curve to generate a high-quality RGB color image.

[0150] In this way, the embodiment can directly obtain a near-infrared image from the multispectral image, which not only can realize high-quality color RGB imaging in a low-illumination scene and output a near-infrared image, but also can effectively avoid the problem of infrared light leakage from the RGB-IR near-infrared camera to the visible RGB image.

[0151] As an implementation, the image imaging device can also obtain a near-infrared image from the multispectral image based on the wavelength information of the near-infrared light.

[0152] Specifically, when the broadband spectral modulation capability of the light filtering array of the image sensor is extended from the visible light band to the near-infrared band, the image imaging module can directly obtain a near-infrared image from the multispectral image based on the wavelength information of the near-infrared band while projecting the multispectral image output by the spectral reconstruction module onto the standard CIE (International Commission on Illumination)-RGB spectral response curve to generate a high-quality RGB color image.

[0153] Therefore, the embodiment can directly obtain the near-infrared image from the multispectral image, can realize high-quality color RGB imaging under low illumination, can output the near-infrared image, and can effectively avoid the problem of infrared light leakage from the RGB-IR near-infrared camera to the visible RGB image.

[0154] In addition, the U-Net image imaging model can be trained using a low-illumination training sample set. Specifically, each low-illumination training sample in the low-illumination training sample set includes a first full spatial resolution image sample object and a corresponding denoised first multispectral image sample object, and the first mosaic image sample object corresponding to the first full spatial resolution image sample object is obtained by the light filter array in a low-illumination environment. It can be understood that the first full spatial resolution image sample corresponds to the first mosaic image sample object obtained by the light filter array in a low-illumination environment, i.e., a weak light environment, such as at night. The first full spatial resolution image sample object is a 3 or 4 channel image.

[0155] Alternatively, as an optional implementation, in addition to constructing the low-illumination training sample set, a regular-illumination training sample set can also be constructed.

[0156] In the regular-illumination training sample set, each regular-illumination training sample includes a second full spatial resolution image sample object and a corresponding denoised second multispectral image sample object, and the second mosaic image sample object corresponding to the second full spatial resolution image sample object is obtained by the light filter array in a regular-illumination environment. It can be understood that the second full spatial resolution image sample corresponds to the second mosaic image sample object obtained by the light filter array in a regular-illumination environment. The regular-illumination environment is a daytime natural light environment, or can also be a light environment provided by a residence, an office building, a commercial building, a public building, a sports ground, etc. that meets a specified illumination standard value. The second full spatial resolution image sample object is a 3 or 4 channel image.

[0157] During specific training, the training sample set is composed of the low-illumination training sample set and the regular-illumination training sample set. After the initial U-Net image imaging model is constructed, the model is trained using the aforementioned training sample set and loss function, and the network weight parameters and gradient values of the initial U-Net image imaging model are iteratively updated until the training is terminated to obtain the trained U-Net image imaging model.

[0158] Therefore, the U-Net image imaging model finally obtained through training can be used not only for imaging processing in a normal regular-illumination scene, but also for image imaging processing in a low-illumination scene.

[0159] In addition, the conventional illumination training samples are added to the training sample set, and the spectral reconstruction capability can be avoided from being reduced due to overfitting of the network model in the training process.

[0160] Based on the same inventive concept, refer to Figure 10 The application also provides an image imaging device, comprising:

[0161] An image acquisition module is configured to acquire a mosaic image.

[0162] A spectral reconstruction module is configured to perform denoising and spectral reconstruction synchronous processing on the mosaic image by using a U-Net image imaging model to obtain a multispectral image.

[0163] The encoder network comprises a plurality of encoders, each encoder comprises a down-sampling layer, and at least one encoder further comprises a first feature aggregation feedforward module configured to perform aggregation processing on the features in the neighborhood space of the image input into the first feature aggregation feedforward module.

[0164] The decoder network comprises a plurality of decoders, each decoder comprises a down-sampling layer, and at least one decoder further comprises a second feature aggregation feedforward module configured to perform aggregation processing on the features in the neighborhood space of the image input into the second feature aggregation feedforward module.

[0165] In addition, the application also provides a computer storage medium, and the computer storage medium stores a computer program. When the computer program is executed by a processor, the steps of the image imaging method described above are implemented. Therefore, the description will not be repeated here. In addition, the beneficial effects of using the same method will not be described again. For technical details not disclosed in the computer readable storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application. For example, the program instructions can be deployed to execute on one computing device, or on multiple computing devices located in one place, or on multiple computing devices distributed in multiple places and interconnected through a communication network.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The above-mentioned program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0167] In addition, it should be noted that the above-described device embodiments are only schematic, and units described as separate components can or can not be physically separate, and components shown as units can or can not be physical units, i.e., they can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the device embodiments provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement without creative labor.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by software and necessary general hardware, and of course, it can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and specific hardware structures for realizing the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disc, U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disc or optical disc, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various embodiments of the present application.

[0169] The above merely describes optional embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural changes made according to the content of the present application and the drawings, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image imaging method characterized by, The method comprises: acquiring a mosaic image; performing denoising and spectral reconstruction on the mosaic image simultaneously by a U-Net image imaging model to obtain a multispectral image; the U-Net image imaging model comprises an encoder network and a decoder network; wherein the encoder network comprises a plurality of encoders, each of the encoders comprises a down-sampling layer, and at least one of the encoders further comprises a first feature aggregation feedforward module configured to aggregate features in a neighborhood space of an image input to the first feature aggregation feedforward module; the decoder network comprises a plurality of decoders, each of the decoders comprises an up-sampling layer, and at least one of the decoders further comprises a second feature aggregation feedforward module configured to aggregate features in a neighborhood space of an image input to the second feature aggregation feedforward module; the U-Net image imaging model further comprises a channel expansion layer, a position encoding layer, and a position decoding layer; the channel expansion layer is configured to expand the mosaic image into a channel image with a preset number of channels; the position encoding layer is configured to perform spatial position encoding on the channel image to obtain spatial position features of the channel image; the position decoding layer is configured to fuse a channel image with the preset number of channels output by a last decoder in the decoder network based on the spatial position features to obtain the multispectral image.

2. The image imaging method according to claim 1, characterized by, at least one of the encoders further comprises a first multi-head attention module configured to extract sparse spectral features of an image input to the first multi-head attention module to obtain a spectral feature map, and the first feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the spectral feature map; at least one of the decoders further comprises a second multi-head attention module configured to use a multi-head attention mechanism to perform sparse feature recovery on an image input to the second multi-head attention module to obtain a first information recovery image, and the second feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the first information recovery image.

3. The image imaging method according to claim 2, characterized by, The U-Net image imaging model further comprises a bottleneck layer comprising a third multi-head attention module and a third feature aggregation feedforward module; the third multi-head attention module is configured to use a multi-head attention mechanism to perform sparse feature recovery on an image output by a last encoder layer of the encoder network to obtain a second information recovery image; the third feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the second information recovery image to obtain a spectral feature aggregation image, and input the spectral feature aggregation image to a first decoder layer of the decoder network.

4. The image imaging method according to any one of claims 2 to 3, characterized in that, In at least one of the encoders, the first multi-head attention module is configured to extract sparse spectral features of an image input into the encoder to obtain a spectral feature map, and the first feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the spectral feature map to obtain an aggregated feature map, and the down-sampling layer is configured to down-sample the aggregated feature map to obtain a feature map; wherein the aggregated feature map is also input into a corresponding decoder. In at least one of the decoders, the up-sampling layer is configured to up-sample an image input into the up-sampling layer to obtain a restored image, the second multi-head attention module is configured to use a multi-head attention mechanism to perform sparse feature restoration processing on a spliced image obtained by splicing the restored image and the corresponding aggregated feature map to obtain a first information restored image, and the second feature aggregation feedforward module is configured to aggregate features in a neighborhood space of the first information restored image to obtain a spectral restored image.

5. The image imaging method according to claim 4, characterized by, All of the encoders in the encoder network include the first multi-head attention module and the first feature aggregation feedforward module; and / or All of the decoders in the decoder network include the second multi-head attention module and the second feature aggregation feedforward module.

6. The image imaging method of claim 5, wherein, The encoder network includes three layers of the encoders, and the decoder network includes three layers of the decoders.

7. An image imaging apparatus characterized by comprising: Comprising: an image acquisition module configured to acquire a mosaic image; a spectral reconstruction module configured to perform denoising and spectral reconstruction synchronous processing on the mosaic image by using a U-Net image imaging model to obtain a multi-spectral image; the U-Net image imaging model includes an encoder network and a decoder network; wherein the encoder network includes a plurality of encoders, each of the encoders includes a down-sampling layer, and at least one of the encoders further includes a first feature aggregation feedforward module configured to aggregate features in a neighborhood space of an image input into the first feature aggregation feedforward module; the decoder network includes a plurality of decoders, each of the decoders includes an up-sampling layer, and at least one of the decoders further includes a second feature aggregation feedforward module configured to aggregate features in a neighborhood space of an image input into the second feature aggregation feedforward module; the U-Net image imaging model further includes a channel expansion layer, a position encoding layer, and a position decoding layer; the channel expansion layer is configured to expand the mosaic image into a channel image with a preset number of channels; the position encoding layer is configured to perform spatial position encoding processing on the channel image to obtain spatial position features of the channel image; the position decoding layer is configured to perform fusion processing on an image with a preset number of channels output by a last layer of the decoder network based on the spatial position features to obtain the multi-spectral image.

8. An image signal processing apparatus characterized by comprising: Comprising: a processor, a memory, and a computer program stored in the memory, wherein the computer program is run by the processor to implement the image imaging method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the image imaging method in any one of claims 1 to 6.

Images