Optical system, configuration method thereof and image denoising method

Abstract

The invention discloses an optical system, a configuration method thereof and an image denoising method, the configuration method comprises the following steps: a sample data set is acquired, each training sample in the sample data set comprises original image information and N pieces of noise image information, the ith noise image information in the N pieces of noise image information is information obtained by performing i times of noise adding processing on the original image information, i is greater than or equal to 1 and less than or equal to N, and i and N are integers; a configuration of the light modulation component is trained based on the sample data set. According to the optical system configuration method, the configuration of the optical system is trained through the original image information and the multiple pieces of noise-added image information corresponding to the original image information, so that the configuration of the optical system capable of restoring the original image information based on the noise image information is determined.

Description

Technical Field

[0001] The present invention relates to the technical field of optical design, and in particular to an optical system and a configuration method thereof, and an image denoising method. Background Art

[0002] In the field of machine learning, the diffusion model effectively learns the probability distribution in a given data set through its unique forward process, reverse process and sampling process. Among them, the denoising diffusion model gradually adds noise to the image until the image converges with the Gaussian noise distribution, and then samples samples close to the original image from the Gaussian noise distribution through the reverse elimination diffusion process. This model shows great application potential in computer vision, such as image denoising, restoration, super-resolution enhancement and image generation, and mainly relies on the electrical computing method of the central processing unit (CPU) or the image processing unit (GPU) to achieve it.

[0003] However, as the main carriers of model operation, the high energy consumption of CPU and GPU has become an issue that cannot be ignored. In addition, in the process of electrical data processing, the transmission and processing of signals are often accompanied by high time consumption and delay, which is particularly prominent in scenarios such as embedded devices that pursue real-time response and low latency. In order to deploy denoising diffusion models in these restricted environments, researchers have tried to massively simplify the model structure or truncate the input and output data, but these methods often sacrifice model accuracy and may even cause the application to fail to function properly. Summary of the invention

[0004] One of the purposes of the present invention is to provide a configuration method of an optical system, wherein the optical system includes a light modulation component, and the training method includes:

[0005] Acquire a sample data set, wherein each training sample in the sample data set includes original image information and N noise image information, wherein the i-th noise image information in the N noise image information is information obtained after performing i-time noise addition processing on the original image information, 1≤i≤N, i and N are integers;

[0006] The configuration of the light modulation component is trained based on the sample data set.

[0007] In some technical solutions of the present invention, the step of training the configuration of the light modulation component based on the sample data set includes:

[0008] Based on the Nth noisy image information, generate N denoised image information;

[0009] The configuration of the light modulation component is trained based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information.

[0010] In some technical solutions of the present invention, the j-th denoised image information among the N denoised image information is information obtained after the N-th noisy image information is processed j times by the light modulation component with the same configuration, 1≤j≤N, and j is an integer;

[0011] The step of training the configuration of the light modulation component based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information includes:

[0012] Determine a kth first loss value based on the kth denoised image information and the Nkth noise image information, wherein l≤k≤N, and the 0th noise image information is the original image information;

[0013] Determining a comprehensive loss value based on all first loss values;

[0014] Based on the comprehensive loss value, the configuration of the light modulation component is trained.

[0015] In some technical solutions of the present invention, the lth denoised image information among the N denoised image information is information obtained after the l-1th denoised image information is processed once by the current light modulation component, and the configuration of the current light modulation component is determined based on the training of the l-1th denoised image information, wherein l≤l≤N, and the 0th denoised image information is the Nth noise image information;

[0016] The step of training the configuration of the light modulation component based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information includes:

[0017] Based on the mth denoised image information and the Nmth noise image information, determining the mth second loss value, 1≤m≤N, m is an integer, and the 0th noise image information is the original image information;

[0018] Based on the mth second loss value, the configuration of the light modulation component is trained.

[0019] In some technical solutions of the present invention, the light modulation component includes at least one light modulation element, and the training of the configuration of the light modulation component based on the sample data set includes at least one of the following:

[0020] Based on the sample data set, updating a configuration parameter of at least one of the light modulation elements;

[0021] Based on the sample data set, the relative position relationship between the light modulation elements is updated.

[0022] In some technical solutions of the present invention, the light modulation element includes at least one of the following: a refractive optical element, a diffractive optical element, a scattering medium element and a metasurface element.

[0023] In some technical solutions of the present invention, the optical modulation component includes a plurality of optical modulation elements, and the plurality of optical modulation elements are arranged in sequence. Except for the first optical modulation element, each of the optical modulation elements is on the reflection path or transmission path of the previous optical modulation element.

[0024] In some technical solutions of the present invention, the noise adding process is to add random noise to the original image information or the noisy image information.

[0025] A second object of the present invention is to provide an optical system, comprising:

[0026] A light modulation component, wherein the light modulation component is configured according to any of the above methods;

[0027] Photosensitive element;

[0028] The light guide component is used to couple the output signal of the light modulation component to the input end of the light modulation component for a preset number of times and then couple it to the photosensitive element.

[0029] In some technical solutions of the present invention, the optical modulation component includes a plurality of optical modulation elements, and the plurality of optical modulation elements are arranged in sequence. Except for the first optical modulation element, each of the optical modulation elements is on the reflection path or transmission path of the previous optical modulation element.

[0030] The third object of the present invention is to provide an image denoising method based on an optical system, wherein the optical system includes a light modulation component, a photosensitive element and a light guide component;

[0031] importing the image information to be processed into the optical system;

[0032] Processing the image information using the light modulation component;

[0033] By using the light guide component, after the output information of the image information processed by the light modulation component is coupled to the input end of the light modulation component for a preset number of times, it is coupled to the photosensitive element.

[0034] Compared with the prior art, the configuration method of the optical system provided by the present invention obtains a sample data set, each training sample in the sample data set includes original image information and N noise image information, wherein the i-th of the N noise image information is information obtained after performing i-times of noise addition processing on the original image information, 1≤i≤N, i and N are integers; and the configuration of the light modulation component is trained based on the sample data set. The configuration method of the optical system provided by the present invention trains the configuration of the optical system through the original image information and its corresponding multiple noise-added image information that are subjected to step-by-step noise addition processing, so as to determine the configuration of the optical system that can restore the original image information based on the noise image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic structural diagram of an optical system in one embodiment of the present invention.

[0036] Figure 2 It is a schematic structural diagram of an optical system based on a transmission-type light modulation element in one embodiment of the present invention.

[0037] Figure 3 It is a schematic structural diagram of an optical system based on a reflective light modulation element in one embodiment of the present invention.

[0038] Figure 4 It is a schematic diagram of the steps of a method for configuring an optical system in one embodiment of the present invention.

[0039] Figure 5 It is a schematic diagram of a sample data set generation process in one embodiment of the present invention.

[0040] Figure 6 2 is a schematic diagram of step S2 in one embodiment of the present invention.

[0041] Figure 7 Schematic diagram of a process for generating N denoised image information in one embodiment of the present invention.

[0042] Figure 8 2 is a schematic diagram of step S22 in an embodiment of the present invention.

[0043] Fig. 9 4 is a schematic diagram of a process for generating N denoised image information in another embodiment of the present invention.

[0044] Fig.10 It is a schematic diagram of step S22 in another embodiment of the present invention.

[0045] Fig.11 It is a flowchart of a method for configuring an optical system in one embodiment of the present invention.

[0046] Fig.121 is a schematic diagram of the steps of an image denoising method based on an optical system in one embodiment of the present invention.

[0047] Fig.13 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0049] It should be noted that the term "comprise" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0050] An embodiment of the present invention provides an optical system, which can be used to generate an image based on an input noisy image, or to achieve effects such as denoising and repairing the noisy image, and then output a new image.

[0051] The optical system of the embodiment of the present application can be a system formed by combining multiple optical elements in a certain order, and is used for optical information processing, such as image denoising, image restoration, super-resolution enhancement, and image generation.

[0052] like Figure 1 As shown, the optical system 200 includes a light modulation component 21, a light guide component 22, and a photosensitive element 23. The light modulation component 21 is used to modulate the light signal of the image to be processed input into the optical system; the light guide component 22 is used to couple the output signal of the light modulation component 21 to the input end of the light modulation component 21 for a preset number of times, and then couple to the photosensitive element, so as to convert the light signal into an electrical signal through the photosensitive element 23, and then obtain the output image.

[0053] Here, the optical modulation component 21 is used to modulate the optical signal of the image to be processed which is input into the optical system, and realizes image denoising, restoration, super-resolution enhancement and image generation by changing the intensity, phase, frequency or polarization state of the light; the light guide component 22 is used to guide the light to propagate along a predetermined path, and specifically, the optical signal output from the output end of the optical modulation component 21 can be coupled to the input end of the optical modulation component 21, or the optical signal output from the output end of the optical modulation component 21 can be coupled to the photosensitive element 23.

[0054] In the embodiment of the present application, the light modulation component 21 may include at least one of the following: a refractive optical element, a diffractive optical element, a scattering medium element, and a metasurface element, and the number of each optical element may be one or more, which is not limited in the present application. Specifically, the light modulation component 21 modulates the optical signal of the input image through the refractive optical element, the diffractive optical element, the scattering medium element, and / or the metasurface element.

[0055] Exemplarily, refractive optical elements include but are not limited to lenses or prisms made of optical glass, optical plastics, optical crystals and other materials; diffractive optical elements include but are not limited to two-step or multi-step diffractive optical elements, gratings, Dammann gratings, metasurfaces, holograms, diffusers, phase masks, intensity masks, spatial light modulators and the like; scattering medium elements include but are not limited to frosted glass and the like; metasurface elements refer to an artificial layered material whose size is less than or approximately equal to the wavelength, which can be regarded as the two-dimensional counterpart of metamaterials. Metasurface elements can achieve the regulation of the polarization, phase, amplitude, frequency, propagation mode and other characteristics of electromagnetic waves by showing sub-wavelength nanostructure units, and realize the characteristics of beam shaping, beam deflection, superlens, superhologram, optical rotation, anti-reflection and anti-reflection.

[0056] In some exemplary embodiments, the light modulation component 21 includes only one light modulation element 211 .

[0057] In some exemplary embodiments, the optical modulation component 21 includes a plurality of optical modulation elements 211 , and the plurality of optical modulation elements 211 are arranged in sequence. Except for the first optical modulation element 211 , each optical modulation element 211 is on the reflection path or transmission path of the previous optical modulation element 211 .

[0058] In some specific embodiments, the light modulation component 21 includes a plurality of transmissive light modulation elements, that is, the light modulation element 211 is a transmissive light modulation element. Figure 2 As shown, the optical modulation component 21 includes four transmission-type optical modulation elements 211 (denoted as 211a, 211b, 211c, and 211d), wherein the optical modulation element 211a, the optical modulation element 211b, the optical modulation element 211c, and the optical modulation element 211d are arranged in sequence, except for the first optical modulation element 211a, the optical modulation element 211b is on the transmission path of the optical modulation element 211a, the optical modulation element 211c is on the transmission path of the optical modulation element 211b, and the optical modulation element 211d is on the transmission path of the optical modulation element 211c, that is, the optical signal is transmitted through the optical modulation element 211a, the optical modulation element 211b, the optical modulation element 211c, and the optical modulation element 211d in sequence.

[0059] Specifically, the light modulator 211a, the light modulator 211b, the light modulator 211c, and the light modulator 211d can be transmissive metasurface elements, which can precisely control the transmitted light without changing the geometric structure of the optical path.

[0060] In some exemplary embodiments, two or more optical modulation elements 211 may be independently separated or integrated. When integrated, the two or more optical modulation elements may be combined by thermal bonding, plasma bonding, adhesive bonding, etc. Alternatively, another optical modulation element 211 may be directly prepared on one side of an optical modulation element 211. For example, taking a metasurface element as an example, first, an optical film layer is formed on the substrate side or the nanostructure unit side of the metasurface element, and then the optical film layer is etched into a metasurface by a photolithography process.

[0061] In some other specific embodiments, the light modulation assembly 21 includes a plurality of reflective light modulation elements, that is, the light modulation element 211 is a reflective light modulation element. Figure 3 As shown, the optical modulation component 21 includes four reflective optical modulation elements 211 (denoted as 211e, 211f, 211g, and 211h), wherein the optical modulation element 211e, the optical modulation element 211f, the optical modulation element 211g, and the optical modulation element 211h are arranged in sequence. Except for the first optical modulation element 211e, the optical modulation element 211f is on the reflection path of the optical modulation element 211e, the optical modulation element 211g is on the reflection path of the optical modulation element 211f, and the optical modulation element 211h is on the reflection path of the optical modulation element 211g, that is, the optical signal is reflected by the optical modulation element 211e, the optical modulation element 211f, the optical modulation element 211g, and the optical modulation element 211h in sequence.

[0062] Specifically, the light modulator 211e, the light modulator 211f, the light modulator 211g, and the light modulator 211h may be reflective metasurface elements, which can precisely control the reflected light without occupying additional space.

[0063] In the embodiment of the present application, a focusing lens may be further disposed on the light incident side of the light modulation element 211 .

[0064] The light guide component 22 is used to couple the output signal of the light modulation component 21 to the input end or the photosensitive element of the light modulation component 21, and is specifically used to couple to the photosensitive element after being coupled to the input end of the light modulation component 21 a preset number of times.

[0065] In some exemplary embodiments, the light guide assembly 22 includes a plurality of light deflection elements 221 and a beam splitter 222. The beam splitter 222 has a first surface, and the beam splitter 222 can be used to split the incident light beam on the first surface into reflected light and transmitted light according to a certain ratio.

[0066] Specifically, Figure 1 As shown, the optical signal outputted from the output end of the optical modulation component 21 enters the first surface of the beam splitter 222 after being reflected by the plurality of optical deflection elements 221 of the light guide component 22, and the beam splitter 222 reflects or transmits the incident light beam on the first surface. At the same time, the input end of the optical modulation component 21 is located in the reflection light path of the beam splitter 222, and the photosensitive element 23 is located in the transmission light path of the beam splitter 222; or, the input end of the optical modulation component 21 is located in the transmission light path of the beam splitter 222, and the photosensitive element 23 is located in the reflection light path of the beam splitter 222.

[0067] Here, the beam splitter can adopt a voltage-controlled beam splitter, which is a dynamically adjustable beam splitter that uses electro-optical effects (such as Pockels effect or liquid crystal effect) to change the reflectivity and transmittance of light. By applying different voltages, the ratio of reflected light to transmitted light can be adjusted in real time. The embodiment of the present application can apply a corresponding voltage at a preset time according to actual needs, so that the incident light on the first surface of the beam splitter is completely reflected or completely transmitted. Among them, the preset time can be determined according to a preset number of times.

[0068] It can be understood that the embodiment of the present application does not limit the positional relationship between the beam splitter 222 and the plurality of light deflection elements 221, and only needs to ensure that the incident light on the first surface of the beam splitter 222 is completely reflected or completely transmitted, and finally coupled to the input end of the light modulation component 21 again or directly coupled to the photosensitive element 23. That is, the light signal output from the output end of the light modulation component 21 can directly enter the first surface of the beam splitter 222, or can be guided to the first surface of the beam splitter 222 after being reflected by the plurality of light deflection elements 221; and the light reflected by the first surface of the beam splitter 222 can directly enter the input end of the light modulation component 21 or directly enter the photosensitive element 23, or can be guided to the input end of the light modulation component 21 or the photosensitive element 23 after being reflected by the plurality of light deflection elements 221.

[0069] For example, the optical signal outputted from the output end of the optical modulation component 21 first enters the first surface of the beam splitter 222 of the optical guide component 22, and is reflected or transmitted by the beam splitter 222; at the same time, the plurality of optical deflection elements 221 of the optical guide component 22 are sequentially located in the reflection optical path of the beam splitter 222, and the photosensitive element 23 is located in the transmission optical path of the beam splitter 222. For another example, the plurality of optical deflection elements 221 of the optical guide component 22 are sequentially located in the transmission optical path of the beam splitter 222, and the photosensitive element 23 is located in the reflection optical path of the beam splitter 222. Finally, the plurality of optical deflection elements 221 of the optical guide component 22 reflect the optical signal to the input end of the optical modulation component 21.

[0070] Here, the light guide component 22 guides the light signal output from the output end of the light modulation component 21 to the input end of the light modulation component 21 again, or guides the light signal output from the output end of the light modulation component 21 to the photosensitive element 23. In this process, the light guide component 22 keeps the characteristics of the light (such as brightness, direction, etc.) unchanged or only slightly changed.

[0071] In one embodiment, the light deflection element 221 may be a reflector, a prism, etc. In a specific embodiment, the light deflection element 221 is a right-angle prism.

[0072] It should be noted that the light guide component 22 can also be implemented by other components or component combinations, and this application does not limit this. For example, the light guide component 22 can be composed of a series of optical components, such as lenses, reflectors, deflection mirrors, etc. The function of these components is to focus, reflect or refract the light output by the light modulation component 21 to guide the light to propagate along a predetermined path.

[0073] It is understandable that in the process of changing the propagation direction of the light beam, it may be necessary to use one light deflection element 221 to achieve the deflection of the light beam at a preset angle, or it may be necessary to use multiple light deflection elements 221 to achieve the deflection of the light beam at a preset angle. There is no specific limitation on the number of light deflection elements 221 used and / or the deflection angle of the light deflection element 221.

[0074] The photosensitive element 23 is an electronic device that converts light signals into electrical signals, and may specifically be a complementary metal oxide semiconductor (CMOS), a CMOS image sensor (CIS), a charge coupled device (CCD) or an array photodetector, etc., without any specific limitation.

[0075] When the optical system 200 is working, the light modulation component 21 modulates the received initial incident light carrying noise image information, and the processed light beam is coupled to the input end of the light modulation component 21 by the light guide component 22 and modulated again by the light modulation component 21, and the cycle repeats. Finally, after the output signal of the light modulation component 21 is coupled to the input end of the light modulation component 21 for a preset number of times, the light guide component 22 couples the light beam to the photosensitive element 23 to obtain an output image.

[0076] It should be noted that the optical system 200 can be configured according to the configuration method described in any of the following embodiments. Specifically, the light modulation component 21 of the optical system 200 can be configured according to the configuration method described in any of the following embodiments.

[0077] like Figure 4 As shown, an embodiment of the present invention provides a configuration method for an optical system. The configuration method can be used to train and determine the configuration of any of the above-mentioned optical systems, and is specifically used to determine the configuration of the light modulation component in the optical system. Figure 4 As shown, the configuration method includes but is not limited to the following steps.

[0078] Step S1: obtaining a sample data set, wherein each training sample in the sample data set includes original image information and N noise image information, wherein the i-th noise image information in the N noise image information is information obtained after performing i-time noise addition processing on the original image information, 1≤i≤N, i and N are integers;

[0079] Here, the noise image information is obtained by performing noise addition processing on the original image information. The original image information is subjected to noise addition processing once to generate the first noise image information, the original image information is subjected to noise addition processing twice to generate the second noise image information, and so on, the original image information is subjected to noise addition processing N times to generate the Nth noise image information.

[0080] The following, such as Figure 5 As shown, the generation process of the sample data set is explained with a specific example.

[0081] First, the original image information is subjected to a noise addition process to generate the first noise image information; then, the first noise image information is subjected to a noise addition process to generate the second noise image information, where the second noise image information can be generated based on the first noise image information subjected to a noise addition process; and so on, until the N-1th noise image information is subjected to a noise addition process to generate the Nth noise image information.

[0082] In this way, each original image information corresponds to N noise image information, which together with the original image information constitute a training sample including N+1 image information. Thus, taking N as 999 as an example, if the sample data set includes 1000 original image information, the entire sample data set includes 1000*1000=1000000 image information.

[0083] In the embodiment of the present application, when preparing the sample data set, appropriate original image information can be collected according to the actual application scenario of the optical system, and the collected data can be further cleaned, normalized, enhanced and other pre-processing operations, and then noise processing can be performed. Finally, the sample data set can also be divided into training set data, verification set data and test set data. The information content and information type of the original image information can be selected according to the actual application needs, and this application does not limit this.

[0084] In some exemplary embodiments, the original image information may be a grayscale image. In this embodiment, the grayscale image is used to simulate a specific physical phenomenon, that is, a single-channel grayscale image, which can be regarded as an intensity map of input single-wavelength, single-polarized light. Here, the grayscale value in the grayscale image directly corresponds to the intensity of the light. In other words, each pixel in the grayscale image represents the intensity of the single-wavelength, single-polarized light at that location.

[0085] In some other embodiments, the original image information may be an RGB image, and the RGB image may be further converted into a corresponding grayscale image.

[0086] In another specific embodiment, the original image information may be a handwritten digital image. Finally, the trained optical system may process the inputted noisy image information to obtain the handwritten digital image information.

[0087] In the embodiment of the present application, the noise addition process refers to adding noise to the original image information to simulate various interference factors in the real world. Exemplarily, the noise added in each noise addition process may be random noise. For example, the noise added in each noise addition process may be one or more of the following: Gaussian noise, salt and pepper noise, Poisson noise, random noise, etc. Preferably, the noise added in each noise addition process is Gaussian distributed random noise.

[0088] Step S2: training the configuration of the light modulation component based on the sample data set.

[0089] In the embodiment of the present application, the light modulation component is used to modulate the optical signal of the input image, and realizes image denoising, restoration, super-resolution enhancement, image generation, etc. by changing the intensity, phase, frequency or polarization state of the light. Therefore, based on the modulation effect of the light modulation component on the training sample, the training determines the configuration of the light modulation component that meets the expectation.

[0090] The light modulation component may include one or more of the following light modulation elements: a refractive optical element, a diffractive optical element, a scattering medium element, and a metasurface element, and the number of each light modulation element may be one or more, which is not limited in this application. Specifically, the light modulation component modulates the optical signal of the input image through the refractive optical element, the diffractive optical element, the scattering medium element, and / or the metasurface element.

[0091] Here, refractive optical elements include but are not limited to lenses or prisms made of optical glass, optical plastics, optical crystals and other materials; diffractive optical elements include but are not limited to two-step or multi-step diffractive optical elements, gratings, Dammann gratings, metasurfaces, holograms, diffusers, phase masks, intensity masks, spatial light modulators, etc., and scattering medium elements include but are not limited to frosted glass, etc.; metasurface elements refer to an artificial layered material with a size less than or approximately equal to the wavelength, which can be regarded as the two-dimensional counterpart of metamaterials. Metasurface elements can achieve the regulation of the polarization, phase, amplitude, frequency, propagation mode and other characteristics of electromagnetic waves by showing sub-wavelength nanostructure units, and realize beam shaping, beam deflection, superlenses, superholograms, optical rotation, anti-reflection and anti-reflection, etc.

[0092] Furthermore, in the above step S2, training the configuration of the light modulation component based on the sample data set includes at least one of the following:

[0093] Step S21': training configuration parameters of at least one of the light modulation elements based on the sample data set;

[0094] Step S22 ′: Based on the sample data set, training the relative position relationship between the light modulation elements.

[0095] In some exemplary embodiments, the configuration parameters of the light modulation element may include:

[0096] refractive index, focal length, numerical aperture and / or radius of curvature of refractive optical elements;

[0097] The focal length characteristics of the diffractive optical element, the phase function of the diffractive surface, the radial radius of each annular zone mutation point of the diffractive surface, the annular zone depth and / or the diffraction efficiency of the diffractive surface;

[0098] Scattering coefficient, absorption coefficient, anisotropy factor, particle size distribution, refractive index, thickness, scattering phase function and / or medium type of the scattering medium element;

[0099] The arrangement period, material, shape, radius, length, width, aspect ratio, height and / or rotation angle of the nanostructure units of the metasurface element.

[0100] In some exemplary embodiments, the relative positional relationship between the light modulation elements specifically includes the distance between the light modulation elements and / or the optical path setting between the light modulation elements.

[0101] Here, the optical path between the optical modulation elements is set, for example, the optical modulation component includes multiple metasurface elements, and the multiple metasurface elements are arranged in sequence. Except for the first metasurface element, each metasurface element is arranged on the reflection path or transmission path of its previous metasurface element.

[0102] Regarding the distance between the light modulators, for example, two or more light modulators can be independently separated or integrated, and the distance between the light modulators can be determined. When integrated, two or more light modulators can be combined by thermal bonding, plasma bonding, adhesive bonding, etc.; or, another light modulator can be directly prepared on one side of a light modulator, for example, taking a metasurface element as an example, first, an optical film layer is formed on the substrate side or the nanostructure unit side of the metasurface element, and then the optical film layer is etched into a metasurface by a photolithography process.

[0103] In the above step S2, when training the configuration of the light modulation component, the noise image information is processed once by the light modulation component to achieve a denoising process on the noise image, so that by optimizing the configuration of the light modulation component, the light modulation component can learn the mapping relationship between the noise image information and the original image information from the sample data.

[0104] Specifically, Figure 6 As shown, the above step S2 includes:

[0105] Step S21: generating N denoised image information based on the Nth noisy image information;

[0106] Step S22: Based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information, training the configuration of the light modulation component.

[0107] In an exemplary embodiment, the light modulation component trains the configuration of the light modulation component in units of one training sample.

[0108] like Figure 7 As shown, in the above step S21, among the N denoised image information generated based on the Nth noise image information, the jth denoised image information is the information obtained after the Nth noise image information is processed j times by the same configuration of light modulation components, 1≤j≤N, j is an integer.

[0109] Below, continue to refer to Figure 7 , the generation process of N denoised image information is explained:

[0110] First, the Nth noisy image information is input into the light modulation component, which processes it once to generate the first denoised image information; then, the first denoised image information is input into the light modulation component, which processes it once to generate the second denoised image information, where the second denoised image information can be generated based on the first denoised image information by the light modulation component after processing it once; and so on, until the N-1th denoised image information is input into the light modulation component, which processes it once to generate the Nth denoised image information.

[0111] Here, the configuration adopted by the light modulation component of the same configuration may be the initial configuration at the beginning of the light modulation component training, or the configuration of the light modulation component after the training of the previous set of training samples is completed.

[0112] Accordingly, if Figure 8 As shown, the above step S22 includes:

[0113] Step S221: determining a kth first loss value based on the kth denoised image information and the Nkth noise image information, wherein l≤k≤N, and the 0th noise image information is the original image information;

[0114] Step S222, determining a comprehensive loss value based on all first loss values;

[0115] Step S223: training the configuration of the optical modulation component based on the comprehensive loss value.

[0116] Here, the optical modulation component processes the Nth noise image information and the N-1th denoised image information based on the same configuration; the kth denoised image information is the information obtained after the Nth noise image information is processed k times by the optical modulation component, and the first loss value is used to measure the difference between the kth denoised image information and the Nkth noise image information.

[0117] In an embodiment of the present application, the first loss value can be calculated based on a loss function such as a mean square error loss function, a root mean square error loss function, a mean absolute error loss function, etc., and the present application does not limit this.

[0118] Specifically, the first loss value may be calculated based on the light field intensity corresponding to the kth denoised image information and the light field intensity corresponding to the Nkth noisy image information.

[0119] Below, taking the mean square error function as an example, the calculation of the first loss value MSE corresponding to the k-th denoised image information is explained:

[0120]

[0121] Wherein, n is the number of pixels corresponding to the kth denoised image information or the Nkth noisy image information, each pixel of the kth denoised image information or the Nkth noisy image information corresponds to one or more nanostructure units of each metasurface element in the optical modulation component, and Y i is the light field intensity of the i-th pixel of the k-th denoised image information, is the light field intensity of the i-th pixel of the Nk-th noise image information.

[0122] The comprehensive loss value may be determined based on a weighted calculation of the N first loss values. For example, when calculating the comprehensive loss value, the weight may be determined based on the number of times the denoised image information corresponding to the first loss value corresponds to the noise image information N processed by the light modulation component. The more times the light modulation component processes, the smaller the weight, so as to reduce the influence of the accumulated error after multiple processings by the light modulation component.

[0123] If the change in the comprehensive loss value is less than the first preset threshold within a certain number of consecutive rounds, the training of the optical modulation component is stopped.

[0124] Of course, the configuration of the light modulation component may be stopped when at least one of the following conditions is met:

[0125] Reach the scheduled number of training rounds;

[0126] The calculation times of the comprehensive loss value is greater than the preset times;

[0127] The training duration exceeds the preset time.

[0128] In the embodiment of the present application, the configuration of the light modulation component is trained based on the comprehensive loss value of the light modulation component in the denoising process of all noise image information in a training sample.

[0129] In an exemplary embodiment, the light modulation component trains the configuration of the light modulation component based on each of the N denoised image information.

[0130] like Fig. 9 As shown, in the above step S21, the lth denoised image information among the N denoised image information is the information obtained after the l-1th denoised image information is processed once by the current light modulation component, and the configuration of the current light modulation component is determined based on the training of the l-1th denoised image information, wherein l≤l≤N, and the 0th denoised image information is the Nth noise image information.

[0131] Below, continue to refer to Fig. 9 , the generation process of N denoised image information is explained:

[0132] First, the Nth noisy image information is input into the optical modulation component 1, which is processed once by the optical modulation component 1 to generate the first denoised image information, and the configuration of the optical modulation component 1 is updated to obtain the optical modulation component 2; then, the first denoised image information is input into the optical modulation component 2, which is processed once by the optical modulation component 2 to generate the second denoised image information, and the configuration of the optical modulation component 1 is updated to obtain the optical modulation component 2, where the second denoised image information can be generated by processing once by the optical modulation component 2 based on the first denoised image information; and so on, until the N-1th denoised image information is input into the optical modulation component N, which is processed once by the optical modulation component N to generate the Nth denoised image information.

[0133] Accordingly, if Fig.10 As shown, the above step S22 includes:

[0134] Step S221': determining an mth second loss value based on the mth denoised image information and the Nmth noisy image information;

[0135] Step S222 ′: training the configuration of the light modulation component based on the m th second loss value.

[0136] In the embodiment of the present application, the configuration of the light modulation component is trained based on the loss value existing in the denoising process of the light modulation component on each noise image information in a training sample.

[0137] In an embodiment of the present application, the second loss value can be calculated based on a loss function such as a mean square error loss function, a root mean square error loss function, a mean absolute error loss function, etc., and the present application does not limit this.

[0138] Specifically, the second loss value can be calculated based on the light field intensity corresponding to the lth denoised image information and the light field intensity corresponding to the Nlth noisy image information.

[0139] If the second loss value changes less than a second preset threshold value within a certain number of consecutive rounds, the training of the light modulation component is stopped.

[0140] Of course, the configuration of the light modulation component may be stopped when at least one of the following conditions is met:

[0141] Reach the scheduled number of training rounds;

[0142] The calculation number of the second loss value is greater than the preset number;

[0143] The training duration exceeds the preset time.

[0144] In an embodiment of the present application, one processing of the noise image information by the light modulation component is equivalent to the inverse processing of one noise addition process, so that the Nth noise image information is processed N times by the light modulation component to approximately obtain the original image information.

[0145] In some exemplary embodiments, the configuration process of the optical system may be performed based on an optical simulation platform. Specifically, the optical response or optical model of the optical system, especially the optical response or optical model of the optical modulation component of the optical system, may be determined by an existing optical simulation method to determine the output information of the Nth noise image information or the denoised image information after being processed by the optical modulation component. Here, the optical modulation component in the optical system may be a simulated optical response relationship, and its output may be determined by calculation.

[0146] In some exemplary embodiments, the configuration process of the optical system may also be performed based on a physical optical system, that is, by constructing a physical optical element to determine the corresponding output of the Nth noisy image information or denoised image information after passing through the optical system.

[0147] When a physical optical system is used for configuration training, since the Nth noise image information needs to be processed by the light modulation component multiple times, the optical system for training may include not only the light modulation component but also a light guide component to couple the output signal of the light modulation component to the input end of the light modulation component or the photosensitive element.

[0148] Furthermore, to facilitate the training and updating of the optical modulation component, the optical modulation element that needs to be trained preferably adopts a corresponding programmable optical modulation element. For example, taking the metasurface element as an example, a programmable metasurface element can be used. When the configuration of the metasurface element needs to be updated, it can be directly adjusted based on the physical metasurface element without replacing the hardware. Here, the specific implementation scheme of the programmable metasurface element can adopt any scheme in the prior art, and the embodiments of the present application are not limited to this.

[0149] Fig.11 In a preferred embodiment shown, the complete design process of the optical system is explained.

[0150] like Fig.11 As shown, first, according to the design requirements, the usage scenario is specified, such as generating handwritten numbers. After the design requirements are determined, after the trained optical system is obtained by the training method described in any of the above embodiments of the present invention, the optical system can output a handwritten number image based on the input noise image.

[0151] Then, after determining the design requirements, the structural framework of the optical system can be determined in combination with the accuracy requirements and cost requirements, such as the number of layers and volume of the metasurface element. The fewer the number of layers, the smaller the volume, and the worse the accuracy performance. A balance needs to be found among these three. The configuration of the optical system can be gradually adjusted and optimized through the training of the optical system, and finally the optimal solution that meets the requirements is obtained.

[0152] Finally, after determining the structural framework of the optical system, it is necessary to prepare a sample data set. Specifically, the training data set can be generated by adding noise to each original image information and inputting it into the optical system (which can be a physical optical system or an optical system simulated by simulation software) for continuous training until the defined loss function converges and the final configuration of the optical system is obtained.

[0153] An embodiment of the present invention provides an image denoising method based on an optical system, wherein the optical system includes a light modulation component, a photosensitive element and a light guide component, or the optical system is the optical system described in any of the above embodiments. Fig.12 As shown, the image denoising method includes:

[0154] Step S31: importing the image information to be processed into the optical system;

[0155] Step S32: using the light modulation component to process the image information;

[0156] Step S33: using the light guide component, after coupling the output information of the image information processed by the light modulation component to the input end of the light modulation component for a preset number of times, coupling it into the photosensitive element.

[0157] Here, when the optical system is working, the light modulation component modulates the initial incident light of the received noise image information, and the processed light beam is guided by the light guide component to enter the input end of the light modulation component again and is modulated by the light modulation component again, and the cycle is repeated. Finally, after the preset conditions are met (such as the output information of the light modulation component is coupled to the input end of the light modulation component for a preset number of times), the light guide component couples the output information of the light modulation component to the photosensitive element, thereby obtaining the output image information.

[0158] The optical system of the embodiment of the present application can be used to perform noise reduction processing on the image information input therein. Specifically, the process of forming the sample data set can be understood as the process of gradually adding noise to the original image information, and the working process of the optical system can be understood as the process of gradually reducing the noise image information.

[0159] Of course, in addition to performing noise reduction processing on noisy image information input thereto, the optical system can also be used to perform decryption processing on encrypted image information input thereto, and this application does not impose any specific restrictions on this.

[0160] The above-mentioned various implementation modes, embodiments or specific examples provided by the present invention can be combined with each other to finally form multiple better implementation modes.

[0161] Based on the above embodiments, the present invention implements denoising based on a trained optical system, which has the following advantages compared to the traditional method of implementing image denoising based on an electrical neural network:

[0162] 1. Low energy consumption: Traditional electrical neural networks require high-performance CPUs and GPUs for algorithm processing, which consume hundreds of watts of electrical power. However, the trained optical system of the present invention only needs to input a single-wavelength, single-polarization incident light beam that meets the requirements. The entire metasurface element combination does not require power supply at all, and the overall energy consumption is about a few watts to tens of watts.

[0163] 2. Low time consumption: Traditional electrical neural networks require time to execute algorithms, while the optical system's metasurface elements take a signal processing time close to the thickness of the metasurface element / the speed of light, which can be said to be completed instantaneously, with better real-time performance.

[0164] 3. Low latency: For electrical neural networks, data needs to interact between various components, and the interaction time is long, so data transmission may cause a certain delay. However, the optical system does not have a complex interaction process, and the propagation of the light beam is unidirectional, completing the entire process at the speed of light, and the delay between steps is very small.

[0165] like Fig.13 As shown, an embodiment of the present invention provides an electronic device 100.

[0166] The electronic device 100 may be any device for executing the optical system configuration method or the optical system-based image denoising method provided by the present invention.

[0167] The electronic device 100 may be a computer device, and the computer device may be a terminal device or a server. Fig.13 Only an exemplary structure of the electronic device 100 is shown, not all structures, and the electronic device 100 may be implemented as required. Fig.13 Partial or complete structure shown.

[0168] In an exemplary embodiment, the electronic device 100 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the configuration method of the optical system provided above.

[0169] The electronic device 100 includes at least one processor 11. The configuration method of the optical system or the image denoising method based on the optical system provided by the present invention can be applied to the processor 11 or implemented by the processor 11.

[0170] The processor 11 may be an integrated circuit chip with signal processing capability. In the process of implementing the technical solution provided by the present invention, each step of the configuration method of the optical system or the image denoising method based on the optical system may be completed by an integrated logic circuit of hardware in the processor 11 or by instructions in the form of software.

[0171] The processor 11 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 11 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor.

[0172] The software module may be located in a storage medium located in the memory 12. The processor 11 reads the information in the memory 12 and combines its hardware to complete the steps of the configuration method of the optical system or the image denoising method based on the optical system provided in the embodiment of the present invention.

[0173] The electronic device 100 includes a memory 12. The memory 12 is used to store various types of data to support the operation of the electronic device 100. Examples of such data include: any computer program for operating on a computer device.

[0174] The memory 12 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0175] When the processor 11 executes the computer program stored in the memory 12, the steps of the method for configuring the optical system or the method for image denoising based on the optical system in any technical solution of the present invention are executed.

[0176] The electronic device 100 includes a user interface 14 and at least one network interface 15. The various components in the electronic device 100 are coupled together via a bus system 13. The bus system 13 is used to achieve communication connections between these components. In addition to the data bus, the bus system 13 also includes a power bus, a control bus, and a status signal bus. To simplify the description, Figure 1 The various buses are collectively labeled as a bus system 13.

[0177] The user interface 14 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0178] An embodiment of the present invention provides a computer storage medium.

[0179] The computer storage medium stores the computer program executed by the processor mentioned above, or instructions of the configuration method of the optical system in any of the technical solutions mentioned above or the image denoising method based on the optical system.

[0180] When the processor executes the computer program, it can execute the steps of the configuration method of the optical system or the image denoising method based on the optical system in any of the above technical solutions. In addition, the description of the beneficial effects corresponding to the same method is not repeated. For technical details not disclosed in the embodiment of the computer storage medium involved in the present invention, reference can be made to the description in the embodiment of the configuration method of the optical system of the present invention.

[0181] The computer storage device may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0182] An embodiment of the present invention provides a computer program product.

[0183] The computer program product may include at least one processor and computer instructions, which may be stored in a computer storage medium. The processor reads the computer instructions from the computer storage medium, and the processor may execute the computer instructions, which may implement the description of the configuration method of the optical system or the image denoising method based on the optical system in any of the above technical solutions.

[0184] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0185] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for configuring an optical system, characterized in that: The optical system includes a light modulation component, and the training method includes: Acquire a sample data set, wherein each training sample in the sample data set includes original image information and N noise image information, wherein the i-th noise image information in the N noise image information is information obtained after performing i-time noise addition processing on the original image information, 1≤i≤N, i and N are integers; The configuration of the light modulation component is trained based on the sample data set.

2. The method according to claim 1, characterized in that The step of training the configuration of the light modulation component based on the sample data set comprises: Based on the Nth noisy image information, generate N denoised image information; The configuration of the light modulation component is trained based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information.

3. The method according to claim 2, characterized in that The j-th denoised image information among the N denoised image information is information obtained after the N-th noisy image information is processed j times by the light modulation component with the same configuration, 1≤j≤N, and j is an integer; The step of training the configuration of the light modulation component based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information includes: Determine a kth first loss value based on the kth denoised image information and the Nkth noise image information, wherein l≤k≤N, and the 0th noise image information is the original image information; Determining a comprehensive loss value based on all first loss values; Based on the comprehensive loss value, the configuration of the light modulation component is trained.

4. The method according to claim 2, characterized in that: The lth denoised image information among the N denoised image information is information obtained after the l-1th denoised image information is processed once by the current light modulation component, and the configuration of the current light modulation component is determined based on the training of the l-1th denoised image information, wherein l≤l≤N, and the 0th denoised image information is the Nth noise image information; The step of training the configuration of the light modulation component based on the original image information of the training sample, N-1 noise image information except the Nth noise image information, and N denoised image information includes: Based on the mth denoised image information and the Nmth noise image information, determining the mth second loss value, 1≤m≤N, m is an integer, and the 0th noise image information is the original image information; Based on the mth second loss value, the configuration of the light modulation component is trained.

5. The method according to claim 1, characterized in that The light modulation component includes at least one light modulation element, and the training of the configuration of the light modulation component based on the sample data set includes at least one of the following: Based on the sample data set, updating a configuration parameter of at least one of the light modulation elements; Based on the sample data set, the relative position relationship between the light modulation elements is updated.

6. The method according to claim 5, characterized in that The light modulation element includes at least one of the following: a refractive optical element, a diffractive optical element, a scattering medium element and a metasurface element.

7. The method according to claim 5, characterized in that The light modulation component includes a plurality of light modulation elements, which are arranged in sequence. Except for the first light modulation element, each of the light modulation elements is on a reflection path or a transmission path of a previous light modulation element.

8. The method according to claim 1, characterized in that The noise adding process is to add random noise to the original image information or the noise image information.

9. An optical system, characterized in that: include: A light modulation component, wherein the light modulation component is configured according to the method described in any one of claims 1 to 8; Photosensitive element; The light guide component is used to couple the output signal of the light modulation component to the input end of the light modulation component for a preset number of times and then couple it to the photosensitive element.

10. The system according to claim 9, characterized in that The light modulation component includes a plurality of light modulation elements, which are arranged in sequence. Except for the first light modulation element, each of the light modulation elements is on a reflection path or a transmission path of a previous light modulation element.

11. An image denoising method based on an optical system, characterized in that: The optical system includes a light modulation component, a photosensitive element and a light guide component; importing the image information to be processed into the optical system; Processing the image information using the light modulation component; By using the light guide component, after the output information of the image information processed by the light modulation component is coupled to the input end of the light modulation component for a preset number of times, it is coupled to the photosensitive element.

Images