A knowledge distillation-based photoelectric hybrid color image denoising method and system

Abstract

The application discloses a kind of photoelectric hybrid color image denoising method and system based on knowledge distillation, for the high computational complexity of existing deep learning denoising method, existing optical denoising scheme is multiple for gray or single channel image, it is difficult to fully model the problem of color channel correlation, noisy color image is input into multi-core parallel optical front end, executes the physical constraint optical convolution coding, obtains multi-channel optical response feature, then the multi-channel optical response feature is input into light-weight electronic recovery back end after photoelectric conversion, and outputs noise estimation result or denoising image;The application can significantly reduce the burden of back-end electronic calculation while maintaining high color image denoising performance, suitable for low-power, resource-limited edge imaging equipment.

Description

Technical Field

[0001] This invention belongs to the fields of computational imaging, optoelectronic fusion intelligent processing and image restoration, and specifically relates to a method and system for denoising optoelectronic hybrid color images based on knowledge distillation. Background Technology

[0002] With the development of intelligent imaging terminals, mobile vision devices, and edge sensing systems, image denoising is a fundamental processing step for improving image quality and ensuring the stability of subsequent recognition. In recent years, deep learning-based image denoising methods have made significant progress in terms of restoration accuracy. However, these methods typically rely on deep network structures, large parameter scales, and high computational complexity, which pose significant obstacles when deployed in resource-constrained and power-constrained devices.

[0003] Compared to pure electronic computing, optical computing possesses inherent parallelism, high throughput, and low power consumption. It can perform some linear operations during light propagation, thus showing high application potential in scenarios such as image preprocessing, feature encoding, and computational imaging. By migrating some front-end linear feature extraction processes to the optical domain, and performing operations such as convolution, filtering, or feature encoding during light propagation, the computational burden on the back-end electronic network is reduced, providing a new implementation path for low-power vision systems.

[0004] However, existing optical or optoelectronic hybrid image restoration schemes still have several shortcomings. First, existing research mostly focuses on grayscale images, failing to adequately consider the coupling relationships between different channels in color images, making it difficult to simultaneously achieve color consistency, edge restoration, and local texture preservation. Second, some schemes rely on multilayer diffraction structures or complex modulation devices, making system assembly and adjustment difficult and highly sensitive to manufacturing and propagation errors. Third, digital convolution weights generally contain both positive and negative responses, while optical modulation and intensity detection typically satisfy non-negative realization constraints, making it difficult to directly map digital convolution to physical optical structures.

[0005] Furthermore, existing optoelectronic hybrid frameworks are primarily geared towards discrimination tasks such as image classification. Their capabilities in color image restoration, especially in noise suppression, color consistency preservation, and detail recovery for color image denoising, still require improvement. Additionally, the physical response output by the optical front-end differs from the internal features of a purely electronic network in terms of formation mechanism and representation space. Directly using intermediate features from a purely electronic large model for constraints may limit the trainable degrees of freedom of the optical front-end, affecting its physical realizability and task adaptability.

[0006] Therefore, a photoelectric hybrid color image denoising technology solution needs to be proposed. This solution uses a multi-core parallel optical front end to perform joint optical encoding on the three RGB color channels, and uses an electronic restoration back end to model the cross-channel relationship of multi-channel features. This approach aims to balance color consistency, edge restoration, and texture preservation, while also meeting the requirements of optical physical constraints, lightweight electronic back end, and stable training. Summary of the Invention

[0007] The purpose of this invention is to address the problems of high model complexity, difficult optical implementation, and insufficient color channel coordination in existing color image denoising technologies, and to provide a knowledge distillation-based optoelectronic hybrid color image denoising method and system to achieve efficient restoration of noisy color images.

[0008] The specific technical solution for the objective of this invention is as follows:

[0009] A knowledge distillation-based photoelectric hybrid color image denoising method includes the following steps:

[0010] Acquire the noisy color image to be processed;

[0011] The noisy color image is input into a constructed and trained optoelectronic hybrid network to obtain noise estimation results or a denoised image;

[0012] The optoelectronic hybrid network includes a multi-core parallel optical front end and an electron recovery back end;

[0013] The multi-core parallel optical front end is used to perform linear optical coding on the noisy color image to obtain multi-channel optical response features;

[0014] The electronic recovery backend is used to output noise estimation results or denoised images based on the multi-channel optical response characteristics after photoelectric conversion.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] (1) This scheme moves some of the linear feature extraction processes to the optical domain, which can effectively reduce the computational burden of the back-end electronic network and help reduce the computational pressure and power consumption of the edge imaging platform. In addition, compared with the pure electrical teacher network, it significantly reduces the number of model parameters and floating-point operations, which helps to reduce the weight storage space and improve the image processing throughput.

[0017] (2) The multi-core parallel optical front-end of this scheme solves the problem that digital negative weights are difficult to directly map to optical non-negative systems through the positive and negative response splitting mechanism, which improves the feasibility of transferring digital convolution to physical optical structures.

[0018] (3) This scheme adopts an initialization strategy based on pre-trained shallow convolution response, which improves the training stability of optical front end and provides a more feasible target response for subsequent optical device design and manufacturing.

[0019] (4) This scheme adopts result domain knowledge distillation to transfer the recovery prior of the pure electronic teacher network to the lightweight student network, which improves the recovery performance of the student network and preserves the representational freedom and physical realizability of the optical layer.

[0020] (5) This solution is applicable to color image denoising scenarios. By using a lightweight electronic restoration backend to perform cross-channel modeling of the multi-channel features after optical encoding, it is beneficial to improve color consistency, edge restoration and texture fidelity, and can reasonably balance denoising performance and subsequent device design difficulty.

[0021] (6) This scheme uses a multi-core parallel optical front end to handle part of the linear feature extraction operations and a lightweight electronic recovery back end, which effectively reduces the parameter size and electronic computation complexity of the student optoelectronic hybrid denoising network. In one embodiment, the number of parameters of the student optoelectronic hybrid denoising network is about 0.364M, which is about 98.61% less than the Restormer pure electronic teacher network with about 26.11M parameters; the floating-point operation volume of the student network's electronic end is about 21.72G, which is about 92.30% less than the teacher network's 281.98G, thus helping to reduce the model storage overhead and electronic computation burden in the inference stage.

[0022] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a knowledge distillation-based photoelectric hybrid color image denoising architecture in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the multi-core parallel metasurface optical front end and positive and negative response differential readout in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the specific structure of the electronic recovery backend in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the training process of the optoelectronic hybrid network based on result domain knowledge distillation according to the present invention.

[0027] Figure 5 This is a schematic diagram of the denoising results of the test set in an embodiment of the present invention.

[0028] Figure 6This is a schematic diagram comparing the parameters of the optoelectronic hybrid network and the digital network in the example of the present invention, where (a) is a comparison of parameter quantities and (b) is a comparison of floating-point operation quantities.

[0029] Figure 7 This is a schematic diagram of the positive response target optical convolution kernel, the corresponding point spread function (PSF), and the phase distribution of the multi-wavelength metasurface in an embodiment of the present invention.

[0030] Figure 8 This is a schematic diagram of the negative response target optical convolution kernel, the corresponding point spread function (PSF), and the phase distribution of the multi-wavelength metasurface in an embodiment of the present invention.

[0031] Figure 9 This is a schematic diagram of the target point diffusion function (PSF) and its corresponding metasurface unit width distribution in an embodiment of the present invention. Detailed Implementation

[0032] Example

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0035] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0036] CombinationFigure 1 A knowledge distillation-based photoelectric hybrid color image denoising method includes the following steps:

[0037] Acquire the noisy color image to be processed;

[0038] The noisy color image is input into a constructed and trained optoelectronic hybrid network to obtain noise estimation results or a denoised image;

[0039] The optoelectronic hybrid network includes a multi-core parallel optical front end and an electron recovery back end;

[0040] The multi-core parallel optical front end is used to perform linear optical coding on the noisy color image to obtain multi-channel optical response features;

[0041] The electronic recovery backend is used to output noise estimation results or denoised images based on the multi-channel optical response characteristics after photoelectric conversion.

[0042] Through the synergistic processing of the optical front end and the electronic recovery back end, the ability to restore color consistency, edge structure and local texture in color images can be improved while reducing the computational burden on the back end.

[0043] More specifically, the multi-core parallel optical front end is a single-layer multi-core parallel metasurface optical front end, which includes multiple parallel-arranged metasurface optical response regions;

[0044] Combination Figure 2 Each of the metasurface optical response regions corresponds to a set of target point spread functions (PSFs), which are used to perform a set of optical convolutional encodings on the input noisy color image to obtain the corresponding optical response features.

[0045] The multi-core parallel optical front-end is not a fixed preprocessor, but rather participates in end-to-end joint optimization as a trainable optical layer in the student optoelectronic hybrid network, thereby learning the target point spread function (PSF) response that matches the color image denoising task. To improve the convergence instability caused by random initialization and the difficulty in physically realizing the optical response, the initial parameters of the PSF are obtained by dimensionality reduction of multiple convolutional kernels in the shallow convolutional layers of the pre-trained image restoration network. Each set of convolutional kernels obtained after dimensionality reduction serves as the initial value of the corresponding PSF, enabling the multi-core parallel optical front-end to extract edge, orientation texture, or local structural information during the initial training phase.

[0046] The pre-trained image restoration network is a pre-trained denoising network, including the DnCNN pre-trained denoising network.

[0047] The dimensionality reduction process includes one or more of principal component extraction, channel compression, or convolution kernel recombination.

[0048] Furthermore, since digital convolution weights typically include both positive and negative responses, and optical modulation and intensity detection usually satisfy non-negative intensity constraints, directly mapping signed digital convolutions to the optical front end can easily lead to physical implementation difficulties. Therefore, the multi-core parallel optical front end does not directly output the signed convolution result, but instead outputs the positive response convolution result and the negative response convolution result separately. Specifically:

[0049] Each group of metasurface optical response regions in the single-layer multi-core parallel metasurface optical front end corresponds to a signed target optical convolution kernel and includes a positive response metasurface region and a negative response metasurface region.

[0050] The signed target optical convolution kernel is split into a positive response convolution kernel and a negative response convolution kernel using a weight splitting method. The positive response convolution kernel is composed of the non-negative weights in the signed target optical convolution kernel, and the negative response convolution kernel is composed of the absolute values ​​of the negative weights in the signed target optical convolution kernel.

[0051] The positive response convolution kernel and the negative response convolution kernel correspond to the positive response target point spread function and the negative response target point spread function, respectively. The noisy color image is optically convolved and encoded by the positive response metasurface region and the negative response metasurface region, respectively, to generate two non-negative light intensity responses.

[0052] The two non-negative light intensity responses are converted into positive and negative electrical features after photoelectric conversion, respectively. These features are then paired and subtracted using a fixed-parameter difference layer to obtain signed optical encoded features, satisfying the non-negative constraints of optical modulation and light intensity detection. For example, in this embodiment, the fixed difference layer is used to compress the 32-channel complementary response after photoelectric conversion into a 16-channel differential feature. The fixed difference layer is preferably implemented using 1×1 convolution with fixed convolution weights, which do not participate in training.

[0053] In terms of physical implementation, this can be achieved through independent detection response channels, time-division multiplexing sampling, space-division multiplexing sampling, or multiple exposure sampling.

[0054] To address the convergence instability caused by random initialization and the difficulty in physically realizing optical responses, this invention extracts shallow responses from a pre-trained image restoration network and performs dimensionality reduction on these shallow responses to obtain several initial optical convolutional responses. These initial optical convolutional responses are used to construct the target point spread function set of the multi-kernel parallel optical front-end and serve as the initial values ​​for the corresponding target point spread functions, enabling the multi-kernel parallel optical front-end to extract edge, directional texture, or local structural information during the initial training phase. The target point spread functions are further optimized during subsequent end-to-end joint training to obtain optical responses that match the color image denoising task.

[0055] Combination Figure 3 The student optoelectronic hybrid denoising network sequentially comprises an optical convolutional layer, a positive response branch and a negative response branch, a photoelectric conversion and fixed difference layer, and a lightweight electronic recovery backend. The electronic recovery backend adopts a lightweight encoder-decoder structure and includes an attention unit for enhancing cross-channel modeling capabilities and a gated feedforward unit for enhancing local nonlinear characterization capabilities, used to output noise estimation results or denoised images based on the multi-channel optical coding features after differentiation.

[0056] Preferably, the lightweight encoder-decoder structure adopts a Restormer-like style, recovering the differential features through multi-scale feature extraction, cross-layer information fusion, and inter-channel relationship modeling to suppress noise, alleviate color channel aliasing and color artifacts, and compensate for edge and texture detail loss. The lightweight encoder-decoder structure includes a multi-head attention unit and a gated feedforward unit to enhance cross-channel interaction capabilities and local nonlinear reconstruction capabilities.

[0057] In a preferred embodiment, the electronic reconstruction backend adopts a Restormer-style structure, wherein the attention residual block sequentially includes a two-dimensional layer normalization, a multi-head transposed attention module, and a gated feedforward network module. Preferably, the number of attention heads is 8, and the gated feedforward network is used to enhance local texture recovery capabilities. During the inference phase, the electronic reconstruction backend outputs a predicted noise residual map N, and the final denoising result is obtained by subtracting the predicted noise residual map from the input noisy image. This residual learning method can accelerate network convergence and improve the recovery stability of high-frequency texture regions.

[0058] Furthermore, combining Figure 4 To compensate for the limited representation capabilities of the lightweight student optoelectronic hybrid network, the optoelectronic hybrid network is optimized during the training phase using result domain knowledge distillation. The optoelectronic hybrid network is set as a student optoelectronic hybrid denoising network, and a teacher pure electronic denoising network is constructed. The same noisy color image from the preset training set is input into the teacher pure electronic denoising network and the student optoelectronic hybrid denoising network respectively to obtain the teacher output result and the student output result respectively.

[0059] Distillation loss is determined based on the difference between the teacher's output and the student's output to establish a result domain distillation constraint between the teacher's output and the student's output, without constraining the intermediate features of the teacher's pure electronic denoising network and the student's optoelectronic hybrid denoising network.

[0060] This invention does not force the alignment of the intermediate feature layers of the two networks, but only applies distillation constraints between the output images of the teacher network and the output images of the student network. This improves the recovery ability of the student network while avoiding damage to the physical meaning of the optical signal, the trainable degrees of freedom of the optical layer, and the feasibility of subsequent metasurface devices.

[0061] The task loss and structure preservation loss are determined based on the student output and the real clean images in the training set, and the optical regularization loss is determined based on the optical response of the multi-core parallel optical front end.

[0062] The task loss, structure preservation loss, distillation loss, and optical regularization loss are weighted and summed to obtain the total loss, and the student optoelectronic hybrid denoising network is then supervised and optimized based on the total loss.

[0063] To improve the modeling ability and recovery stability of the student network in noisy scenarios, a dynamic distillation weight scheduling strategy that changes with the training stage is adopted during the training and optimization process of the student optoelectronic hybrid denoising network. The dynamic distillation weight is used to adjust the weight of the distillation loss in the total loss.

[0064] In the early stages of training, a low sampling ratio of high-noise samples is maintained, allowing the network to prioritize the initial convergence of basic optical response formation and electronic recovery paths. In the later stages of training, the sampling probability of high-noise samples is gradually increased, enabling the network to further enhance its adaptability to high-noise distribution, complex texture degradation, and detail burying after it has acquired a certain basic recovery capability. This improves the noise suppression capability, edge preservation capability, and cross-channel color consistency recovery capability of the optoelectronic hybrid denoising network under heavy noise conditions.

[0065] In an optional embodiment, the task loss is a Chabonier loss determined based on the difference between the student's output and the real clean image;

[0066] The structure preservation loss is a structural similarity loss determined based on the structural differences between the student's output and the real clean image;

[0067] The distillation loss is the Chabonier loss determined based on the difference between the student's output and the teacher's output;

[0068] The optical regularization loss is used to constrain the optical response of the multi-core parallel optical front end to meet the device implementation requirements, which include one or more of non-negative detection, response smoothness, and controllable energy distribution.

[0069] The following is combined with Figures 5 to 9The following further explains the color image denoising results, the comparison of model parameter quantity and floating-point operation quantity, the positive and negative response target optical convolution kernels and their corresponding point spread function (PSF), the phase distribution of multi-wavelength metasurfaces, and the mapping process of target optical response to metasurface structure parameters in this embodiment.

[0070] This embodiment provides a photoelectric hybrid processing method for color image denoising, and the overall application system includes:

[0071] An optical front-end module is used for optical convolutional coding of noisy color images;

[0072] The photoelectric conversion module is used to collect the light field intensity distribution output by the optical front end and convert it into an electrical signal;

[0073] An electronic recovery module is used to perform multi-scale recovery and denoising reconstruction of the electrical signal to obtain noise estimation results or denoised images;

[0074] The training optimization module is used to introduce the teacher network and perform knowledge distillation of the result domain during the training phase to obtain the parameters of the student optoelectronic hybrid network for the inference phase.

[0075] Among them, the metasurface optical front-end module is used to perform physically constrained optical encoding of RGB noisy color images, the photoelectric conversion module (image sensor) is used to convert the light field intensity distribution output by the optical front-end into an electrical signal, the electronic recovery module is used to output a predicted noise residual map or a denoised image based on the electrical signal, and the training optimization module is used to introduce a teacher network to perform result domain knowledge distillation on the student photoelectric hybrid network during the training phase.

[0076] Combination Figure 5 The figure shows the partial color image denoising results of the method of the present invention on a test set, including noisy images and corresponding denoising results, to intuitively reflect the method's ability to suppress image noise and preserve color, edge, and local texture information. In this embodiment, CBSD68 is preferably used as the main test set for Gaussian denoising of color images, and quantitative evaluation is performed under noise standard deviations σ of 15, 25, and 50; Kodak, McMaster, and CSet9 test sets can also be used for supplementary verification. The preferred evaluation metrics are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), to characterize the pixel reconstruction accuracy and structural preservation ability of the denoising results, respectively. Figure 6As shown, according to the network configuration of this embodiment, the number of parameters in the student optoelectronic hybrid denoising network is approximately 0.364M, and the floating-point operation volume on the electronic side is approximately 21.72G. In comparison, the number of parameters in the pure electronic DnCNN network is approximately 0.67M, and the floating-point operation volume is approximately 87.65G; the number of parameters in the Restormer pure electronic teacher network is approximately 26.11M, and the floating-point operation volume is approximately 281.98G. Therefore, this scheme reduces the number of parameters and floating-point operation volume by more than 92% compared to the Restormer pure electronic teacher network, and can complete the color image denoising task under conditions of lower model size and computational complexity.

[0077] In this embodiment, the multi-core parallel optical front-end is not a fixed filter, but a learnable optical layer participating in end-to-end training. Preferably, the multi-core parallel optical front-end uses the PCA-compressed target point spread function (PSF) as the initial convolution kernel, retaining more than 97% of the compressed information. The convolution kernel size is preferably 3×3, with each group of convolution kernels corresponding to the three RGB input channels, forming a total of 16 basic optical convolution responses. Considering that physical optical modulation and intensity detection satisfy non-negativity constraints, each group of optical convolution kernels is dynamically split into positive and negative response parts during training, forming two sets of complementary convolution outputs, thereby preserving the expressive power of positive and negative weights in digital convolution while meeting device implementation constraints. This setting enables the multi-core parallel optical front-end to gradually learn the target point spread function response matching the color image denoising task during joint training, and provides a target basis for subsequent metasurface device design.

[0078] Combination Figure 7 and Figure 8 , Figure 7 The distribution of one set of positive response target optical convolution kernels in the three color channels of RGB, the corresponding point spread function (PSF), and the metasurface phase distribution under blue light (450 nm), green light (520 nm), and red light (635 nm) conditions are shown. Figure 8 The distribution of the corresponding negative response target optical convolution kernel in the three RGB color channels, the corresponding point spread function (PSF), and the metasurface phase distribution under the above three working wavelength conditions are shown.

[0079] The metasurface optical front end employs a trainable multicolor metasurface with a metasurface unit period of 400 nm, and phase modulation is achieved through discrete unit side lengths ranging from 80 nm to 320 nm. The camera pixel size can be 3.45 μm, thus each camera pixel corresponds to approximately 8.625 metasurface units, completing the mapping from the camera sampling scale to the metasurface structure scale.

[0080] To satisfy the non-negativity constraints of optical modulation and intensity detection, the metasurface optical front end does not directly output the signed convolution result, but instead outputs the positive response convolution result and the negative response convolution result separately. For the i-th optical convolution response, its positive response channel and negative response channel are differentially reconstructed in the electronic domain after photoelectric conversion, thereby restoring the signed linear feature representation capability. In physical implementation, this can be achieved through 32 independent detection response channels or equivalent time-division multiplexing and space-division multiplexing sampling methods.

[0081] The teacher network employs a high-performance pure electronic image denoising network. In this embodiment, a Restormer network is used, with 3 input and 3 output channels and a feedforward expansion coefficient of 2.66. The teacher network parameters are frozen during the distillation training phase and are used only to generate the teacher output image.

[0082] The student network employs a hybrid optoelectronic structure consisting of the aforementioned metasurface optical front end and a Restormer-style electron recovery back end. The noise standard deviation σ is randomly sampled within the range [0, 50]. PSNR and SSIM indices are calculated at three noise levels: σ = 15, 25, and 50.

[0083] To improve the modeling ability and recovery stability of the student optoelectronic hybrid network in noisy scenes, a progressive enhancement strategy using high-noise samples is adopted during training. Specifically, in the early stage of training, considering that the network is still in the initial convergence stage of basic optical response formation and electronic recovery path, introducing a large number of high-noise samples too early can easily lead to instability in the optimization process and affect the optical front-end's learning of the diffusion function of effective target points. Therefore, a low sampling ratio of high-noise samples is maintained in the initial stage of training. The sampling probability of high-noise samples in each training batch is gradually increased, so that after the network has acquired a certain basic recovery ability, it can further enhance its adaptability to high-noise distribution, complex texture degradation, and detail burying. Preferably, the high-noise threshold is set to σ=35, that is, when the noise standard deviation is greater than or equal to 35, the corresponding sample is classified as a high-noise sample. By using the aforementioned progressive sampling method, the model can prioritize stable fitting of low-to-medium noise samples in the early stages of training, and gradually strengthen the learning of high-noise and difficult samples in the later stages of training. This improves the noise suppression, edge preservation, and cross-channel color consistency recovery capabilities of the optoelectronic hybrid denoising network under heavy noise conditions, and further enhances the generalization performance and robustness of the model under different noise intensity distributions.

[0084] The training loss consists of task loss, structure preservation loss, distillation loss, and optical regularization loss. Each loss term jointly constrains the student optoelectronic hybrid network from different perspectives, including pixel reconstruction accuracy, structural information preservation, teacher knowledge transfer, and device physical realizability. Specifically, the task loss measures the pixel-level difference between the denoised image output by the student optoelectronic hybrid network and the real clean image, ensuring the network possesses basic noise suppression and residual recovery capabilities. The structure preservation loss enhances the consistency between the recovered image and the real image in terms of edge contours, texture details, and local contrast. The distillation loss constrains the output of the student optoelectronic hybrid network to converge with the output of the teacher network, thereby gradually transferring the recovery priors inherent in the high-performance teacher network to the lightweight student optoelectronic hybrid network. The optical regularization loss restricts the distribution of the optical convolution kernel or the target point spread function, ensuring that the learned optical response maintains task relevance while meeting the requirements of subsequent metasurface device design and manufacturing.

[0085] The task loss employs Charbonnier loss, which, compared to ordinary L2 loss, exhibits better robustness to outliers and strong local noise, and can more stably constrain the deviation between the restored result and the real image during color image denoising. The structure preservation loss uses structural similarity loss (SSIM) to evaluate the consistency between the restored image and the real image from three dimensions: brightness, contrast, and structure. The distillation loss, by constraining the output space, avoids compromising the trainability and degrees of freedom of the optical front-end due to differences in feature formation mechanisms between pure electronic teacher networks and optoelectronic hybrid student networks. The optical regularization loss includes at least an energy constraint term and a smoothness constraint term. The energy constraint term limits the deviation of the current total energy of the optical convolution kernel from the initial total energy of the optical convolution kernel, preventing imbalances such as excessively strong or weak optical responses during training. The smoothness constraint term suppresses high-frequency oscillations of the convolution kernel in the spatial dimension, avoiding drastic local changes that are difficult to achieve in subsequent metasurface inversion and processing.

[0086] The distillation weights employ a four-stage dynamic scheduling strategy to coordinate the relationship between the autonomous shaping process of the optical front-end and the knowledge injection process of the teacher network. Specifically, the initial stage is a warm-up stage, in which the distillation weights are set to 0, and no teacher output constraints are imposed on the student network. This allows the optical front-end and the electronic reconstruction back-end to prioritize initial joint modeling around the current imaging link, noise distribution characteristics, and residual recovery target, thereby avoiding premature teacher supervision that could inhibit the formation of the optical response. The subsequent stage is a ramp-up stage, in which the distillation weights increase linearly from 0, allowing the high-quality recovery results from the teacher network to be gradually injected into the student network. This enables the student network to gradually learn better denoising mapping relationships while maintaining the optimizable degrees of freedom of the optical layer. The next stage is a decay stage, in which the distillation weights decrease linearly, allowing the student network to gradually reduce its dependence on teacher output based on the prior guidance it has received from the teacher, and instead enhance its autonomous adaptation ability to actual optical coding features, electronic reconstruction paths, and color channel coupling relationships. Finally, in the fine-tuning stage, distillation is kept with low weight. By maintaining a weak but continuous distillation constraint, parameter oscillations and local degradation phenomena in the later stages of training are suppressed, further improving the model's convergence stability and output consistency. Through the above four-stage dynamic scheduling method, the student network can prioritize forming optical responses and residual representations that match the imaging link in the early stages of training, fully absorb the recovery experience of the teacher network in the middle stages of training, and balance autonomous optimization capabilities with distillation supervision effects in the later stages of training. Thus, while meeting the physical implementation constraints of optical devices, the recovery accuracy, training stability, and model generalization ability of color image denoising are improved.

[0087] Furthermore, during joint training, the aforementioned loss terms are weighted and summed according to preset weights to form the overall optimization objective of the student optoelectronic hybrid network. By applying both the task loss and the structure-preserving loss to a real, clean image, the network's restoration results can be guaranteed to balance denoising capability and visual structure fidelity. Introducing distillation loss compensates for the lightweight student optoelectronic hybrid network's limitations in model capacity, improving restoration performance in complex texture regions and high-frequency edge regions. Adding optical regularization loss ensures that the training results not only achieve high-precision restoration at the algorithmic level but also map to metasurface target responses that satisfy physical implementation constraints. Thus, the joint loss function establishes a unified optimization framework between denoising accuracy, structure preservation, knowledge transfer, and device manufacturability, providing a training path for optoelectronic hybrid networks aimed at color image denoising that balances performance and physical implementability.

[0088] Combination Figure 9The figure shows the target optical convolution kernel and its corresponding metasurface unit width distribution obtained through joint training. To map the target point diffusion function response obtained during the training phase to metasurface structure parameters with practical manufacturability, the target optical convolution kernel and its positive and negative response components obtained through joint training are first read.

[0089] Preferably, 16 sets of RGB target optical convolution kernels, along with corresponding 16 sets of positive response convolution kernels and 16 sets of negative response convolution kernels, are read and normalized to unify the amplitude scale of each target response and reduce the impact of target energy differences on the stability of subsequent device fitting. Then, each target response is embedded into a preset metasurface mesh region, and combined with a multi-wavelength propagation model, the width distribution of the metasurface unit matching the target response is numerically optimized. This allows the fitted metasurface to generate a spatial light intensity distribution matching the corresponding target optical convolution kernel under different incident wavelengths of red, green, and blue. Through this process, the target optical convolution response obtained from network training is converted into metasurface structure parameters that can be used for fabrication and verification.

[0090] In a preferred embodiment, the size of the local camera region used to fit the target response is 6×6 camera pixels. Based on the calculation that each camera pixel corresponds to approximately 8.625 metasurface units, the local region corresponding to the target region on the metasurface plane is approximately 51×51 metasurface units. For each target response, the Adam optimizer is preferably used for fitting. During optimization, the multi-wavelength PSF obtained after propagation is normalized and compared with the target response, and the consistency of the overall shape of each color channel is used as the optimization criterion to reduce the error between the fitted metasurface output and the target convolution kernel. Through the above settings, a multi-wavelength metasurface PSF matching the target response can be obtained within a reasonable number of iterations, while balancing fitting accuracy and solution efficiency. Furthermore, for the target convolution kernels corresponding to the positive and negative responses, the above fitting process can be performed independently, thereby obtaining positive and negative response metasurface design results suitable for complementary optical sampling structures.

[0091] To facilitate subsequent device assembly and system alignment, this invention can further design five metasurface lenses for positioning. These positioning lenses are not used for performing denoising convolution calculations, but rather to provide clear and stable reference spots during actual assembly, assisting in calibrating the device at tilt angles, rotation angles, and focal length positions. After design, the positioning lens structure can be replicated and arranged at multiple preset positions on the device layout. During the assembly and adjustment phase, by observing the position and shape changes of each positioning spot, rapid correction of system tilt, rotation, and focal plane offset can be achieved.

[0092] During the layout design phase, multiple sets of positive-response metasurfaces, negative-response metasurfaces, and positioning lens structures can be integrated onto the same device plane according to preset rules to form a metasurface device array suitable for the optoelectronic hybrid denoising system of this invention. For each metasurface structure corresponding to a target response, its cell width distribution, target response fitting error, and propagation PSF results can be saved separately as a basis for subsequent device fabrication, performance verification, and process tolerance analysis. Through the above implementation methods, this invention can not only obtain optical target responses suitable for color image denoising at the algorithm level, but also further transform the target responses into practically manufacturable metasurface device structures, thereby forming a complete technical link from network training, target response extraction to physical device design.

[0093] In addition, this solution also provides a knowledge distillation-based photoelectric hybrid color image denoising system, which includes the following modules:

[0094] An optical front-end module is used for optical convolutional coding of noisy color images;

[0095] The photoelectric conversion module is used to collect the light field intensity distribution output by the optical front end and convert it into an electrical signal;

[0096] An electronic recovery module is used to perform multi-scale recovery and denoising reconstruction of the electrical signal to obtain noise estimation results or denoised images;

[0097] The training optimization module is used to introduce the teacher network during the training phase and perform knowledge distillation of the result domain to obtain the parameters of the student optoelectronic hybrid network for the inference phase.

[0098] This solution also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0099] Acquire the noisy color image to be processed;

[0100] The noisy color image is input into a constructed and trained optoelectronic hybrid network to obtain noise estimation results or a denoised image;

[0101] The optoelectronic hybrid network includes a multi-core parallel optical front end and an electron recovery back end;

[0102] The multi-core parallel optical front end is used to perform linear optical coding on the noisy color image to obtain multi-channel optical response features;

[0103] The electronic recovery backend is used to output noise estimation results or denoised images based on the multi-channel optical response characteristics after photoelectric conversion.

[0104] The embodiments described above are merely one implementation method of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A photoelectric hybrid color image denoising method based on knowledge distillation, characterized in that, Includes the following steps: Acquire the noisy color image to be processed; The noisy color image is input into a constructed and trained optoelectronic hybrid network to obtain noise estimation results or a denoised image; The optoelectronic hybrid network includes a multi-core parallel optical front end and an electron recovery back end; The multi-core parallel optical front end is used to perform linear optical coding on the noisy color image to obtain multi-channel optical response features; The electronic recovery backend is used to output noise estimation results or denoised images based on the multi-channel optical response characteristics after photoelectric conversion.

2. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 1, characterized in that, The multi-core parallel optical front end is a single-layer multi-core parallel metasurface optical front end, which includes multiple parallel-arranged metasurface optical response regions; Each of the metasurface optical response regions corresponds to a set of target point spread functions (PSFs), which are used to perform a set of optical convolutional encodings on the input noisy color image to obtain the corresponding optical response features; The initial parameters of the target point spread function (PSF) are obtained by dimensionality reduction of multiple convolutional kernels in the shallow convolutional layer of the pre-trained image restoration network. Each set of convolutional kernels obtained after dimensionality reduction serves as the initial value of the corresponding target point spread function (PSF), enabling the multi-kernel parallel optical front end to have the ability to extract edge, orientation texture, or local structural information in the initial training stage.

3. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 2, characterized in that, The pre-trained image restoration network is a pre-trained denoising network, including the DnCNN pre-trained denoising network; The dimensionality reduction process includes one or more of principal component extraction, channel compression, or convolution kernel recombination.

4. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 2, characterized in that, Each group of metasurface optical response regions in the single-layer multi-core parallel metasurface optical front end corresponds to a signed target optical convolution kernel and includes a positive response metasurface region and a negative response metasurface region. The signed target optical convolution kernel is split into a positive response convolution kernel and a negative response convolution kernel using a weight splitting method. The positive response convolution kernel is composed of the non-negative weights in the signed target optical convolution kernel, and the negative response convolution kernel is composed of the absolute values ​​of the negative weights in the signed target optical convolution kernel. The positive response convolution kernel and the negative response convolution kernel correspond to the positive response target point spread function and the negative response target point spread function, respectively. The noisy color image is optically convolved and encoded by the positive response metasurface region and the negative response metasurface region, respectively, to generate two non-negative light intensity responses. The two non-negative light intensity responses are converted into positive and negative response electrical features respectively after photoelectric conversion. They are then paired and subtracted through a differential layer with fixed parameters to obtain a signed optical coding feature, thus satisfying the non-negative constraints of optical modulation and light intensity detection.

5. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 2, characterized in that, The electronic recovery backend adopts a lightweight encoder-decoder structure and includes an attention unit for enhancing cross-channel modeling capabilities and a gated feedforward unit for enhancing local nonlinear characterization capabilities; the electronic recovery backend is used to output noise estimation results or denoised images based on the multi-channel optical response characteristics.

6. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 2, characterized in that, The aforementioned optoelectronic hybrid network is optimized during the training phase using result domain knowledge distillation. The photoelectric hybrid network is set as the student photoelectric hybrid denoising network, and a teacher pure electronic denoising network is constructed; the same noisy color image from the preset training set is input into the teacher pure electronic denoising network and the student photoelectric hybrid denoising network respectively, and the teacher output result and the student output result are obtained respectively. Distillation loss is determined based on the difference between the teacher's output and the student's output to establish a result domain distillation constraint between the teacher's output and the student's output; The task loss and structure preservation loss are determined based on the student output and the real clean images in the training set, and the optical regularization loss is determined based on the optical response of the multi-core parallel optical front end. The task loss, structure preservation loss, distillation loss, and optical regularization loss are weighted and summed to obtain the total loss, and the student optoelectronic hybrid denoising network is then supervised and optimized based on the total loss.

7. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 6, characterized in that, During the training and optimization process of the student optoelectronic hybrid denoising network, a dynamic distillation weight scheduling strategy that changes with the training stage is adopted. The dynamic distillation weight is used to adjust the weight of the distillation loss in the total loss. In the early stages of training, a lower distillation weight or no distillation constraint is applied. In the middle stages of training, the distillation weight is increased, and in the later stages of training, the distillation weight is decreased. This is done to adjust the intensity of distillation constraints at different training stages and to achieve a balance between teacher network knowledge transfer and student optoelectronic hybrid network autonomous optimization.

8. The photoelectric hybrid color image denoising method based on knowledge distillation according to claim 6, characterized in that, The task loss is the Chabonier loss, which is determined based on the difference between the student's output and the real clean image. The structure preservation loss is a structural similarity loss determined based on the structural differences between the student's output and the real clean image; The distillation loss is the Chabonier loss determined based on the difference between the student's output and the teacher's output; The optical regularization loss is used to constrain the optical response of the multi-core parallel optical front end to meet the device implementation requirements, which include one or more of non-negative detection, response smoothness, and controllable energy distribution.

9. A photoelectric hybrid color image denoising system based on knowledge distillation, characterized in that, Includes the following modules: An optical front-end module is used for optical convolutional coding of noisy color images; The photoelectric conversion module is used to collect the light field intensity distribution output by the optical front end and convert it into an electrical signal; An electronic recovery module is used to perform multi-scale recovery and denoising reconstruction of the electrical signal to obtain noise estimation results or denoised images; The training optimization module is used to introduce the teacher network during the training phase and perform knowledge distillation of the result domain to obtain the parameters of the student optoelectronic hybrid network for the inference phase.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

Images