Design method and device of computational imaging system and electronic equipment

Abstract

The embodiment of the invention provides a design method and device of a computational imaging system and electronic equipment. The method comprises the following steps: respectively determining degradation parameters after light passes through each first simplified optical system under each first working condition; performing degradation on a sample clear image by using each first degradation parameter; each first degradation parameter is used to repair each first degradation image; determining a first simplified optical system of which the corresponding first repair image meets a preset repairable condition; determining an optical structure parameter threshold value including the optical structure parameters of each repairable simplified optical system, and taking the optical structure parameter threshold value as a recoverability boundary; and designing to obtain a second simplified optical system of which the optical structure parameters are located in the restorable boundary, and training to obtain a target neural network for repairing an image acquired by the second simplified optical system. And the feasibility of the designed computational imaging system can be improved.

Description

Technical Field

[0001] This invention relates to the field of optical technology, and in particular to a design method, apparatus, and electronic device for a computational imaging system. Background Technology

[0002] In applications such as remote sensing imaging, high-resolution cameras, UAV payloads, and mobile platform imaging, imaging systems typically need to achieve high resolution, wide field of view, and stable image quality within limited size, weight, and power consumption constraints, while also adapting to complex operating conditions such as temperature drift and vibration shock. To meet these requirements, traditional optical imaging systems often employ combinations of multiple high-precision lenses, introduce advanced components such as aspherical surfaces, and use strict processing and assembly tolerances to control aberrations and improve image quality. However, this results in complex structures, high manufacturing, testing, and assembly costs, and long development cycles, limiting the miniaturization and large-scale deployment of optical systems.

[0003] With increasing demands for low cost, lightweight design, and rapid iteration, computational imaging systems are gradually developing towards a "physical simplification + computational compensation" approach. This means that the front end can use simpler, more user-friendly optical structures to acquire necessary information, and the algorithm can then perform compensation in the digital domain. In current engineering practice, front-end design and algorithm compensation are often relatively separate: the front end is mainly optimized based on traditional image quality indicators (such as MTF, puncta plot, and RMS wavelet aberration), while the algorithm trains a restoration model under given degradation conditions. This results in a lack of a unified design and evaluation loop between front-end design and algorithm compensation, leading to the inability of the designed computational imaging system to function properly due to the lack of coordination between the front-end design and algorithm compensation. Summary of the Invention

[0004] The purpose of this invention is to provide a design method, apparatus, and electronic device for a computational imaging system, thereby improving the feasibility of the designed computational imaging system. The specific technical solution is as follows:

[0005] In a first aspect of this application, a design method for a computational imaging system is provided, the method comprising:

[0006] The degradation parameters of light rays after passing through each of the first simplified optical systems under each first operating condition are determined respectively, and are used as the first degradation parameters of each of the first simplified optical systems, wherein the degradation parameters are used to represent the wavefront aberration of the light rays;

[0007] The clear image of the sample is degraded using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems;

[0008] Each of the first degraded images is repaired using the first degradation parameters respectively, to obtain the first repaired image corresponding to each of the first simplified optical systems;

[0009] A first simplified optical system that satisfies the preset repairable conditions for the corresponding first repaired image is identified as a repairable simplified optical system;

[0010] A threshold for optical structural parameters, including the optical structural parameters of each of the repairable simplified optical systems, is determined as a recoverability boundary.

[0011] A second simplified optical system with optical structure parameters located within the recoverability boundary was designed, and a target neural network was trained to repair the image acquired by the second simplified optical system.

[0012] In one possible embodiment, the training to obtain a target neural network for repairing images acquired by the second simplified optical system includes:

[0013] The target neural network is trained using a real pairing dataset, wherein the real pairing dataset includes multiple sample data, each sample data including a second degraded image and a second degraded parameter obtained by a second simplified optical system acquiring the sample environment under the second operating condition, and a reference image acquired by a reference optical system acquiring the sample environment. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition, and the second degraded image included in different sample data is different, and the different second degraded images are obtained by acquiring different sample environments under different second operating conditions.

[0014] In one possible embodiment, training the target neural network using a real pairing dataset includes:

[0015] The second degraded image and the second degradation parameter in the sample data are input into the original neural network to obtain the image output by the original neural network, which is used as the second repaired image;

[0016] The second restored image is degraded using the second degradation parameter to obtain a third degraded image;

[0017] The difference between the second restored image and the reference image in the sample data is calculated as the first difference; and the difference between the third degraded image and the second degraded image is calculated as the second difference;

[0018] The model parameters of the original neural network are adjusted in the direction of reducing the first difference and the second difference to obtain the target neural network.

[0019] In one possible embodiment, the preset repairable conditions include: the peak signal-to-noise ratio of the first repaired image is higher than a preset first threshold, and / or, the structural similarity between the first repaired image and the clear sample image is higher than a preset second threshold.

[0020] In a second aspect of this application, a computational imaging method based on a simplified optical system is provided, the method comprising:

[0021] Acquire a fourth degraded image and a fourth operating condition acquired by a second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image;

[0022] The degradation parameters of light rays passing through the second simplified optical system under the fourth operating condition are determined as the third degradation parameter;

[0023] The fourth degraded image and the third degraded parameter are input into the target neural network to obtain the repaired image output by the target neural network, which is used as the imaging result;

[0024] The second simplified optical system and the target neural network are obtained in advance by the design method of the computational imaging system as described in any of the first aspects above.

[0025] In a third aspect of this application, a design apparatus for a computational imaging system is provided, the apparatus comprising:

[0026] The first parameter determination module is used to determine the degradation parameters of light rays after passing through each first simplified optical system under each first operating condition, as the first degradation parameters of each first simplified optical system, wherein the degradation parameters are used to represent the wavefront aberration of light rays;

[0027] The degradation module is used to degrade the clear image of the sample using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems.

[0028] The repair module is used to repair each of the first degraded images using each of the first degradation parameters, so as to obtain the first repaired image corresponding to each of the first simplified optical systems.

[0029] The filtering module is used to determine the first simplified optical system that meets the preset repairable conditions for the corresponding first repaired image, and to identify it as a repairable simplified optical system.

[0030] A boundary determination module is used to determine an optical structure parameter threshold that includes the optical structure parameters of each of the repairable simplified optical systems, as a recoverability boundary.

[0031] The design module is used to design a second simplified optical system whose optical structural parameters are located within the recoverability boundary, and to train a target neural network for repairing the image acquired by the second simplified optical system.

[0032] In one possible embodiment, the design module trains a target neural network for repairing images acquired by the second simplified optical system, including:

[0033] The target neural network is trained using a real pairing dataset, wherein the real pairing dataset includes multiple sample data, each sample data including a second degraded image and a second degraded parameter obtained by a second simplified optical system acquiring the sample environment under the second operating condition, and a reference image acquired by a reference optical system acquiring the sample environment. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition, and the second degraded image included in different sample data is different, and the different second degraded images are obtained by acquiring different sample environments under different second operating conditions.

[0034] In one possible embodiment, the design module trains the target neural network using a real pairing dataset, including:

[0035] The second degraded image and the second degradation parameter in the sample data are input into the original neural network to obtain the image output by the original neural network, which is used as the second repaired image;

[0036] The second restored image is degraded using the second degradation parameter to obtain a third degraded image;

[0037] The difference between the second restored image and the reference image in the sample data is calculated as the first difference; and the difference between the third degraded image and the second degraded image is calculated as the second difference;

[0038] The model parameters of the original neural network are adjusted in the direction of reducing the first difference and the second difference to obtain the target neural network.

[0039] In one possible embodiment, the preset repairable conditions include: the peak signal-to-noise ratio of the first repaired image is higher than a preset first threshold, and / or, the structural similarity between the first repaired image and the clear sample image is higher than a preset second threshold.

[0040] In a fourth aspect of this application, a computational imaging apparatus based on a simplified optical system is also provided, the apparatus comprising:

[0041] The input module is used to acquire the fourth degraded image and the fourth operating condition acquired by the second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image;

[0042] The second parameter determination module is used to determine the degradation parameters of light after passing through the second simplified optical system under the fourth operating condition, as the third degradation parameter;

[0043] The algorithm module is used to input the fourth degraded image and the third degraded parameter into the target neural network to obtain the repaired image output by the target neural network as the imaging result;

[0044] The second simplified optical system and the target neural network are obtained in advance by the design method of a computational imaging system as described in any of the first aspects above.

[0045] In a fifth aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0046] Memory, used to store computer programs;

[0047] When a processor executes a program stored in memory, it implements the design method steps of the computational imaging system described in either the first or second aspect above.

[0048] In a sixth aspect of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the design method steps of the computational imaging system described in either the first or second aspect above.

[0049] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the design method of the computational imaging system described in either the first or second aspect above.

[0050] Beneficial effects of the embodiments of the present invention:

[0051] The present invention provides a design method, apparatus and electronic device for a computational imaging system. By determining the recoverability boundary under the repair process, the boundary and selection criteria for optical structure parameters that simplify the optical system design are set, so that the front-end design and algorithm compensation form a unified design and evaluation closed loop, thereby improving the feasibility of the designed computational imaging system.

[0052] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0054] Figure 1 A flowchart illustrating the design method of the computational imaging system provided in this application;

[0055] Figure 2 A flowchart illustrating the target neural network training method provided in this application;

[0056] Figure 3 A schematic flowchart of the computational imaging method based on a simplified optical system provided in this application;

[0057] Figure 4 A schematic diagram of the structural device for the design of the computational imaging system provided in this application;

[0058] Figure 5 A schematic diagram of the structure of the computational imaging device based on a simplified optical system provided in this application;

[0059] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0061] To more clearly illustrate the design method of the computational imaging system provided in this application, a possible application scenario of the design method of the computational imaging system provided in this application is first illustrated below. It is understood that the following example is only one possible application scenario of the design method of the computational imaging system provided in this application, and the design method of the computational imaging system provided in this application can also be applied to other scenarios; the following example does not limit its application in any way.

[0062] Due to limitations in cost, size, weight, and power consumption, mobile phone optical systems are often designed as simplified optical systems. To address the wavefront distortion and image degradation issues caused by these simplified optical systems, existing technologies often employ algorithmic compensation, which involves using algorithms to repair images acquired by the simplified optical system. This paper refers to such a system combining a simplified optical system with algorithmic compensation as a computational imaging system.

[0063] Existing computational imaging system designs can be summarized into the following three approaches:

[0064] Method 1: A simplified design based on traditional imaging optical indicators follows the design goals and optimization process of traditional point-to-point imaging. In optical design software such as Zemax, image quality indicators such as point plots, modulus transfer function (MTF), wavefront aberration, root mean square (RMS) wavefront aberration, and distortion are used as the main evaluation criteria. The structure is simplified by reducing the number of lenses, adjusting the position of the aperture, optimizing the radius of curvature, thickness and spacing, and selecting different materials and film systems. Furthermore, the feasibility of the design scheme under manufacturing and assembly conditions is verified through tolerance analysis and thermal analysis.

[0065] Method Two: Simplified design based on specialized optical elements. This method integrates functional components such as diffractive optical elements (DOEs), freeform surfaces, or metasurfaces. Chromatic aberration correction, aberration compensation, or some imaging functions are transferred from multiple optical lenses to a single or a few components, reducing the number of lenses and shortening the system length. This approach typically involves jointly optimizing parameters such as refractive indexes / diffraction / freeform surfaces within an optical design platform and evaluating system performance under manufacturing and assembly constraints.

[0066] Method 3: In the optical-algorithm co-design scheme, degradation characterization and compensation methods usually revolve around the spatially varying point spread function (PSF), optical transfer function (OTF), or wavefront aberration parameters. One type of method is based on the PSF or degradation physical model for restoration, such as deconvolution and variational reconstruction. Another type of method is based on wavefront / aberration estimation for model-driven correction, such as using Zernike coefficients to characterize the wavefront and compensate accordingly. In addition, there are methods that use pure data-driven end-to-end deep learning to directly learn the mapping relationship between degraded and clear images.

[0067] However, for the aforementioned method one, since it relies on image plane indicators such as point plots, modulus transfer function, wavefront aberration, root mean square wavefront aberration, and distortion to guide simplification and optimization, when the optical system is further simplified, degradation often manifests as increased frequency domain attenuation, obvious spatial variation point spread function, and sensitivity to noise. At this point, designers need to rely on their own design experience to continue to simplify the optical system. However, for designers with insufficient experience, the images acquired by the simplified optical system they design may not be able to be restored to clear images by the algorithm.

[0068] For the aforementioned Method 2, in engineering implementation, it is often more sensitive to manufacturing and assembly errors. Non-ideal factors such as device surface deviation, diffraction efficiency and film layer differences, assembly eccentricity and tilt, and stray light can cause the wavefront and spatial variation point diffusion function morphology shift, making it more difficult to stably characterize and calibrate the degradation characteristics in the field of view and working condition dimensions. Consequently, the same computational imaging system designed cannot effectively repair images under some working conditions, which leads to insufficient reproducibility and consistency of the repair.

[0069] For the aforementioned method three, a sequential process of "first completing the design of the simplified optical system, then performing algorithm training and compensation" is often adopted, or joint parameter tuning of the simplified optical system and algorithm compensation is only performed within a limited range. Since there is no unified mapping relationship between the range of degradation parameters such as wavefront aberration and the restoration effect, when the restoration effect is not up to standard, how to further optimize the simplified optical system depends entirely on experience and repeated experiments, making it difficult to design a computational imaging system with satisfactory restoration effect.

[0070] It is evident that the computational imaging systems designed using the aforementioned methods one, two, and three may be unable to effectively restore images due to various limitations, meaning that the feasibility of these computational imaging systems is poor. Therefore, this application provides a design method for a computational imaging system to improve the feasibility of the designed system.

[0071] See Figure 1 , Figure 1 The diagram shown is a flowchart illustrating the design method of the computational imaging system provided in this application, including:

[0072] Step S101: Determine the degradation parameters of each simplified optical system under each first operating condition after the light passes through it, and use them as the first degradation parameters of each simplified optical system.

[0073] The degradation parameter is used to represent the wavefront aberration of the light rays.

[0074] Step S102: Degrade the clear image of the sample using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems.

[0075] Step S103: Repair each first degraded image using each first degradation parameter to obtain the first repaired image corresponding to each first simplified optical system.

[0076] Step S104: Determine the first simplified optical system that satisfies the preset repairable conditions for the corresponding first repaired image, and use it as a repairable simplified optical system.

[0077] Step S105: Determine the optical structure parameter threshold that includes the optical structure parameters of each repairable simplified optical system, as the recoverability boundary.

[0078] Step S106: Design a second simplified optical system whose optical structure parameters are within the recoverability boundary, and train a target neural network for repairing the image acquired by the second simplified optical system.

[0079] By using the design method of the computational imaging system provided in this application, the recoverability boundary is determined under the repair process, so as to set the boundary for simplifying the optical system design and the selection criteria for optical structure parameters. This enables the front-end design and algorithm compensation to form a unified design and evaluation closed loop, thereby improving the feasibility of the designed computational imaging system.

[0080] The following will explain steps S101 to S106 respectively:

[0081] In step S101, the first simplified optical system can be a real simplified optical system or a virtual simplified optical system. For example, the first simplified optical system can be a simplified optical system simulated using optical simulation software. Furthermore, if the first simplified optical system is a real simplified optical system, the first degradation parameter of the first simplified optical system can be obtained through actual measurement; if the first simplified optical system is a virtual simplified optical system, the first degradation parameter of the first simplified optical system can be obtained through optical software simulation.

[0082] There are multiple first simplified optical systems, and the optical structural parameters of different first simplified optical systems are different. Furthermore, to more accurately determine the repairable boundary, the optical structural parameters of each first simplified optical system should cover as many possible optical structural parameters as possible. For example, assuming the optical structural parameters are the radius of curvature and thickness of the lens, and the possible values ​​for the radius of curvature are 100mm to 130mm, and the possible values ​​for the thickness are 6mm to 10mm, then the following 12 first simplified optical systems can be selected: a first simplified optical system with a radius of curvature of 100mm and a thickness of 6mm, a first simplified optical system with a radius of curvature of 100mm and a thickness of 8mm, a first simplified optical system with a radius of curvature of 100mm and a thickness of 10mm, a first simplified optical system with a radius of curvature of 110mm and a thickness of 6 ... The first simplified optical system has a radius of curvature of 8mm, a radius of curvature of 110mm and a thickness of 10mm, a radius of curvature of 120mm and a thickness of 6mm, a radius of curvature of 120mm and a thickness of 8mm, a radius of curvature of 120mm and a thickness of 10mm, a radius of curvature of 130mm and a thickness of 6mm, a radius of curvature of 130mm and a thickness of 8mm, and a radius of curvature of 130mm and a thickness of 10mm.

[0083] There are multiple first operating conditions, and in order to more accurately determine the repairable boundary, each first operating condition covers as many possible operating conditions as possible. Here, possible operating conditions refer to the operating conditions that the computational imaging system to be designed may be in during actual operation.

[0084] It is understandable that the degradation parameters of light after passing through the first simplified optical system are different under different operating conditions. Therefore, each first simplified optical system has multiple first degradation parameters, and different first degradation parameters correspond to different first operating conditions.

[0085] For example, suppose there are three different first operating conditions, denoted as c1, c2, and c3, and two different first simplified optical systems, denoted as first simplified optical systems 1 and 2, respectively. The optical structural parameters of first simplified optical systems 1 and 2 are denoted as p1 and p2, respectively. Then, first simplified optical system 1 has four degenerate parameters: g(p1, c1), g(p1, c2), g(p1, c3), and g(p1, c4). First simplified optical system 2 also has four degenerate parameters: g(p2, c1), g(p2, c2), g(p2, c3), and g(p2, c4). Where g(·) is a function with optical structure parameters and operating conditions as independent variables and degradation parameters as dependent variables, g(p1, c1) is the degradation parameter of light passing through the first simplified optical system 1 under c1, g(p1, c2) is the degradation parameter of light passing through the first simplified optical system 1 under c2, and so on.

[0086] The degradation parameter can be wavefront aberration, any parameter characterizing wavefront aberration, or a combination of both. For example, in one possible embodiment, the degradation parameter includes one or more of the following parameters: aberration parameter, equivalent pupil, and spatially varying point spread function. The aberration parameter can be expressed in the form of a Zemike parameter, and the equivalent pupil can be expressed in the form of an optical transfer function.

[0087] Operating conditions can be any parameters that can affect the imaging of a simplified optical system. For example, in one possible embodiment, operating conditions include: temperature drift, adjustment drift, vibration and shock, focal plane offset, and field of view position.

[0088] In step S102, the clear sample image can be acquired through a non-simplified optical system or selected from a public dataset. The content in the clear sample image should be as consistent as possible with the content to be captured by the computational imaging system to be designed. For example, assuming that the computational imaging system to be designed is a remote sensing imaging system responsible for capturing remote sensing images, the clear sample image should be a remote sensing image as much as possible.

[0089] Furthermore, the degradation process for clear sample images should be consistent with the imaging chain of the computational imaging system to be designed. For example, assuming the imaging chain of the computational imaging system directly images the acquired images, and the noise of this imaging chain is negligible, degradation can be performed according to formula (1):

[0090] I=H θ (S) … (1);

[0091] Where I is the first degraded image, S is the clear sample image, θ is the first degradation parameter, and H... θ (·) represents the degradation operator, which is a function with degradation parameters and clear image as independent variables and degraded image as dependent variable. Further assuming that the imaging link of the computational imaging system directly images the acquired image, and that the noise of this imaging link cannot be ignored, then degradation can be performed according to formula (2):

[0092] I=H θ (S) + n … (2);

[0093] Where n represents noise. Furthermore, assuming the imaging link of the computational imaging system includes detector sampling, detector sampling must also be introduced during the degradation process. Also assuming the imaging link of the computational imaging system includes a quantization process, a quantization process must also be introduced during the degradation process.

[0094] It is understandable that, since each first simplified optical system has multiple first degradation parameters, each first simplified optical system corresponds to multiple first degradation images. For example, suppose there are two different first simplified optical systems, denoted as first simplified optical systems 1 and 2. First simplified optical system 1 has four first degradation parameters, denoted as θ11 to θ14, and first simplified optical system 2 also has four first degradation parameters, denoted as θ21 to θ24. There are a total of three clear sample images, denoted as S1 to S3. Then, θ11 to θ14 are used to degrade S1 to S3 respectively, resulting in a total of 12 first degradation images as the first degradation images corresponding to first simplified optical system 1. Similarly, θ21 to θ24 are used to degrade S1 to S3 respectively, resulting in a total of 12 first degradation images as the first degradation images corresponding to first simplified optical system 2.

[0095] In step S103, this application does not limit the method of repairing the first degraded image, but the method of repairing the first degraded image should be the same as the physical model on which the target neural network is subsequently trained. For example, the original neural network can be used to repair the first degraded image, where the original neural network is the original model on which the target neural network is trained.

[0096] It is understood that, since each first simplified optical system corresponds to multiple first degraded images, each first simplified optical system corresponds to multiple first restored images.

[0097] In step S104, the preset repairability condition can be set according to actual needs and / or user experience, but it should satisfy the following: if the first repaired image meets the preset repairability condition, then the first repaired image can be considered a clear image. For example, in one possible embodiment, the preset repairability condition is: the peak signal-to-noise ratio (PSNR) of the first repaired image is higher than a preset first threshold. In another possible embodiment, the preset repairability condition is: the structural similarity (SSIM) between the first repaired image and the sample clear image is higher than a second threshold. In yet another possible embodiment, the preset repairability condition is: the peak signal-to-noise ratio of the first repaired image is higher than the preset first threshold, and the structural similarity between the first repaired image and the sample clear image is higher than the second threshold.

[0098] As explained above, each first simplified optical system corresponds to multiple first repair images. Therefore, the statement that a first repair image meets the preset repairability condition can mean that all corresponding first repair images meet the preset repairability condition, or it can mean that images with a percentage of Th or higher among the corresponding first repair images meet the preset repairability condition. Here, Th is a preset value, and its value ranges from (0 to 100). Users can set the value of Th according to actual needs and / or experience. The larger Th is set, the more accurate the determined recoverability boundary will be. The smaller Th is set, the more flexible the subsequent design of the second simplified optical system will be.

[0099] In step S105, assuming that the optical structure parameters consist of n parameters, the optical structure parameters of each simplified optical system can be regarded as a point in n-dimensional space, and the optical structure parameters of all repairable simplified optical systems can be regarded as multiple points in the n-dimensional space. Any closed surface in the n-dimensional space that can enclose the multiple points can be used as a recoverability boundary. For example, in one possible embodiment, the minimum convex hull of the multiple points in the n-dimensional space can be determined as the recoverability boundary.

[0100] In step S106, an optical structure parameter may be randomly selected from within the recoverability boundary, and a simplified optical system based on that optical structure parameter may be designed as a second simplified optical system.

[0101] It is understandable that although the first degraded image has been repaired in step S103, the second simplified optical system is different from the first simplified optical system. Therefore, the method of repairing the first degraded image cannot be directly applied to the second simplified optical system. Thus, in step S106, it is necessary to retrain the target neural network suitable for the second simplified optical system.

[0102] In one possible embodiment, the data used to train the target neural network can be obtained through simulation. For example, a degradation simulation can be used to degrade a clear sample image, and the target neural network can be trained using the degraded image as a sample and the clear sample image as the ground truth.

[0103] In another possible embodiment, the data used to train the target neural network is a real pairing dataset, which includes multiple sample data. Each sample data includes a second degraded image, a second degraded parameter, and a reference image obtained by a reference optical system acquiring the sample environment under the second operating condition. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition. The second degraded images included in different sample data are different, and the different second degraded images are obtained by acquiring different sample environments under different second operating conditions.

[0104] Since the second degraded image in the real paired dataset is acquired by the second simplified optical system itself, it can compensate for non-ideal factors of the sensor link (such as noise statistics, nonlinear response, etc.), processing and assembly errors and stray effects that are difficult to fully cover by the simulation degradation, and improve robustness and consistency in the presence of real degradation parameter estimation errors.

[0105] The real-world paired dataset consists of multiple distinct sample datasets. Each sample dataset includes a second degraded image and a reference image. The second degraded image and the reference image are obtained by acquiring the same sample environment using a second simplified optical system and a reference optical system, respectively. The reference optical system in this paper can be any optical system capable of acquiring a clear image; for example, the reference optical system can be an optical system with a complex optical structure.

[0106] Since the reference optical system can acquire a clear image, and considering both the reference image and the second degraded image, the reference image acquired by the reference optical system can be considered as the restored second degraded image. Therefore, during the training of the target neural network, the reference image can be used as the ground truth for supervised learning, enabling the target neural network to learn how to restore the image acquired by the second simplified optical system to a clear image. Furthermore, since the second degraded images in different sample data were acquired under different second operating conditions, the target neural network can further learn how to restore the images acquired by the second simplified optical system under different operating conditions to a clear image.

[0107] The training process will be illustrated below. It is understood that the following example is only one possible training method for the target neural network. In other possible embodiments, the target neural network can also be trained in other ways. This application does not limit it in any way.

[0108] Please see Figure 2 , Figure 2 The diagram shown is a flowchart of a target neural network training method provided in this application, including:

[0109] Step S201: Input the second degraded image and the second degraded parameters from the sample data into the original neural network to obtain the image output by the original neural network, which serves as the second repaired image.

[0110] Step S202: Calculate the difference between the second repaired image and the reference image in the sample data as the first difference.

[0111] As explained above, the reference image can be regarded as the restored second image. Therefore, the second restored image should be as consistent as possible with the reference image. Thus, the first difference can be used to constrain the original neural network to minimize the consistency between the restoration result and the clear reference.

[0112] In one possible embodiment, the first difference can be calculated according to the following formula (3):

[0113] … (3);

[0114] in, S is the second restored image, and L is the reference image. rec The first difference is defined as follows. In other possible embodiments, the first difference may also be calculated in other ways, such as by calculating the cosine similarity between the second restored image and the reference image as the first difference.

[0115] Step S203: Degrade the second restored image using the second degradation parameter to obtain the third degraded image.

[0116] The method of degrading the second restored image should be the same as the method of degrading the clear sample image in step S102 above. Please refer to the relevant description of step S102 above, which will not be repeated here.

[0117] Step S204: Calculate the difference between the third degraded image and the second degraded image, and use it as the second difference.

[0118] It is understandable that the third degraded image is obtained by restoring and degrading the second degraded image in sequence. Since degradation and restoration can be regarded as a pair of inverse processes, the third degraded image should be as consistent as possible with the second degraded image. Therefore, the second difference can be used to constrain the original neural network, so as to constrain the original neural network to restore the degraded image based on the same physical model as the degradation process, thereby achieving physical consistency between restoration and degradation.

[0119] In one possible embodiment, the second difference can be calculated according to the following formula (4):

[0120] … (4);

[0121] in, I is the third degraded image, and L is the second degraded image. phy The second difference is the difference between the third and second degraded images. In other possible embodiments, the second difference may also be calculated in other ways, such as by calculating the cosine similarity between the third and second degraded images.

[0122] Step S205: Adjust the model parameters of the original neural network in the direction of reducing the first difference and the second difference to obtain the target neural network.

[0123] Select Figure 2 The target neural network trained in the manner shown can improve its robustness and consistency through the first difference, achieving physical consistency between restoration and degradation. Therefore, by combining the first and second differences, the target neural network can achieve high restoration quality while maintaining consistency with the real imaging link, thereby improving stability and interpretability across fields of view and operating conditions.

[0124] The above text has explained how to train the target neural network. It can be understood that after training the target neural network, the second simplified optical system is used as the front end, and the final algorithm of the target neural network can form a computational imaging system.

[0125] By using the design method for the computational imaging system provided in this application, the following technical effects can be achieved:

[0126] Firstly, since this application is designed based on the recoverability boundary, it allows the front end to use fewer lenses, lower processing precision, and more relaxed assembly tolerances to complete image acquisition while ensuring that degradation is recoverable. This significantly reduces processing and assembly costs and cycles, reduces the size and weight of the simplified optical system, and reduces hardware redundancy caused by excessive pursuit of optical quality, thereby improving the feasibility of mass production and deployment.

[0127] Secondly, because the degradation physics model in this application is consistent with the real imaging link at the algorithm level and is used to constrain the training and inference of the neural network, the restoration process is more in line with physical laws. Compared with pure data-driven or traditional restoration methods, the computational imaging system designed by the design method of the computational imaging system provided in this application is more likely to maintain stable restoration effect under the conditions of field of view change, working condition drift and prototype batch difference, and enhance the interpretability and engineering controllability of the restoration process.

[0128] Third, the target neural network used in the algorithm of this application introduces physical consistency constraints during training, which enables the algorithm to have iterative upgrade capabilities. Under the condition that the front-end optical hardware remains unchanged, the imaging performance can be continuously improved with data accumulation, expansion of working conditions coverage or update of task indicators. When necessary, it can adapt to changes in system state through a lightweight update mechanism, realize the intelligent evolution path of "hardware simplification and software upgrade", thereby further reducing the overall cost of the computational imaging system and improving its long-term availability.

[0129] The imaging method of the computational imaging system designed using the design method of the computational imaging system provided in this application will be described below. See [link to relevant documentation]. Figure 3 , Figure 3 The image shown is a specific implementation of the computational imaging method based on a simplified optical system provided in this application, including:

[0130] Step S301: Obtain the fourth degraded image and the fourth operating condition acquired by the second simplified optical system.

[0131] The fourth operating condition refers to the operating condition under which the second simplified optical system acquires the fourth degraded image.

[0132] Step S302: Determine the degradation parameters of the light after passing through the second simplified optical system under the fourth operating condition, and use them as the third degradation parameters.

[0133] Step S303: Input the fourth degraded image and the third degraded parameter into the target neural network to obtain the repaired image output by the target neural network as the imaging result.

[0134] Corresponding to the aforementioned design method for a computational imaging system, this application also provides a design apparatus for a computational imaging system, such as... Figure 4 As shown, it includes:

[0135] The first parameter determination module 401 is used to determine the degradation parameters of light after passing through each first simplified optical system under each first operating condition, as the first degradation parameters of each first simplified optical system, wherein the degradation parameters are used to represent the wavefront aberration of light.

[0136] The degradation module 402 is used to degrade the clear image of the sample using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems.

[0137] Repair module 403 is used to repair each of the first degraded images using each of the first degradation parameters respectively, to obtain the first repaired image corresponding to each of the first simplified optical systems.

[0138] The screening module 404 is used to determine the first simplified optical system that meets the preset repairable conditions for the corresponding first repaired image, and to identify it as a repairable simplified optical system.

[0139] Boundary determination module 405 is used to determine an optical structure parameter threshold that includes the optical structure parameters of each of the repairable simplified optical systems, as a recoverability boundary.

[0140] Design module 406 is used to design a second simplified optical system whose optical structure parameters are located within the recoverability boundary, and to train a target neural network for repairing the image acquired by the second simplified optical system.

[0141] In one possible embodiment, the design module trains a target neural network for repairing images acquired by the second simplified optical system, including:

[0142] The target neural network is trained using a real pairing dataset, wherein the real pairing dataset includes multiple sample data, each sample data including a second degraded image and a second degraded parameter obtained by a second simplified optical system acquiring the sample environment under the second operating condition, and a reference image acquired by a reference optical system acquiring the sample environment. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition, and the second degraded image included in different sample data is different, and the different second degraded images are obtained by acquiring different sample environments under different second operating conditions.

[0143] In one possible embodiment, the design module trains the target neural network using a real pairing dataset, including:

[0144] The second degraded image and the second degradation parameter in the sample data are input into the original neural network to obtain the image output by the original neural network, which is used as the second repaired image;

[0145] The second restored image is degraded using the second degradation parameter to obtain a third degraded image;

[0146] The difference between the second restored image and the reference image in the sample data is calculated as the first difference; and the difference between the third degraded image and the second degraded image is calculated as the second difference;

[0147] The model parameters of the original neural network are adjusted in the direction of reducing the first difference and the second difference to obtain the target neural network.

[0148] In one possible embodiment, the preset repairable conditions include: the peak signal-to-noise ratio of the first repaired image is higher than a preset first threshold, and / or, the structural similarity between the first repaired image and the clear sample image is higher than a preset second threshold.

[0149] Corresponding to the aforementioned computational imaging method based on a simplified optical system, this application also provides an apparatus for a computational imaging method based on a simplified optical system, such as... Figure 5 As shown, it includes:

[0150] Input module 501 is used to acquire the fourth degraded image and the fourth operating condition acquired by the second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image;

[0151] The second parameter determination module 502 is used to determine the degradation parameters of the light after passing through the second simplified optical system under the fourth operating condition, as the third degradation parameter;

[0152] Algorithm module 503 is used to input the fourth degraded image and the third degraded parameter into the target neural network to obtain the repaired image output by the target neural network as the imaging result;

[0153] The second simplified optical system and the target neural network are obtained in advance by the design method of any of the computational imaging systems described above.

[0154] This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0155] Memory 603 is used to store computer programs;

[0156] When processor 601 executes a program stored in memory 603, it performs the following steps:

[0157] The degradation parameters of light rays after passing through each of the first simplified optical systems under each first operating condition are determined respectively, and are used as the first degradation parameters of each of the first simplified optical systems, wherein the degradation parameters are used to represent the wavefront aberration of the light rays;

[0158] The clear image of the sample is degraded using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems;

[0159] Each of the first degraded images is repaired using the first degradation parameters respectively, to obtain the first repaired image corresponding to each of the first simplified optical systems;

[0160] A first simplified optical system that satisfies the preset repairable conditions for the corresponding first repaired image is identified as a repairable simplified optical system;

[0161] A threshold for optical structural parameters, including the optical structural parameters of each of the repairable simplified optical systems, is determined as a recoverability boundary.

[0162] A second simplified optical system with optical structure parameters located within the recoverability boundary was designed, and a target neural network was trained to repair the image acquired by the second simplified optical system.

[0163] Alternatively, perform the following steps:

[0164] Acquire a fourth degraded image and a fourth operating condition acquired by a second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image;

[0165] The degradation parameters of light rays passing through the second simplified optical system under the fourth operating condition are determined as the third degradation parameter;

[0166] The fourth degraded image and the third degraded parameter are input into the target neural network to obtain the repaired image output by the target neural network, which is used as the imaging result.

[0167] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0169] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0170] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0171] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the design method of any of the above-described computational imaging systems or the computational imaging method based on a simplified optical system.

[0172] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the design method of any of the computational imaging systems described above or the computational imaging method based on a simplified optical system.

[0173] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0174] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of devices, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A design method for a computational imaging system, characterized in that, The method includes: The degradation parameters of light rays after passing through each of the first simplified optical systems under each first operating condition are determined respectively, and are used as the first degradation parameters of each of the first simplified optical systems, wherein the degradation parameters are used to represent the wavefront aberration of the light rays; The clear image of the sample is degraded using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems; Each of the first degraded images is repaired using the first degradation parameters respectively, to obtain the first repaired image corresponding to each of the first simplified optical systems; A first simplified optical system that satisfies the preset repairable conditions for the corresponding first repaired image is identified as a repairable simplified optical system; A threshold for optical structural parameters, including the optical structural parameters of each of the repairable simplified optical systems, is determined as a recoverability boundary. A second simplified optical system with optical structure parameters located within the recoverability boundary was designed, and a target neural network was trained to repair the image acquired by the second simplified optical system.

2. The method according to claim 1, characterized in that, The training yields a target neural network for repairing images acquired by the second simplified optical system, comprising: The target neural network is trained using a real pairing dataset, wherein the real pairing dataset includes multiple sample data, each sample data including a second degraded image and a second degraded parameter obtained by a second simplified optical system acquiring the sample environment under the second operating condition, and a reference image acquired by a reference optical system acquiring the sample environment. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition, and the second degraded image included in different sample data is different, and the different second degraded images are obtained by acquiring different sample environments under different second operating conditions.

3. The method according to claim 2, characterized in that, The process of training the target neural network using a real pairing dataset includes: The second degraded image and the second degradation parameter in the sample data are input into the original neural network to obtain the image output by the original neural network, which is used as the second repaired image; The second restored image is degraded using the second degradation parameter to obtain a third degraded image; The difference between the second restored image and the reference image in the sample data is calculated as the first difference; and the difference between the third degraded image and the second degraded image is calculated as the second difference; The model parameters of the original neural network are adjusted in the direction of reducing the first difference and the second difference to obtain the target neural network.

4. The method according to claim 1, characterized in that, The preset repairable conditions include: the peak signal-to-noise ratio of the first repaired image is higher than a preset first threshold, and / or the structural similarity between the first repaired image and the clear sample image is higher than a preset second threshold.

5. A computational imaging method based on a simplified optical system, characterized in that, The method includes: Acquire a fourth degraded image and a fourth operating condition acquired by a second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image; The degradation parameters of light rays passing through the second simplified optical system under the fourth operating condition are determined as the third degradation parameter; The fourth degraded image and the third degraded parameter are input into the target neural network to obtain the repaired image output by the target neural network, which is used as the imaging result; The second simplified optical system and the target neural network are obtained in advance by the design method of the computational imaging system as described in any one of claims 1-4.

6. A design apparatus for a computational imaging system, characterized in that, The device includes: The first parameter determination module is used to determine the degradation parameters of light after passing through each first simplified optical system under each first operating condition, as the first degradation parameters of each first simplified optical system, wherein the degradation parameters are used to represent the wavefront aberration of light. The degradation module is used to degrade the clear image of the sample using each of the first degradation parameters to obtain the first degraded image corresponding to each of the first simplified optical systems. The repair module is used to repair each of the first degraded images using each of the first degradation parameters, so as to obtain the first repaired image corresponding to each of the first simplified optical systems. The filtering module is used to determine the first simplified optical system that meets the preset repairable conditions for the corresponding first repaired image, and to identify it as a repairable simplified optical system. A boundary determination module is used to determine an optical structure parameter threshold that includes the optical structure parameters of each of the repairable simplified optical systems, as a recoverability boundary. The design module is used to design a second simplified optical system whose optical structural parameters are located within the recoverability boundary, and to train a target neural network for repairing images acquired by the second simplified optical system.

7. The apparatus according to claim 6, characterized in that, The design module trains a target neural network for repairing images acquired by the second simplified optical system, including: A target neural network is trained using a real-world pairing dataset, wherein the real-world pairing dataset includes multiple sample data. Each sample data includes a second degraded image, a second degraded parameter, and a reference image obtained by a reference optical system acquiring the sample environment under the second operating condition. The second degraded parameter is the degraded parameter after light passes through the second simplified optical system under the second operating condition. Different sample data include different second degraded images, which are obtained by acquiring different sample environments under different second operating conditions; and / or, The design module uses a real pairing dataset to train the target neural network, including: The second degraded image and the second degradation parameter in the sample data are input into the original neural network to obtain the image output by the original neural network, which is used as the second repaired image; The second restored image is degraded using the second degradation parameter to obtain a third degraded image; The difference between the second restored image and the reference image in the sample data is calculated as the first difference; and the difference between the third degraded image and the second degraded image is calculated as the second difference; Adjust the model parameters of the original neural network in the direction of reducing the first difference and the second difference to obtain the target neural network; and / or, The preset repairable conditions include: the peak signal-to-noise ratio of the first repaired image is higher than a preset first threshold, and / or, the structural similarity between the first repaired image and the clear sample image is higher than a preset second threshold; and / or, Operating conditions include one or more of the following parameters: temperature drift, assembly drift, vibration and shock, focal plane offset, and field of view position; and / or, Degradation parameters include one or more of the following: aberration parameters, equivalent pupil, and spatial variation point spread function.

8. A computational imaging device based on a simplified optical system, characterized in that, The device includes: The input module is used to acquire the fourth degraded image and the fourth operating condition acquired by the second simplified optical system, wherein the fourth operating condition is the operating condition under which the second simplified optical system acquires the fourth degraded image; The second parameter determination module is used to determine the degradation parameters of light after passing through the second simplified optical system under the fourth operating condition, as the third degradation parameter; The algorithm module is used to input the fourth degraded image and the third degraded parameter into the target neural network to obtain the repaired image output by the target neural network as the imaging result; The second simplified optical system and the target neural network are obtained in advance by the design method of the computational imaging system as described in any one of claims 1-4.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-4 or 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-4 or 5.

Images