Adaptive aberration correction and color restoration method based on folded super-mixing optical prior
Patent Information
- Application Number
- CN202611526040.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-29
- Publication Date
- 2026-10-30
AI Technical Summary
[0004]本发明实施例提供了一种基于折超混合光学先验的自适应像差校正与色彩还原方法,旨在解决现有技术中用于折超混合镜头的成像图像校正方法所存在的校正质量较差的问题
[0009]本发明提供了一种基于折超混合光学先验的自适应像差校正与色彩还原方法,上述方法包括:接收相机采集的网格化视场图像,根据预置构建规则及光路硬件基础参数构建光学先验库;基于退化模型构建规则及光学先验库构建专属复合退化模型;利用训练数据集及专属复合退化模型训练双支路退化估计网络;接收退化图像,利用光学先验库及训练后的双支路退化估计网络解析得到退化估计解析信息;根据预置复原规则对退化估计解析信息进行复原得到还原图像。本申请中通过训练双支路退化估计网络并对退化图像进行还原,可以实现像素级、逐视场差异化的像差补偿与高精度色彩还原,大幅提升全域成像清晰度与色彩一致性。
Smart Images

Figure CN123020994A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of refractive and hyperbolic imaging processing technology, and in particular to an adaptive aberration correction and color restoration method based on refractive and hyperbolic hybrid optical priors. Background Technology
[0002] With the rapid development of optoelectronic imaging equipment towards ultra-thinness, miniaturization, and integration, hybrid refractive-superimaginary optical systems, combining the high transmittance of traditional refractive lenses with the subwavelength dispersion modulation capabilities of superimaginary lenses, have become a core development direction for lightweight high-end imaging devices such as automotive imaging, portable smart cameras, and minimally invasive medical endoscopes. These systems typically utilize the complementary characteristics of refractive positive dispersion and superimaginary negative dispersion to initially offset some chromatic aberration and geometric distortion at the hardware level, achieving broadband achromatic imaging with minimal lenses and an ultra-thin structure. In practical engineering applications, the industry generally adopts a technical approach combining optical simulation modeling and digital image post-processing to optimize the image quality of such systems. Conventional solutions first obtain the system's basic optical parameters through ray tracing to construct a general imaging degradation model. Then, deep learning networks or traditional filtering algorithms are used to perform globally unified restoration processing on the acquired degraded images, aiming to restore image sharpness and color accuracy.
[0003] However, under complex dynamic operating conditions and large-scale mass production, existing technologies face persistent performance bottlenecks in addressing the unique spatial variational nonlinear degradation of hybrid refractive and hyperrefractive systems. Due to the influence of multiple factors such as manufacturing tolerances, assembly errors, and environmental disturbances, the degree of imaging degradation exhibits a significant three-dimensional non-uniform gradient distribution with varying field of view radius, operating wavelength, and object distance, displaying spatial anisotropy characteristics of weak degradation at the center and strong degradation at the edges. Existing globally unified correction mechanisms struggle to match this coupled degradation field that varies drastically with spatial location, leading to a mismatch in correction degrees of freedom and an inability to simultaneously preserve details in the central region and suppress aberrations in the edge regions. Furthermore, conventional data-driven models lack physical constraints on the refractive-hyperrefractive coupling mechanism, resulting in insufficient correction results in terms of cross-scene generalization, color consistency, and high-frequency texture fidelity, failing to meet the stringent requirements of high-precision imaging equipment for image quality uniformity and stability across the entire field. Existing image correction methods for refractive-hyperrefractive lenses suffer from poor correction quality. Summary of the Invention
[0004] This invention provides an adaptive aberration correction and color restoration method based on a priori optics of refracting and hyper-hybrid lenses, aiming to solve the problem of poor correction quality in existing image correction methods for refracting and hyper-hybrid lenses.
[0005] In a first aspect, embodiments of the present invention provide an adaptive aberration correction and color restoration method based on a priori refractive-hyper-hybrid optics. The method is applied to a resolution terminal, which is communicatively connected to a camera to transmit data. A refractive-hybrid lens group, a microscope, and the camera are arranged sequentially and located on the same optical axis. The method includes: Receive the gridded field-of-view image captured by the camera, and construct an optical prior library corresponding to the gridded field-of-view image according to the preset construction rules and the pre-stored optical path hardware basic parameters; Based on the preset degradation model construction rules and the optical prior library, a corresponding exclusive composite degradation model is constructed. The pre-set dual-branch degradation estimation network is trained based on the pre-set training dataset and the dedicated composite degradation model. The system receives the degraded image captured by the camera, and analyzes the degraded image based on the optical prior library and the trained dual-branch degradation estimation network to obtain the corresponding degradation estimation analysis information. The degradation estimation parsing information is restored according to the preset restoration rules to obtain the corresponding restored image.
[0006] Secondly, embodiments of the present invention also provide an adaptive aberration correction and color restoration system based on a priori refractive-hybrid optics. The system is configured in a resolution terminal, which is communicatively connected to a camera to transmit data. A hybrid refractive-hybrid lens group, a microscope, and the camera are arranged sequentially and located on the same optical axis. The system is used to execute the adaptive aberration correction and color restoration method based on a priori refractive-hybrid optics as described in the first aspect. The system includes: An optical prior library construction unit is used to receive the gridded field-of-view image acquired by the camera and construct an optical prior library corresponding to the gridded field-of-view image according to preset construction rules and pre-stored optical path hardware basic parameters. A dedicated composite degradation model construction unit is used to construct a corresponding dedicated composite degradation model based on the preset degradation model construction rules and the optical prior library. The network training unit is used to train the pre-set dual-branch degradation estimation network based on the pre-set training dataset and the dedicated composite degradation model. The degraded image parsing unit is used to receive the degraded image acquired by the camera, and parse the degraded image according to the optical prior library and the trained dual-branch degradation estimation network to obtain the corresponding degradation estimation parsing information; The image restoration unit is used to restore the degradation estimation parsing information according to the preset restoration rules to obtain the corresponding restored image.
[0007] Thirdly, embodiments of the present invention also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.
[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect.
[0009] This invention provides an adaptive aberration correction and color restoration method based on hybrid optical priors. The method includes: receiving a gridded field-of-view image acquired by a camera; constructing an optical prior library based on preset construction rules and basic optical path hardware parameters; constructing a dedicated composite degradation model based on degradation model construction rules and the optical prior library; training a dual-branch degradation estimation network using a training dataset and the dedicated composite degradation model; receiving a degraded image; resolving degradation estimation information using the optical prior library and the trained dual-branch degradation estimation network; and restoring the degradation estimation information according to preset restoration rules to obtain a restored image. This application, by training a dual-branch degradation estimation network and restoring the degraded image, can achieve pixel-level, field-by-field differentiated aberration compensation and high-precision color restoration, significantly improving global imaging sharpness and color consistency. Attached Figure Description
[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a flowchart illustrating the adaptive aberration correction and color restoration method based on refractive-super-hybrid optical priors provided in an embodiment of the present invention. Figure 2 A schematic diagram illustrating the application scenario of the adaptive aberration correction and color restoration method based on refractive-hybrid optics prior provided in this embodiment of the invention; Figure 3 A schematic block diagram of an adaptive aberration correction and color restoration system based on super-refractive hybrid optical priors provided in an embodiment of the present invention; Figure 4 A schematic block diagram of an electronic device provided in an embodiment of the present invention; Figure 5 The image shows the application effect of the adaptive aberration correction and color restoration method based on super-refractive optics prior provided in the embodiments of the present invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. Embodiments of this invention provide an adaptive aberration correction and color restoration method based on a refractive-hybrid optics prior. This adaptive aberration correction and color restoration method based on a refractive-hybrid optics prior is applied in a parsing terminal, and the method is executed by application software installed in the parsing terminal; as... Figure 2 As shown, the analysis terminal 10 communicates with the camera 13 to transmit data information; the refracting-hybrid lens group 11, the microscope magnifier 12, and the camera 13 are arranged in sequence and located on the same optical axis; the analysis terminal 10 is a terminal device used to execute the adaptive aberration correction and color restoration method based on the refracting-hybrid optical prior to perform aberration correction and color restoration on the degraded image acquired by the camera, such as a desktop computer, laptop computer, tablet computer, or mobile phone.
[0016] The hybrid refractive and super-chromatic lens group 11 is an integrated visible light hybrid imaging lens group, composed of three traditional achromatic refractive lenses and a single metasurface lens cascaded together, capable of receiving and imaging light across the entire visible light band from 430nm to 650nm. The microscope magnifier 12 is a rear-mounted fine imaging adjustment structure, connected to the hybrid refractive and super-chromatic lens group, primarily responsible for fine magnification of the imaging field of view and optimization of beam parallelism, uniformity, and collimation. This optical path, after structural adaptation and optimization, perfectly matches the system's wide effective imaging object distance characteristics from 0.3m to 3m, compensating for beam divergence deviations at different object distances, ensuring consistent clarity across the entire field of view in near-field and medium-to-long-field detection conditions, while also regulating the beam transmission path and suppressing stray light interference, providing a regular, stable, and highly uniform imaging beam for accurate acquisition by the downstream CMOS camera. The Camera 13 is a 12-megapixel high-resolution area-array CMOS industrial camera with a global shutter. The pixel size is 3.2μm × 3.2μm, the frame rate is ≥30fps at full resolution, and it features a global shutter to eliminate motion blur. The dynamic range is ≥70dB, and it supports dual interface data transmission via USB 3.0 (Universal Serial Bus 3.0) and GigE (Gigabit Ethernet). The camera can completely acquire raw imaging data modulated by a refractive-hyperoptical hybrid optical path, providing real and reliable data support for subsequent adaptive correction algorithms and color restoration algorithms based on refractive-hyperoptical hybrid optical priors.
[0017] Figure 1 This is a flowchart illustrating the adaptive aberration correction and color restoration method based on refractive-super-hybrid optical priors provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.
[0018] S110. Receive the gridded field-of-view image acquired by the camera, and construct an optical prior library corresponding to the gridded field-of-view image according to the preset construction rules and the pre-stored optical path hardware basic parameters.
[0019] The terminal establishes a communication connection with the camera to acquire a gridded field-of-view image. This gridded field-of-view image is data obtained by discretizing the imaging field of view according to a preset spatial grid, used to characterize the imaging characteristics of different field-of-view regions. The system pre-stores basic optical path hardware parameters, covering physical structural parameters such as the refractive index of the hybrid lens group, the magnification of the microscope, the objective lens distance, and the transmission band. The gridded field-of-view image can be obtained by discretizing the field-of-view image into a 25×25 grid, thus containing 25×25 grid field-of-view points; the objective lens distance can cover multiple typical working conditions such as 0.3m, 1m, 2m, and 3m, and the transmission band can correspond to five specific bands: 430nm, 480nm, 530nm, 580nm, and 650nm. The construction of the optical prior library can be carried out on a specific precision calibration platform. For example, a constant temperature (25℃), constant humidity (50%), and stray light-free darkroom can be built as the corresponding precision calibration platform to eliminate environmental disturbance errors.
[0020] The construction of the optical prior library is typically performed during system initialization or offline calibration. Based on the band information in the basic optical path hardware parameters, field-view point images corresponding to each band and each grid field-view point are extracted from the gridded field-view image. Ray tracing is performed on the field-view point images using the optical path dispersion offset function to calculate the optical path dispersion offset coefficients. The field-view point images are processed using the transmittance analytical function to obtain the spectral transmittance. The field-view point images are coupled and calculated based on the point spread function (PSF) in the construction rules to generate a point spread function matrix. Finally, the point spread function matrix, optical path dispersion offset coefficients, and spectral transmittance are bound and stored to form an optical prior library containing a three-dimensional prior tensor. This three-dimensional prior tensor fully preserves the coupling degradation characteristics of the hyperrefractive system under spatial field of view, spectral wavelength, and imaging conditions, providing underlying data support for subsequent physical modeling and network training.
[0021] In a specific embodiment, step S110 includes the following sub-steps: obtaining field point images corresponding to each band and each grid field point in the gridded field image according to the band parameters of the optical path hardware basic parameters; performing ray tracing on the field point images according to the optical path dispersion offset function in the optical path hardware basic parameters to obtain the corresponding optical path dispersion offset coefficients; performing transmittance analysis on the field point images according to the transmittance analysis function in the optical path hardware basic parameters to obtain the corresponding spectral transmittance; coupling the field point images according to the point spread function in the construction rules to obtain the corresponding point spread function matrix; binding and storing the point spread function matrix with the optical path dispersion offset coefficients and the spectral transmittance to construct an optical prior library containing three-dimensional prior tensors.
[0022] During system initialization or calibration, the entire imaging field of view is first discretized according to a preset gridding rule, dividing the continuous field of view into several independent grid field of view points. Simultaneously, based on the band parameters defined in the optical path hardware fundamental parameters, the target wavelength range to be sampled (e.g., the visible light band from 430nm to 650nm) and the specific sampling interval are determined. Band parameters include, but are not limited to, the starting wavelength, ending wavelength, and wavelength step size. Based on these parameters, multiple characteristic wavelength points are selected within the set wavelength range. For each characteristic wavelength point and each grid field of view point, the corresponding field of view point image is obtained through ray tracing simulation or actual calibration. The field of view point image characterizes the imaging response data of the optical path system to an ideal point light source or target pattern at a specific spatial location and wavelength. It records the shape, energy distribution, and offset information of the light spot, serving as the input data basis for subsequent calculations of various optical prior parameters.
[0023] For example, gridded parameter calibration was completed for the entire field of view, all wavelengths, and all operating conditions of the optical path. The field of view was discretized into 25×25 grid field points, and the wavelengths were discretized into five characteristic wavelengths: 430nm, 480nm, 530nm, 580nm, and 650nm. Object distances covered four typical operating conditions: 0.3m, 1m, 2m, and 3m. After completing the multi-dimensional optical calibration, a three-dimensional prior tensor structure was constructed to obtain the optical prior library. First, the basic parameters of the optical path hardware were entered, then ray tracing simulations were performed by field of view and wavelength. Finally, the coupling point spread function and spectral transmittance data were three-dimensionally bound and stored to complete the construction of the optical prior library, providing complete and accurate underlying optical data support for the subsequent three-dimensional prior tensor structure construction. To characterize the multi-dimensional coupling degradation characteristics of the hyperrefractive system, a three-dimensional optical prior tensor was constructed in the embodiment. The mathematical definition of the tensor is as follows: ; in, T(x,y,λ) For three-dimensional hyper-hybrid optical prior tensors; (x,y) The image space field of view pixel coordinates represent the difference in spatial dimensional degradation. For example, x takes the value of an integer from 1 to 25, and y takes the value of an integer from 1 to 25. λ The wavelength (corresponding to the above transmission band) characterizes the difference in spectral dimension degradation. PSF(x,y,λ) The point spread function matrix corresponds to the grid field of view points and bands, representing spatial aberration blurring and degradation; τ(x,y,λ) Spectral transmittance of the optical system characterizes color energy decay and degradation. Δn(x,y,λ)The optical path dispersion offset coefficient represents the band coupling color shift characteristics. Through ray tracing simulation and precise calibration, the above parameters are solved grid-by-grid and band-by-band. The discrete calibration data is structured and encapsulated into a three-dimensional prior tensor with dimensions [25,25,5]. 25×25 represents the number of grid field points, and 5 represents the band dimension. This fully preserves the coupling correlation characteristics of the spatial field of view, spectral wavelength, and degradation intensity of the hyperrefractive system. The tensor data is differentiable and can be embedded into networks, providing accurate prior constraints for subsequent physical modeling. A single optical path only requires one offline calibration, and tensor data can be directly reused for batch devices.
[0024] The optical path dispersion shift function is a mathematical expression describing the spatial position shift of light rays of different wavelengths after passing through the combination of refractive lenses and metasurfaces in a hybrid refractive-hybrid lens assembly. The obtained field-of-view image is input into a ray tracing module equipped with the optical path dispersion shift function, and the path of each ray passing through the optical system is calculated. This process simulates the complete transmission path of light rays starting from the object space, undergoing multiple refractions and diffraction modulations in the hybrid refractive-hybrid lens assembly, and finally reaching the image plane. During tracing, the focus is on the difference in imaging position of different wavelengths of light at the same grid field-of-view point. The optical path dispersion shift coefficient is obtained by calculating the spatial distance between the imaging center of the dominant wavelength and the imaging centers of each characteristic wavelength. This coefficient quantifies the axial chromatic aberration and magnification chromatic aberration caused by the coupling of the dispersive characteristics of the refractive element and the metasurface. Its value varies with the field-of-view radius and wavelength, accurately reflecting the unique spatial variational dispersion characteristics of the refractive-hybrid system. The optical path dispersion shift coefficient serves as a key input variable for subsequently constructing the color compensation model and calculating the spatial degradation correction kernel.
[0025] The transmittance analytical function defines the transmission attenuation characteristics of an optical system for light energy of different wavelengths, encompassing the combined effects of lens material absorption, surface reflection loss, and the diffraction efficiency of the metasurface. Using the transmittance analytical function, energy statistical analysis is performed on the field-view image. By calculating the ratio between the input and output luminous flux, the spectral transmittance is obtained. Specifically, for each grid field-view point and each characteristic wavelength, the total energy of the light spot in the field-view image is statistically analyzed and compared with the theoretical incident energy or the energy of a calibration light source to generate spectral transmittance data. Spectral transmittance characterizes the light-transmitting ability of an optical system at a specific spatial location and wavelength, and its value directly affects the brightness and color balance of the image. For hybrid refractive-metasurface systems, due to the significant variation in diffraction efficiency of the metasurface with wavelength, the spectral transmittance often exhibits a nonlinear band dependence. Through transmittance analysis, these physical laws are transformed into numerical spectral transmittance parameters, providing a precise calibration benchmark for the color reproduction branch.
[0026] The point spread function (PSF) is a spatial function describing the imaging response of an optical system to a point source, directly reflecting the aberration and blurring characteristics of the system. Based on the PSF calculation logic in the construction rules, the field-of-view point image is coupled. This coupling process involves numerically fitting and modeling the energy distribution of the light spot recorded in the field-of-view point image, transforming the discrete light intensity distribution into a continuous or discretized matrix form. For complex aberrations such as spherical aberration, coma, and field curvature present in refracto-hyperhybrid systems, methods such as Gaussian fitting or Zernike polynomial decomposition are used to extract the diffusion radius, morphological parameters, and energy distribution characteristics of the light spot, ultimately generating the corresponding PSF matrix. The dimension of this matrix corresponds to the grid division and band division, and each element in the matrix represents the intensity of the optical blur kernel at a specific spatial location and wavelength. The PSF matrix accurately characterizes the spatial variational degradation law of the refracto-hyperhybrid imaging system, serving as the core physical basis for subsequent deconvolution restoration and aberration blur elimination.
[0027] After extracting and calculating the above parameters, data fusion and storage operations are performed to construct the final optical prior library. The point spread function matrix, optical path dispersion offset coefficient, and spectral transmittance are aligned and bound according to the grid field coordinates and band coordinates. Specifically, these three types of data are integrated into a multi-dimensional data structure to form a three-dimensional prior tensor. The three dimensions of this three-dimensional prior tensor correspond to the image space coordinates, respectively. (x,y) Operating wavelength λ The system also includes data type channels, which internally store a complete mapping relationship between field of view, wavelength, and degradation. This tensor-based storage method not only preserves the physical completeness of optical priors but also makes them differentiable and embeddable into networks, allowing them to be directly used as inputs or constraints for deep learning networks. By constructing an optical prior library containing three-dimensional prior tensors, the physical properties of a folded hybrid optical system are digitally encapsulated. This eliminates the need for repeated, complex optical calibrations in mass-produced equipment; accurate physical prior support can be obtained simply by calling this pre-built optical prior library, significantly reducing deployment costs and improving batch-to-batch consistency.
[0028] The above steps, through the coordinated operation of discretization sampling, physical quantity calculation, and structured storage, construct an optical prior library covering the entire field of view and all spectral bands, providing a solid physical data foundation for adaptive aberration correction and color restoration in hybrid optical systems. The three-dimensional prior tensor directly correlates the fundamental parameters of the optical path hardware with the network-computable parameters, enabling subsequent image restoration algorithms to perform pixel-level adaptive correction based on real physical degradation patterns, effectively improving imaging quality and physical interpretability.
[0029] S120. Construct the corresponding exclusive composite degradation model according to the preset degradation model construction rules and the optical prior library.
[0030] The dedicated composite degradation model aims to quantify the coupling relationship between aberration blur and color attenuation in hybrid optics systems, serving as a bridge between physical optics priors and image restoration algorithms. Based on the mathematical expression and optimization constraints defined by pre-defined degradation model construction rules, and combined with the point spread function matrix and spectral transmittance from the optical prior library, a spatial variational composite degradation model is established between the original and degraded images. First, a spatial variational composite degradation model corresponding to the optical prior library is constructed; this degradation model is a general model. Spatial variational constraints or band coupling constraints from the degradation model construction rules are introduced to optimize the general model, resulting in the dedicated composite degradation model. This model breaks through the traditional uniform degradation assumption, accurately characterizing the spatial anisotropy features of weak degradation at the center and strong degradation at the edges of the hybrid optics system, providing strict physical regularization constraints for network training.
[0031] In a specific embodiment, step S120 includes the following sub-steps: constructing a corresponding spatial variational composite degradation model between the original image and the degraded image based on the general imaging degradation formula in the degradation model construction rules and the optical prior library; optimizing the general imaging degradation model based on the constraint terms in the degradation model construction rules to obtain the corresponding specific composite degradation model.
[0032] Based on a pre-defined universal imaging degradation formula and the optical prior library obtained in the preceding steps, a mathematical mapping relationship is established between the original image and the degraded image, thereby constructing a spatial variational composite degradation model. This universal imaging degradation formula describes the physical process by which an ideal, sharp image, after being imaged by a hybrid optical system, is transformed into a degraded image due to spatial blurring, color attenuation, and noise interference. It characterizes the reference transmission law of light energy distribution and geometric aberrations in the imaging system. It defines the degraded image I. deg Original Image I gt The fundamental coupling relationship between the point spread function (PSF), spectral transmittance (τ), and noise term (N) can be formally expressed as: ; The above formula can directly establish a correspondence between the degraded image and the original image (the ideal, clear, and distortion-free original image); where, I deg (x,y,λ) For the acquired degraded image, I gt (x,y,λ) For the ideal, clear, and distortion-free original image, This is a two-dimensional convolution operation. N(x,y,λ) This is the superposition term of Gaussian mixture noise and microstructure stray noise. PSF(x,y,λ)The point spread function matrix characterizes spatial aberration blurring and degradation. τ(x,y,λ) The spectral transmittance of an optical system characterizes the degradation of color energy. (x,y) For pixel coordinates, λ The wavelength (corresponding to the transmission band mentioned above).
[0033] In constructing the aforementioned spatial variational composite degradation model, the core parameters in the general imaging degradation formula—namely, the point spread function matrix and spectral transmittance—are directly retrieved from the optical prior library, rather than using theoretical design values or fixed empirical values. The optical prior library stores the corresponding field-of-view points for different grids. (x,y) and different operating wavelengths λ The measured or simulated optical path parameters are used to ensure that the constructed spatial variational composite degradation model can accurately reflect the true physical state of the current hybrid lens group. By substituting the multidimensional coupling parameters in the optical prior library into the general imaging degradation formula, the transformation from the uniform degradation assumption to spatial variational nonstationary degradation modeling is realized, enabling the model to have the physical characterization capabilities of field-of-view position correlation and wavelength correlation at the baseline level.
[0034] After constructing a basic spatial variational composite degradation model based on a general imaging degradation formula, the basic model is specifically optimized according to the constraints in the pre-defined degradation model construction rules to obtain a dedicated composite degradation model adapted to the characteristics of a refractive-hybrid optical system. This optimization aims to solve the nonlinear degradation problem caused by the strong coupling between refractive residual monochromatic aberration and hyperspectral dispersion, which is unique to refractive-hybrid systems, and in particular, to accurately model the spatial anisotropy characteristics where degradation at the field of view edges is stronger than at the center.
[0035] The core of the optimization lies in introducing a constraint term that characterizes the degradation gradient. This constraint term is configured to constrain the spatial variation characteristics of the point spread function. By adding a constraint term related to spatial location coordinates to the general imaging degradation model, the specific composite degradation model can explicitly describe the gradient law of degradation degree with the field of view radius, thereby capturing the aberration evolution trend from the image center to the edge. This optimization process is formally represented as: ; In the formula, T PSF (x,y,λ) This is a slice of the three-dimensional point spread function tensor extracted from the optical prior library. T τ (x,y,λ) The spectral transmittance parameter is built into the three-dimensional prior tensor. PSF(x,y,λ) α is the PSF spatial gradient operator, used to characterize the field-of-view degradation from the center to the edge. α is the spatial variational weight coefficient, used to adjust the contribution of gradient constraints to the overall model. N(x,y,λ)This is the superposition term of Gaussian mixed noise and microstructure stray noise.
[0036] Spatial variational weighting coefficients α The value is determined based on the statistical characteristics of the edge aberration distribution of the hybrid lens group. For example, α The calibration can be performed based on the design parameters of the target optical system and the measured edge blur gradient. The preferred value range is between 0.5 and 1.2; in this embodiment, it can be set to 0.85 to accurately match the edge degradation gradient characteristics of the super-folded optical path. By introducing the aforementioned gradient constraint term, the dedicated composite degradation model not only inherits the physical resolvability of the general imaging degradation formula but also enhances the fitting accuracy for the spatial variational nonlinear degradation of the super-folded hybrid system. This dedicated composite degradation model, as a physical regularization constraint term, provides strict physical rule boundaries for the subsequent training of the dual-branch degradation estimation network, effectively avoiding overfitting or non-physical divergence of the solution space caused by the lack of physical constraints, and ensuring the consistency between the degradation estimation results and the actual imaging mechanism.
[0037] S130. Train the pre-set dual-branch degradation estimation network based on the pre-set training dataset and the dedicated composite degradation model.
[0038] A dual-branch degradation estimation network is used to decouple the estimation of spatial aberration degradation parameters and color degradation parameters of images in parallel. It includes a shared shallow backbone network, a degradation estimation branch, and a color restoration estimation branch. The pre-built training dataset contains a large number of pairs of training degradation images and their corresponding distortion-free original images.
[0039] The training process iteratively optimizes network weights to accurately predict degradation parameters. First, physical regularization constraints are constructed based on a dedicated composite degradation model. Second, a shared shallow backbone network is used to extract training image features from the degraded image and the original, distortion-free image features, respectively. Subsequently, the degradation estimation branch and color restoration estimation branch, combined with an optical prior library, analyze these features, outputting training aberration branch feature maps, distortion-free aberration branch feature maps, training color compensation coefficients, and distortion-free color compensation coefficients. Finally, the total loss value is calculated based on a preset total loss function and physical regularization constraints, and the network weight parameters are iteratively adjusted using a backpropagation algorithm until convergence. The total loss function integrates spatial aberration correction loss, color restoration loss, image detail fidelity loss, and physical prior constraint loss, using a dynamic weight adjustment mechanism to balance the objectives of each task, ensuring the network possesses high-precision degradation parameter estimation capabilities.
[0040] In a specific embodiment, step S130 includes the following sub-steps: constructing the corresponding physical regularization constraints based on the dedicated composite degradation model; extracting corresponding training image features and distortion-free image features from the training degraded image and the distortion-free original image in the training dataset based on the shared shallow backbone network in the dual-branch degradation estimation network; parsing the training image features and the distortion-free image features based on the degradation estimation branches in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding training aberration branch feature maps and distortion-free aberration branch feature maps; and further processing the dual-branch degradation estimation network. The color restoration estimation branch in the degradation estimation network and the optical prior library respectively analyze the training image features and the distortion-free image features to obtain the corresponding training color compensation coefficients and distortion-free color compensation coefficients; according to the preset total loss function and the physical regularization constraint, the total loss value corresponding to the training aberration branch feature map, the distortion-free aberration branch feature map, the training color compensation coefficients, and the distortion-free color compensation coefficients is obtained; based on the total loss value, the network weight parameters in the dual-branch degradation estimation network are iteratively adjusted to train the dual-branch degradation estimation network.
[0041] During the training and configuration phase of the dual-branch degradation estimation network, the parsing terminal reads the parameters of the dedicated composite degradation model and extracts the constraints on key physical parameters such as the spatial gradient of the point spread function and the attenuation of spectral transmittance. This dedicated composite degradation model accurately quantifies the coupled degradation law of refractive residual aberration and hyperspectral wavelength dispersion. By transforming the spatial variational constraint terms and band coupling constraint terms in the model into regularization terms and embedding them into the loss function, a physical regularization constraint is formed. This physical regularization constraint is used to introduce the physical imaging mechanism of the hybrid optics system during training, forcing the degradation parameters learned by the network to strictly follow the physical propagation laws of the optical system, rather than merely matching the statistical characteristics of the data. This prevents the model from overfitting noise or artifacts in the training data and improves the physical interpretability and generalization ability of the correction results.
[0042] The shared shallow backbone network, serving as the common front-end of the dual-branch degradation estimation network, is responsible for extracting general visual features at the lower levels. Each training data point in the training dataset includes both a degraded training image and a distortion-free image. The image pairs contained in the training data are input into this backbone network; these image pairs include degraded images actually captured by a hybrid lens system and their corresponding distortion-free original images. The shared shallow backbone network is composed of stacked multi-layer convolutional neural networks. Through convolution operations and non-linear activation functions, it maps the input pixel-level image data into a high-dimensional feature space. The network abstracts layer by layer, extracting training image features containing degradation information such as blur and color cast from the degraded images, while simultaneously extracting distortion-free image features reflecting the ideal imaging state from the distortion-free original images. This process not only provides the basic feature representation for subsequent dual-branch refined analysis but also enables the network to learn the inherent rules of mapping from degraded features to distortion-free features through feature alignment. The shared shallow backbone network achieves the reuse of general features such as low-level edges and textures in the dual-branch network, effectively reducing the number of model parameters and computational redundancy.
[0043] The degradation estimation branch receives training image features and distortion-free image features from the shared shallow backbone network, and performs deep analysis and feature mapping using a pre-defined optical prior library. The optical prior library contains a 3D folded-hybrid optical prior tensor, providing prior information such as field-of-view coordinates, wavelength indices, and point spread function matrices. This guides the network to focus on feature channels related to aberration degradation and suppresses interference from irrelevant factors such as illumination changes. The training image features are input into the degradation estimation branch, processed through an internal encoder-decoder structure, and combined with the corresponding reference point spread function constraints from the optical prior library to generate a training aberration branch feature map representing the current spatial aberration distribution state of the image. Similarly, the same analytical operation is performed on the distortion-free image features to obtain the distortion-free aberration branch feature map. These two types of feature maps represent aberration features in the degraded and ideal states, respectively, and are used to subsequently calculate the aberration correction loss, driving the network weights to update in a direction that can accurately estimate the spatial degradation correction kernel.
[0044] The color restoration estimation branch runs parallel to the degradation estimation branch and is specifically designed to address the problem of estimating spectral color degradation. This branch also receives training image features and distortion-free image features from a shared shallow backbone network, and deeply embeds spectral prior information from the optical prior library during the parsing process. The optical prior library stores the spectral transmittance and dispersion shift coefficients of the hyperrefractive hybrid system under different fields of view and different wavelengths, providing a precise physical benchmark for color restoration estimation. Through the color encoder and decoder structure within the color restoration estimation branch, multi-scale analysis is performed on the training image features to extract color degradation traces such as color cast and hue distortion present in the image. These are then combined with the spectral transmittance parameters from the optical prior library for feature modulation, ultimately mapping and outputting the corresponding training color compensation coefficients. Correspondingly, the distortion-free image features are parsed to obtain the distortion-free color compensation coefficients. The difference between the trained color compensation coefficient and the distortion-free color compensation coefficient directly reflects the prediction bias of the network in the color restoration task. This bias is quantified by the color restoration loss function, which guides the network to optimize its color parameter estimation ability and ensures that the final output color compensation coefficient can effectively correct the band inconsistency color difference unique to the superconducting system.
[0045] The overall loss function measures the difference between the output of the dual-branch degradation estimation network and the desired target, aiming to balance multiple optimization objectives such as aberration correction, color restoration, image detail preservation, and physical prior consistency. Based on the trained aberration branch feature maps and the distortion-free aberration branch feature maps obtained in the preceding steps, the spatial aberration correction loss is calculated; based on the trained color compensation coefficients and the distortion-free color compensation coefficients, the color restoration loss is calculated; and the image detail preservation loss is calculated by combining the high-frequency gradients of the distortion-free original image and the restored image. The overall loss function can be expressed as: L PAMF =w 1 ×L aber + w 2 ×L color + w 3 ×L detail + w 4 ×L phy ; in, w 1 , w 2 , w 3 and w 4 For real-time dynamic weights, satisfying normalization constraints w1 + w 2 + w 3 + w 4 =1; where, L aber For spatial aberration correction loss, the L2 norm-constrained PSF estimation error is used, which can be specifically expressed as: ; K(x,y) This indicates the spatial degradation correction kernel used in training degraded images. PSF gt (x,y) This is the spatial degradation correction kernel for the lossless original image corresponding to the training degradation image.
[0046] L color To mitigate color reproduction loss, based on the color difference constraints of the CIE LAB color space and matching the characteristics of human vision, it can be specifically expressed as follows: ; In the formula, ΔL, Δp, and Δq represent the luminance and color channel deviations, and C represents the number of spectral channels, with the value of C being equal to the number of transmission bands.
[0047] L detail To compensate for the loss in image detail preservation, a high-frequency gradient loss is used to retain texture details and suppress overcorrection smoothing, which can be specifically expressed as: ; To train a high-frequency gradient loss for degraded images, This is the high-frequency gradient loss for the distortion-free original image.
[0048] L phy The physical prior constraint loss is obtained by binding a 3D prior tensor regularization constraint to a dedicated composite degradation model. This physical prior constraint loss ensures the physical interpretability of the output information. Specifically, it can be expressed as: ; λ phy The physical constraint weights, for example, can be λ in this instance. phy =0.2, ||•||F is the Frobenius norm of the tensor.
[0049] Simultaneously, the physical regularization constraints constructed in the above steps are added as physical prior constraint loss terms to the loss calculation. The weighted sum of the above losses yields the total loss value for the current training batch. This total loss value characterizes the distance between the network's current performance and its ideal performance. Based on this loss value, the gradient of each layer's parameters in the network is calculated using the backpropagation algorithm, and the network weight parameters corresponding to the dual-branch degradation estimation network are iteratively adjusted using gradient descent or its variant optimization algorithms. One adjustment of the network weight parameters completes one training iteration of the dual-branch degradation estimation network; therefore, iterative training of the dual-branch degradation estimation network can be achieved using multiple training data sets. Specifically, during the training process of the dual-branch degradation estimation network, degradation features corresponding to the training degradation images are extracted through a shared shallow backbone network. The internal network for aberration branch feature mapping operations iteratively updates the aberration branch feature maps under the strong physical regularization constraints of this dedicated composite degradation model. F aber This characterizes the aberration degradation information of each field of view, which is used to solve for the spatial degradation correction kernel. During the training phase, the color restoration loss feeds back the residual between the estimated color compensation coefficient and the prior spectral transmittance to the color encoder, guiding the convolution kernel to capture color degradation features between bands, while filtering out spurious brightness fluctuations caused by image blurring, thus ensuring the feature map... F color It only represents information about spectral color degradation.
[0050] This process is repeated until the total loss value converges or the preset number of training rounds is reached. Through a training method jointly driven by physical regularization constraints and multi-task loss, the dual-branch degradation estimation network achieves synergistic optimization of decoupled estimation of aberration degradation and color degradation. This ensures the physical rationality of the correction parameters while avoiding overfitting or loss of detail that may occur with single-task optimization, thus improving the model's robustness and restoration accuracy in practical applications. Dynamic weight adjustment is implemented during training, adaptively updating weights based on the degradation residuals of each iteration. Severely degraded regions receive increased weights for the corresponding tasks, completely resolving the task imbalance, overfitting, and loss of detail issues caused by fixed weights.
[0051] Furthermore, to achieve bidirectional complementarity and conflict filtering of the original features of the aberration and color branches, eliminate coupling crosstalk errors caused by spatial and spectral degradation, and refine the feature representation accuracy of both branches, thereby constraining the pixel-level PSF correction kernel and channel-level color compensation parameters output by the feature mapping functions of the aberration and color branches to make the relevant mapping results more accurate and physically consistent, adaptive fusion constraints of cross-branch features can be implemented during the training of the dual-branch degradation estimation network. The specific constraint algorithm is as follows: F fision =σ( Faber ⊕ F color ); In the formula: F aber Represents the feature map of aberration branches; F color The color branch feature map is represented by ⊕, which represents the pixel-wise addition and fusion of features; σ(•) is the Sigmoid activation function, which enables adaptive selection of feature weights to avoid decoupling conflicts and feature loss due to degradation. During the model inference stage, only the purified two types of degradation parameters are substituted into the following restoration formula to complete image restoration and correction; feature fusion operations are no longer performed. This implementation designs a block-based local PSF adaptive registration and correction algorithm. The specific execution process of the algorithm is as follows: First, the image field of view is preprocessed by grid segmentation, and the entire image is evenly divided into blocks according to the field of view grid (e.g., 25×25 blocks), realizing the splitting of the center, transition area, and edge field of view regions; then, the single-block reference PSF is retrieved, and the three-dimensional prior tensor is indexed according to the field of view coordinates of each block, matching the reference optical PSF prior parameters at the corresponding position; then, local PSF iterative registration is performed, using the reference PSF as a physical constraint, iteratively optimizing and matching the real-time local degradation kernel of the current image block; finally, channel-by-channel deconvolution restoration is performed, and independent fine-tuning operations are completed for each block and each band. In this embodiment, the initial learning rate lr for network training hyperparameter configuration is 1×10⁻⁶. -4 The cosine annealing learning rate decay strategy is adopted, with a batch size of 16, a total number of epochs of 200, and a weight decay coefficient of 1×10⁻⁶. -5 .
[0052] The above steps, by constructing physical regularization constraints based on a dedicated composite degradation model and integrating them into a multi-task joint optimization framework, achieve end-to-end physical constraint training of the dual-branch degradation estimation network. The physical regularization constraints ensure that the degradation parameters output by the network strictly adhere to the imaging mechanism of the hybrid optics system, while the multi-task loss function balances the competitive relationship between spatial aberration correction, spectral color restoration, and image detail fidelity. The synergistic effect of these steps enables the trained network to accurately estimate pixel-level spatial degradation correction kernels and channel-level color compensation coefficients, providing reliable technical support for subsequent image restoration stages and ultimately achieving high-fidelity, high-resolution adaptive aberration correction and color restoration.
[0053] S140. Receive the degraded image acquired by the camera, and parse the degraded image according to the optical prior library and the trained dual-branch degradation estimation network to obtain the corresponding degradation estimation parsing information.
[0054] After model training, the terminal receives degraded images captured by cameras in real-world application scenarios and uses the trained dual-branch degradation estimation network for inference. After the degraded images are input into the network, a shared shallow backbone network extracts a shared shallow feature map containing basic information such as edges, brightness gradients, and coarse spectral responses.
[0055] The degradation estimation branch, combined with the point spread function matrix from the optical prior library, performs degradation analysis on the shared shallow feature map, generating a spatial degradation correction kernel characterizing the degree of blurring at each pixel location. The color restoration estimation branch, combined with spectral transmittance from the optical prior library, performs color restoration analysis on the shared shallow feature map, generating color compensation coefficients to correct spectral attenuation and color shift at different wavelengths. The spatial degradation correction kernel and color compensation coefficients combine to form degradation estimation analytical information, accurately quantifying the spatial aberration degradation state and spectral color degradation state of the image in a parametric form, providing a basis for subsequent image restoration.
[0056] In a specific embodiment, step S140 includes the following sub-steps: extracting the corresponding shared shallow feature map from the degraded image through the shared shallow backbone network in the dual-branch degradation estimation network; performing degradation analysis on the shared shallow feature map according to the degradation estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding spatial degradation correction kernel; performing color restoration analysis on the shared shallow feature map according to the color restoration estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding color compensation coefficient; and combining the spatial degradation correction kernel with the color compensation coefficient to obtain the corresponding degradation estimation parsing information.
[0057] The shared shallow backbone network, serving as the common pre-feature extraction unit of the dual-branch degradation estimation network, is configured to capture low-level visual information with general representational capabilities from the input degradation image. This network consists of stacked multi-layer convolutional neural networks, employing 3×3 convolutional kernels combined with batch normalization and linear rectified activation functions to effectively preserve low-dimensional features such as edge contours, texture gradients, and basic color responses in the image. Specifically, the shared shallow backbone network comprises three stacked 3×3 convolutional-BN-ReLU modules with a stride of 1 and padding of 1. The shared backbone completes the extraction of general low-level features of the image and outputs a shared shallow feature map. This shared shallow feature map is input to both the degradation estimation branch and the color restoration estimation branch, and the two branches begin independent computation from this point.
[0058] Through the unified processing of this shared shallow backbone network, a consistent input feature benchmark can be provided for subsequent degradation analysis and color restoration analysis, avoiding the feature heterogeneity problem that may occur due to the independent feature extraction of the two branches, thus ensuring the consistency of the understanding of the underlying information when different branches process the same image source.
[0059] The degradation estimation branch focuses on resolving spatial blur degradation features introduced by optical system aberrations in the image. After receiving the shared shallow feature map, this branch extracts deep semantic features related to spatial blur layer by layer through its internal encoder-decoder architecture. The spatial aberration degradation estimation network in the degradation estimation branch is used to resolve and obtain the spatial degradation correction kernel corresponding to the degraded image. This spatial degradation correction kernel accurately describes the inverse filtering parameters required to recover a sharp image from the current degraded image, and its magnitude and distribution directly reflect the aberration intensity of the hybrid optical system under different fields of view.
[0060] The color restoration estimation branch is responsible for analyzing the spectral color shift and energy attenuation characteristics of the image. This branch receives the same shared shallow feature map and uses its internal encoding module to extract the brightness differences and spectral attenuation traces between color channels. During feature extraction, the spectral transmittance parameters corresponding to the current field of view and wavelength are obtained by indexing the optical prior library, and these physical parameters are embedded as color priors into the feature extraction path of the branch network. The introduction of the optical prior library enables the color restoration estimation branch to accurately distinguish between inherent optical color shift and dynamic color shift caused by ambient light, thereby accurately representing the physical mechanism of color degradation in the feature space. Based on the feature map fused with color priors, the output channel-level color compensation coefficients are finally calculated. These coefficients are used to quantify the energy compensation ratio required for different color channels to counteract the dispersion effects in the superconducting optical path.
[0061] After obtaining the spatial degradation correction kernel and color compensation coefficients, the two are combined at the data level to generate degradation estimation analytical information containing a complete description of the degradation parameters. This combination process is not a simple data splicing, but rather a structural association between the spatial dimension's blur correction parameters and the spectral dimension's color compensation parameters, forming a comprehensive parameter set that can jointly drive subsequent image restoration steps. The construction of the degradation estimation analytical information allows the subsequent restoration module to simultaneously understand the spatial inverse filtering requirements and color inverse compensation requirements for the current image, thereby achieving coordinated processing of aberration correction and color restoration in the restoration calculation. Through this combination method, the integrity of the analytical output is ensured, providing accurate and non-conflicting parameter basis for high-quality image restoration.
[0062] Through the aforementioned technical process, the dual-branch degradation estimation network achieves consistent extraction of low-level features using a shared shallow backbone network. Guided by an optical prior library, the degradation estimation branch and the color restoration estimation branch accurately quantify spatial aberration degradation and spectral color degradation, respectively. This decoupled analysis mechanism effectively avoids feature aliasing issues when a single branch processes multiple types of degradation. Simultaneously, the combined degradation estimation analysis information provides a physically interpretable and parameter-complete data foundation for subsequent adaptive restoration, thereby improving the overall correction accuracy and robustness of the hybrid optical imaging system in complex degradation scenarios.
[0063] In a specific embodiment, the step of performing degradation analysis on the shared shallow feature map based on the degradation estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding spatial degradation correction kernel includes: inputting the shared shallow feature map into the spatial aberration degradation estimation network in the degradation estimation branch to obtain the corresponding aberration branch feature map; and inputting the aberration branch feature map and the point spread function matrix in the optical prior library into the aberration branch feature mapping function in the degradation estimation branch to map and obtain the corresponding spatial degradation correction kernel.
[0064] To address the unique spatial variational blurring characteristics of hybrid refracto-hyperoptic imaging, a spatial aberration degradation estimation network is employed for deep feature extraction. This network is configured as a feature extractor capable of capturing non-uniform degradation patterns within the image's spatial domain, rather than a simple convolutional stacking structure. After receiving the shared shallow feature map containing edge, gradient, and coarse spectral response information extracted in the preceding steps, a multi-level nonlinear transformation is used to analyze and encode the differentiated blurring levels at different field-of-view locations (e.g., central and edge fields of view) in the image. The output aberration branch feature map is essentially a high-dimensional representation vector of the local aberration degradation state of the image, and its numerical distribution is positively correlated with the blurring intensity caused by refractive residual aberration and metasurface dispersion coupling in the refracto-hyperoptic system.
[0065] In its implementation, the spatial aberration degradation estimation network in the degradation estimation branch includes an aberration encoder, a 3D prior tensor embedding guidance submodule, an aberration decoder, and an output head. The aberration encoder consists of four sets of residual convolutional modules, which extract multi-scale spatial blur features layer by layer to obtain the deep features of the encoder. Subsequently, the 3D optical prior tensor selects the corresponding reference point spread function matrix from the optical prior library based on the current image position coordinates. T PSF (x,y)The reference point spread function matrix is used. By introducing the pre-stored point spread function from the optical prior library as a physical constraint, the prior optical degradation parameters are mapped into feature vectors and fused with the deep features of the encoder, completing the injection of optical prior information. The fused features are fed into the aberration decoder, which restores the original image resolution through transposed convolution layer by layer upsampling, finally generating the aberration branch feature map. During the fusion process, the reference point spread function parameters provided by the optical prior library can guide the network to focus on the real physical degradation laws and suppress the generation of non-physical artifacts. Finally, through the feature mapping operation of the output layer, a spatial degradation correction kernel representing the degree of blurring at each field of view is output. By gradually expanding the receptive field of the convolutional layer, contextual information is aggregated layer by layer, thereby accurately quantifying the degree of aberration influence on each pixel. The generation process of this aberration branch feature map is carried out under the physical constraints of the dedicated composite degradation model constructed in the previous steps. That is, the features learned by the network must conform to the physical prior of weak degradation at the center and strong degradation at the edge of the hyperrefractive optical system, ensuring that the extracted features have clear physical meaning and avoiding becoming a purely mathematical fit.
[0066] To transform abstract depth features into physical correction parameters applicable to image restoration, an aberration branch feature mapping function is used as a bridge connecting the neural network feature space and the optical physical space. This mapping function is configured as a computational unit capable of fusing image content-aware features with optical hardware prior parameters. The input data consists of two parts: first, the aberration branch feature map generated in the above steps, which represents the specific dynamic degradation information of the image to be processed; and second, the point spread function (PSF) matrix retrieved from the optical prior library, which represents the morphology of the reference imaging blur kernel of the hyperrefractive hybrid optical system under specific field-of-view coordinates.
[0067] The point spread function matrix, as core data in the optical prior library, stores the inherent aberration distribution determined by the fundamental parameters of the optical path hardware, constituting physically interpretable constraints. During the mapping process, the aberration branch feature mapping function performs a fusion operation on the two sets of inputs mentioned above. This operation can be formalized as the following decision function: ; In the formula, K(x,y) Indicates pixel coordinates (x,y) The generated spatial degradation correction kernel; N aber (•) indicates the aberration branch feature mapping operation, which contains learnable network weight parameters. F aber Represents the feature map of aberration branches; T PSF (x,y) This represents the coordinates extracted from the optical prior library.(x,y) The corresponding reference point diffusion function matrix. Through this mapping function, the network can fine-tune and correct the reference PSF according to the actual image content, and finally generate a spatial degradation correction kernel that best matches the current degraded image.
[0068] In a specific embodiment, the step of performing color restoration analysis on the shared shallow feature map based on the color restoration estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding color compensation coefficient includes: inputting the shared shallow feature map into the spectral color restoration estimation network of the color restoration estimation branch to obtain the corresponding color branch feature map; and inputting the color branch feature map and the spectral transmittance in the optical prior library into the color branch feature mapping function in the color restoration estimation branch to map and obtain the corresponding color compensation coefficient.
[0069] The Spectral Color Reproduction Estimation Network (SCERN), a deep neural network module specifically designed to extract color degradation information from image features, focuses on extracting high-dimensional semantic features that characterize brightness differences, color shifts, and spectral attenuation traces across spectral channels. This network employs an encoder-decoder architecture, modeling the feature space for wavelength-dependent dispersion features in hybrid optics systems, rather than using simple fully connected layers or stacked convolutions. Internally, the SCERN consists of multiple convolutional layers that nonlinearly transform the shared shallow feature map of the input to a high-dimensional color feature space, thereby separating out pure spectral degradation information independent of spatial blur.
[0070] After the shared shallow feature maps are fed into the spectral color restoration estimation network, they undergo multi-scale analysis and semantic extraction via internal convolutional kernels and activation functions. Specifically, the spectral color restoration estimation network sequentially includes: a color encoder, a spectral prior embedding guidance submodule, a color decoder, and an output head. The color encoder consists of four stacked residual convolutional modules, focusing on extracting spectral dimension features such as brightness differences between RGB channels, spectral attenuation traces, and color shift. Subsequently, it indexes the three-dimensional prior tensor to read the spectral transmittance parameters at the corresponding locations. T τ (x,y,λ)The process maps spectral transmittance to channel guide vectors, which are then concatenated with deep features from the color encoder. After upsampling by the color decoder to restore the original image resolution, a color branch feature map is generated. During this process, spurious brightness fluctuations caused by spatial aberrations are effectively filtered out, while feature components characterizing spectral color degradation are preserved and enhanced. The final output color branch feature map is essentially a non-linear quantitative description of the color deviations in each RGB channel of the original image. This feature map is not a straightforward pixel color value, but rather an abstract feature matrix containing color gradients, channel correlations, and dispersion direction tensors. This processing achieves decoupling and purification from low-level visual features to specialized color degradation features, laying a data foundation for subsequent accurate calculation of compensation coefficients.
[0071] The spectral transmittance stored in the optical prior library reflects the energy transfer ratio of different wavelengths of light by the optical system. Its value directly reflects the attenuation characteristics of the refractive lenses and metasurfaces in the hybrid refractive-hybrid lens assembly for light of various wavelengths. In a hybrid refractive-hybrid optical system, due to the wavelength dependence of the metasurface, the spectral transmittance exhibits a non-uniform distribution with varying field of view and wavelength. This distribution characteristic is the physical root cause of image color shift and distortion. This spectral transmittance data, constructed based on ray tracing simulation and actual calibration measurements, covers accurate transmittance parameters across the entire field of view and all wavelengths of the system.
[0072] The color branch feature mapping function, as a mathematical mapping model connecting the deep learning feature domain and the physical optics parameter domain, is responsible for transforming the abstract color branch feature map into specific physical compensation parameters that can be used for image restoration. This mapping function maps high-dimensional feature vectors into a set of channel-level scalar coefficients by learning the correlation between network features and optical priors. During the mapping process, the color branch feature map generated in the preceding steps and the spectral transmittance parameters retrieved from the optical prior library are used as inputs. Internally, the mapping function performs fusion operations and weighted mapping on the input feature vector (color branch feature map) and spectral transmittance, thereby outputting color compensation coefficients. These coefficients, as the core parameters for inversely correcting color degradation, characterize the gain adjustment amount required for each color channel. Through subsequent inverse operations, they can offset the dispersion attenuation of the optical system, achieving accurate color restoration. This mapping process can be formally represented as: ; In the formula, This represents the color compensation coefficient obtained from the mapping. λ Indicates wavelength; N color (•) represents the color branch feature mapping function; F color Represents the color branch feature map; T τ (x,y,λ)Represents the spectral transmittance in the optical prior library. (x,y) This represents the image space coordinates. Based on the prior constraints of optical physics, the above mapping process can output color compensation parameters that conform to the real imaging mechanism, thereby ensuring the physical accuracy and visual consistency of color reproduction.
[0073] S150. The degradation estimation parsing information is restored according to the preset restoration rules to obtain the corresponding restored image.
[0074] Based on the degradation estimation analysis information, an inverse degradation compensation operation is performed to restore a high-quality image. Pre-defined restoration rules utilize a spatial degradation correction kernel and color compensation coefficients to perform an inverse transformation on the degraded image. Specifically, deconvolution is performed on the degraded image using the spatial degradation correction kernel to eliminate aberration blur; inverse color compensation is performed on each channel using the color compensation coefficients to correct color cast and spectral attenuation. The restoration formula is expressed as: ; Among them, F(•) and F -1 (•) represent the forward and inverse Fourier transforms, respectively, and ε is the minimum value to prevent division by zero (e.g., set to 10). -8 ), This refers to the color inverse compensation coefficient at a specific wavelength λ. G(x,y) This refers to the adaptive gain adjustment coefficient. G(x,y) The correction value is dynamically adjusted according to the intensity of field-of-view degradation. Smaller values are used in the central weak degradation region to avoid overcorrection, while larger values are used in the edge strong degradation region to enhance the correction effect. This results in a clear, high-fidelity restored image across the entire field of view, achieving adaptive optimization of both field of view and wavelength in the hybrid folding-hyper-hybrid imaging system. For example, G∈[0.9,1.0] in the central weak degradation region and G∈[1.1,1.3] in the edge strong degradation region, thus avoiding overcorrection at the center and undercorrection at the edges. The final output is a clear, high-fidelity restored image across the entire field of view, fully realizing integrated adaptive correction of field of view, wavelength, and operating conditions.
[0075] This embodiment uses industry-standard quantitative indicators combined with the invention's proprietary physical indicators to complete the performance evaluation. The core indicator formulas and measured data are as follows: (1) Modulation transfer function (MTF) (sharpness evaluation): The average MTF of the entire field of view after correction is 0.78, which is much higher than the 0.45 of the traditional fixed PSF correction; (2) The formula for image residual error is as follows: ; "||...||2" means that the calculation is performed using the L2 norm function.
[0076] (3) The color deviation formula is shown in (14): ; (4) Anti-artifact performance under low signal-to-noise ratio: Under the condition of SNR=12dB, the proportion of edge ringing artifacts is as low as 1.5%; (5) The inference speed is 17ms for a single frame of 12 million pixel image to complete the correction, which meets the high-speed mass production detection requirements of 30fps and above.
[0077] The aforementioned technical processes collaboratively address the spatial variational nonlinear degradation problem in hybrid refractive and hyperspectral optical systems caused by the strong coupling between residual refractive aberrations and hyperspectral wavelength dispersion. An optical prior library provides three-dimensional physical prior constraints, including field of view, wavelength, and operating conditions, ensuring the model's accurate representation of the real imaging mechanism. A dedicated composite degradation model transforms the physical priors into a computable mathematical form, providing rigorous training guidance for the dual-branch degradation estimation network. The dual-branch degradation estimation network utilizes shared shallow feature extraction and a dual-path parallel decoupling mechanism to accurately estimate the spatial degradation correction kernel and color compensation coefficients, avoiding feature aliasing common in traditional single-branch networks. The restoration rules perform pixel-level adaptive inverse correction based on the estimated parameters, simultaneously improving image sharpness and color consistency. For example... Figure 5 As shown, the restored image after processing by the above method ( Figure 5 (b) Compared to the original degraded image ( Figure 5 As shown in Figure (a), the edge field blur is significantly eliminated, the color shift is effectively corrected, and the details and textures are clearer, verifying the effectiveness of the scheme in improving the quality of fold-hybrid imaging.
[0078] The adaptive aberration correction and color restoration method based on hybrid refracto-super-optical priors disclosed in the above embodiments includes: receiving a gridded field-of-view image acquired by a camera; constructing an optical prior library according to preset construction rules and basic optical path hardware parameters; constructing a dedicated composite degradation model based on degradation model construction rules and the optical prior library; training a dual-branch degradation estimation network using a training dataset and the dedicated composite degradation model; receiving a degraded image; parsing the degradation estimation parsing information using the optical prior library and the trained dual-branch degradation estimation network; and restoring the degradation estimation parsing information according to preset restoration rules to obtain a restored image. In this application, by training a dual-branch degradation estimation network and restoring the degraded image, pixel-level, field-by-field differentiated aberration compensation and high-precision color restoration can be achieved, significantly improving global imaging sharpness and color consistency.
[0079] Figure 3 This is a schematic block diagram of an adaptive aberration correction and color restoration system based on refractive-hybrid optics priors provided in an embodiment of the present invention. Figure 3As shown, corresponding to the above-mentioned adaptive aberration correction and color restoration method based on refractive-hyperoptical hybrid prior, this invention also provides an adaptive aberration correction and color restoration system based on refractive-hyperoptical hybrid prior. For details, please refer to... Figure 3 The adaptive aberration correction and color restoration system 100 based on hybrid optical priors includes: an optical prior library construction unit 110, used to receive a gridded field-of-view image acquired by the camera, and construct an optical prior library corresponding to the gridded field-of-view image according to preset construction rules and pre-stored optical path hardware basic parameters; a dedicated composite degradation model construction unit 120, used to construct a corresponding dedicated composite degradation model according to preset degradation model construction rules and the optical prior library; a network training unit 130, used to train a preset dual-branch degradation estimation network according to a preset training dataset and the dedicated composite degradation model; a degradation image parsing unit 140, used to receive a degradation image acquired by the camera, and parse the degradation image according to the optical prior library and the trained dual-branch degradation estimation network to obtain corresponding degradation estimation parsing information; and an image restoration unit 150, used to restore the degradation estimation parsing information according to preset restoration rules to obtain a corresponding restored image.
[0080] The adaptive aberration correction and color restoration system based on refracto-hybrid optical priors provided in this embodiment of the invention applies the aforementioned adaptive aberration correction and color restoration method based on refracto-hybrid optical priors. It receives a gridded field-of-view image acquired by a camera, constructs an optical prior library based on preset construction rules and basic optical path hardware parameters, constructs a dedicated composite degradation model based on degradation model construction rules and the optical prior library, trains a dual-branch degradation estimation network using a training dataset and the dedicated composite degradation model, receives a degraded image, and uses the optical prior library and the trained dual-branch degradation estimation network to parse and obtain degradation estimation parsing information. Finally, it restores the degradation estimation parsing information according to preset restoration rules to obtain a restored image. In this application, by training a dual-branch degradation estimation network and restoring degraded images, pixel-level, field-by-field differentiated aberration compensation and high-precision color restoration can be achieved, significantly improving global imaging clarity and color consistency.
[0081] The aforementioned adaptive aberration correction and color restoration system based on super-hybrid optics priors can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.
[0082] Please see Figure 4 , Figure 4This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device can be a parsing terminal used to perform phase correction and color restoration on degraded images acquired by a camera by executing an adaptive aberration correction and color restoration method based on a hybrid optics prior.
[0083] See Figure 4 The computer device 500 includes a processor 502, a memory, and a communication interface 505 connected via a communication bus 501. The memory may include a storage medium 503 and internal memory 504.
[0084] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute an adaptive aberration correction and color restoration method based on a hybrid optics prior. The storage medium 503 may be a volatile storage medium or a non-volatile storage medium.
[0085] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.
[0086] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an adaptive aberration correction and color restoration method based on super-refractive optics prior.
[0087] This communication interface 505 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0088] The processor 502 is used to run the computer program 5032 stored in the memory to implement the corresponding functions in the adaptive aberration correction and color restoration method based on super-hybrid optical priors.
[0089] Those skilled in the art will understand that Figure 4 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 4The embodiments shown are consistent and will not be repeated here.
[0090] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0091] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium stores a computer program, wherein when executed by a processor, the computer program implements the steps included in the above-described adaptive aberration correction and color restoration method based on refractive-hybrid optics priors.
[0092] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.
[0093] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.
[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.
[0095] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.
[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. An adaptive aberration correction and color restoration method based on refractive-hybrid optics prior, wherein the method is applied to a resolution terminal, and the resolution terminal communicates with a camera to transmit data information; The hybrid lens assembly, the microscope, and the camera are arranged sequentially and located on the same optical axis, characterized in that... The method includes: Receive the gridded field-of-view image captured by the camera, and construct an optical prior library corresponding to the gridded field-of-view image according to the preset construction rules and the pre-stored optical path hardware basic parameters; Based on the preset degradation model construction rules and the optical prior library, a corresponding exclusive composite degradation model is constructed. The pre-set dual-branch degradation estimation network is trained based on the pre-set training dataset and the dedicated composite degradation model. The system receives the degraded image captured by the camera, and analyzes the degraded image based on the optical prior library and the trained dual-branch degradation estimation network to obtain the corresponding degradation estimation analysis information. The degradation estimation parsing information is restored according to the preset restoration rules to obtain the corresponding restored image.
2. The adaptive aberration correction and color restoration method based on super-refractive hybrid optical priors according to claim 1, characterized in that, The process of constructing a corresponding proprietary composite degradation model based on preset degradation model construction rules and the optical prior library includes: Based on the general imaging degradation formula in the degradation model construction rules, the optical prior library constructs a corresponding spatial variational composite degradation model between the original image and the degraded image; The general imaging degradation model is optimized based on the constraints in the degradation model construction rules to obtain the corresponding specific composite degradation model.
3. The adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in claim 1 or 2, characterized in that, The step of parsing the degraded image based on the optical prior library and the trained dual-branch degradation estimation network to obtain corresponding degradation estimation parsing information includes: The corresponding shared shallow feature map is extracted from the degraded image through the shared shallow backbone network in the dual-branch degradation estimation network; Based on the degradation estimation branches in the dual-branch degradation estimation network and the optical prior library, degradation analysis is performed on the shared shallow feature map to obtain the corresponding spatial degradation correction kernel; Based on the color restoration estimation branch in the dual-branch degradation estimation network and the optical prior library, color restoration analysis is performed on the shared shallow feature map to obtain the corresponding color compensation coefficients. The spatial degradation correction kernel is combined with the color compensation coefficient to obtain the corresponding degradation estimation analytical information.
4. The adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in claim 3, characterized in that, The step of performing degradation analysis on the shared shallow feature map based on the degradation estimation branches in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding spatial degradation correction kernel includes: The shared shallow feature map is input into the spatial aberration degradation estimation network in the degradation estimation branch to obtain the corresponding aberration branch feature map; The aberration branch feature map and the point spread function matrix in the optical prior library are input into the aberration branch feature mapping function in the degradation estimation branch to obtain the corresponding spatial degradation correction kernel.
5. The adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in claim 4, characterized in that, The step of performing color restoration analysis on the shared shallow feature map based on the color restoration estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding color compensation coefficients includes: The shared shallow feature map is input into the spectral color restoration estimation network of the color restoration estimation branch to obtain the corresponding color branch feature map. The color branch feature map and the spectral transmittance in the optical prior library are input into the color branch feature mapping function in the color restoration estimation branch to obtain the corresponding color compensation coefficient.
6. The adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in claim 5, characterized in that, The step of training the pre-set dual-branch degradation estimation network based on the pre-set training dataset and the dedicated composite degradation model includes: The corresponding physical regularization constraints are constructed based on the aforementioned proprietary composite degradation model; Based on the shared shallow backbone network in the dual-branch degradation estimation network, the corresponding training image features and distortionless image features are extracted from the training degradation image and the distortionless original image in the training dataset, respectively. The training image features and the distortion-free image features are analyzed based on the degradation estimation branch in the dual-branch degradation estimation network and the optical prior library to obtain the corresponding training aberration branch feature map and distortion-free aberration branch feature map. Based on the color restoration estimation branch in the dual-branch degradation estimation network and the optical prior library, the training image features and the distortion-free image features are analyzed to obtain the corresponding training color compensation coefficients and distortion-free color compensation coefficients. The total loss value corresponding to the training aberration branch feature map, the distortion-free aberration branch feature map, the training color compensation coefficient, and the distortion-free color compensation coefficient is obtained according to the preset total loss function and the physical regularization constraint. The dual-branch degradation estimation network is trained by iteratively adjusting the network weight parameters in the dual-branch degradation estimation network based on the total loss value.
7. The adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in claim 6, characterized in that, The process of constructing an optical prior library corresponding to the gridded field of view image based on preset construction rules and pre-stored optical path hardware parameters includes: Based on the band parameters of the optical path hardware basic parameters, obtain the field point images corresponding to each band and each grid field point in the gridded field image; Ray tracing is performed on the field point image based on the optical path dispersion offset function in the optical path hardware basic parameters to obtain the corresponding optical path dispersion offset coefficient. The transmittance of the field point image is analyzed based on the transmittance analysis function in the optical path hardware basic parameters to obtain the corresponding spectral transmittance. The field point image is coupled according to the point spread function in the construction rule to obtain the corresponding point spread function matrix; The point spread function matrix is bound and stored with the optical path dispersion offset coefficient and the spectral transmittance to construct an optical prior library containing three-dimensional prior tensors.
8. An adaptive aberration correction and color restoration system based on super-hybrid optical priors, characterized in that, The system is configured on a resolution terminal, which communicates with a camera to transmit data. A hybrid refractive and hyper-hybrid lens group, a microscope, and the camera are arranged sequentially and located on the same optical axis. The system is used to execute the adaptive aberration correction and color restoration method based on hybrid refractive and hyper-hybrid optical priors as described in any one of claims 1-7. The system includes: An optical prior library construction unit is used to receive the gridded field-of-view image acquired by the camera and construct an optical prior library corresponding to the gridded field-of-view image according to preset construction rules and pre-stored optical path hardware basic parameters. A dedicated composite degradation model construction unit is used to construct a corresponding dedicated composite degradation model based on the preset degradation model construction rules and the optical prior library. The network training unit is used to train the pre-set dual-branch degradation estimation network based on the pre-set training dataset and the dedicated composite degradation model. The degraded image parsing unit is used to receive the degraded image acquired by the camera, and parse the degraded image according to the optical prior library and the trained dual-branch degradation estimation network to obtain the corresponding degradation estimation parsing information; The image restoration unit is used to restore the degradation estimation parsing information according to the preset restoration rules to obtain the corresponding restored image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the adaptive aberration correction and color restoration method based on refractive-hybrid optics prior as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the adaptive aberration correction and color restoration method based on refractive-super-hybrid optical priors as described in any one of claims 1-7.
