Structural parameter optimization method of optical lens, electronic equipment and storage medium

Abstract

The invention discloses a structure parameter optimization method of an optical lens, electronic equipment and a storage medium, and the method comprises a plurality of forward propagation stages used for determining evaluation indexes of structure parameters of the optical lens, and a backward propagation stage used for optimizing the structure parameters. Determining and only recording intermediate gradient information corresponding to the current stage in each forward propagation stage, and combining the intermediate gradient information into target gradient information; the structure parameters are adjusted and optimized according to target gradient information in the backward propagation stage, and the target gradient information comprises the gradient of the evaluation index relative to the structure parameters.

Description

Technical Field

[0001] The present application relates to the field of end-to-end design of optical lenses, and in particular to a structural parameter optimization method, electronic device and storage medium of an optical lens. Background Art

[0002] The traditional method of optical lens parameter design is to guide lens design through the evaluation function at the aberration level. The conventional end-to-end design method can simulate the imaging effect of the lens and use the evaluation function at the image level to guide the lens design. This end-to-end design method takes the influence of the back-end image processing algorithm into the lens design process, so that the designed optical lens is most compatible with the back-end algorithm. At the same time, using images instead of aberrations to guide lens design is also more intuitive.

[0003] The traditional lens design method uses the finite difference method to calculate the gradient of the evaluation index relative to the lens parameters. However, in the finite difference method, the step size must be infinitely small so that the gradient calculation result can be consistent with the automatic differentiation method result. In actual calculation, it is obviously impossible to make the step size infinitely small.

[0004] Some current end-to-end optical lens design methods use the AD (Auto Difference) algorithm, which has higher gradient solution accuracy than the finite difference method, but has higher requirements for the storage space of the computing device. As the lens structure becomes more complex, the more parameters need to be optimized and the number of rays to be traced increases, and the more gradient information needs to be saved. Therefore, the existing AD algorithm has extremely high requirements for the memory capacity of the GPU (Graphics Processing Unit), and cannot meet the structural parameter design of complex lenses. Summary of the invention

[0005] The technical solution provided by this application at least partially solves the above technical problems.

[0006] According to a first aspect of the present application, a method for optimizing structural parameters of an optical lens is provided, comprising a plurality of forward propagation stages for determining evaluation indicators of structural parameters of the optical lens, and a backward propagation stage for optimizing the structural parameters, wherein in each forward propagation stage, intermediate gradient information corresponding to the current stage is determined and recorded only, and the intermediate gradient information is merged into target gradient information; in the backward propagation stage, the structural parameters are adjusted and optimized according to the target gradient information, wherein the target gradient information includes the gradient of the evaluation indicator relative to the structural parameters.

[0007] In some embodiments, the forward propagation of each stage includes: according to the current structural parameters of the optical lens, based on the ray tracing algorithm, calculating the propagation process of light in the optical lens, generating the point spread function of the optical lens in each field of view; generating a simulated image of the optical lens according to the point spread function; and constructing an image-level evaluation function of the simulated image to obtain an evaluation index of the current structural parameters; the backward propagation stage includes: adjusting and optimizing the current structural parameters based on a gradient-based optimization algorithm, and iteratively calculating the evaluation index by repeating the forward propagation stage until the evaluation index is reduced to a preset target value, and then using the current structural parameters as the optimized target parameters.

[0008] In some embodiments, the intermediate gradient information includes a first gradient and a second gradient; the multiple forward propagation stages include a first stage for determining and recording only the first gradient, and a second stage for determining and recording only the second gradient; wherein the first gradient includes a gradient of an evaluation index relative to a point spread function; and the second gradient includes a gradient of a point spread function relative to the evaluation index.

[0009] In some embodiments, the intermediate gradient information includes a gradient of the evaluation index relative to the simulated image, a gradient of the simulated image relative to the point spread function, and a gradient of the point spread function relative to the structural parameter.

[0010] In some embodiments, determining and recording only the first gradient in the first stage includes: while generating a simulated image, saving the first sub-gradient of the simulated image relative to the point spread function; while constructing an image-level evaluation function to obtain an evaluation index, saving the second sub-gradient of the evaluation index relative to the simulated image; and merging the first sub-gradient and the second sub-gradient according to the chain rule to obtain the gradient of the evaluation index relative to the point spread function, and recording it as the first gradient.

[0011] In some embodiments, the method further includes: after recording the first gradient, clearing the saved first sub-gradient and second sub-gradient.

[0012] In some implementations, determining and recording the second gradient in the second stage includes: while generating the point spread function, saving the gradient of the point spread function relative to the evaluation index, and recording the gradient as the second gradient.

[0013] In some embodiments, the method further includes: after obtaining the target gradient information, clearing the recorded first gradient and second gradient.

[0014] In some implementations, in the back propagation stage, according to the target gradient information, the current structural parameters of the optical lens are adjusted and optimized based on the following gradient-based optimization algorithm:

[0015]

[0016] In the formula, is the current structural parameter of the optical lens, is the adjusted structural parameter, and α is the learning rate used to control the optimization step size.

[0017] In some embodiments, when calculating the propagation process of light in the optical lens based on the ray tracing algorithm, the value of the parameter t is iteratively updated in the following manner to solve the intersection point of the light and the optical surface of the optical lens:

[0018]

[0019] Where F(t) is the surface equation of the optical surface, F′(t) is the gradient of the surface equation F(t) with respect to the parameter t, is the origin, d is the direction of light propagation, parameter t is the value that makes the surface equation F(t) = 0, and t0 is the initial value of parameter t.

[0020] In some implementations, the method further includes: after solving the value of the parameter t, clearing the gradient information F′(t) saved during the iterative update process.

[0021] In some implementations, a network model with a ResUnet structure is used in the image processing algorithm, and the network model includes 3 downsampling layers and 3 upsampling layers.

[0022] In some embodiments, the evaluation function includes an L1 Loss loss function or an L2 Loss loss function.

[0023] In some embodiments, the structural parameters of the optical lens include at least one of the following parameters or any combination thereof: curvature, thickness, and aspheric coefficient.

[0024] According to the second aspect of the present application, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the first aspect of the present application.

[0025] According to the third aspect of the present application, a computer-readable storage medium is also provided, on which machine-executable instructions are stored. When the machine-executable instructions are executed, the machine executes the method according to the first aspect of the present application.

[0026] According to the structural parameter optimization method of the optical lens provided in the embodiment of the present application, by adopting a step-by-step gradient storage strategy, the forward propagation process is divided into multiple stages, in each stage only the intermediate gradient information of the current stage is stored, and in some embodiments, some useless gradient information in the process is promptly cleared, thereby reducing the video memory overhead of the end-to-end design method of the optical lens, and maximizing the complexity of the designable optical lens under limited video memory resources.

[0027] In addition, according to the structural parameter optimization method of the optical lens provided in some embodiments of the present application, by optimizing the gradient storage scheme in the ray tracing algorithm, redundant gradient information in the intermediate iteration process is discarded, thereby further saving video memory space.

[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Other features, purposes and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present application. In the drawings:

[0030] Figure 1 is a flow chart of a method for optimizing structural parameters of an optical lens according to an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the process and recorded gradient of the structural parameter optimization method of the traditional optical lens;

[0032] Figure 3 It is a schematic diagram of the basic principle of calculating the intersection of light and optical surface based on ray tracing algorithm;

[0033] Figure 4 1 is a schematic diagram of the main steps of the method for optimizing the structural parameters of an optical lens according to the first embodiment of the present application;

[0034] Figure 5 is a flow chart of a method for optimizing the structural parameters of an optical lens according to the first embodiment of the present application;

[0035] Figure 6 is a schematic diagram of a process of solving the intersection of a ray and an optical surface according to the ray tracing algorithm of the second embodiment of the present application;

[0036] Figure 7 is a schematic diagram of the structure of a mobile phone lens according to an application example of the present application;

[0037] Figure 8 This is a diagram showing how the GPU memory space changes with the number of rays.

[0038] Fig. 9 is a schematic diagram of the structure of an image restoration network according to an application example of the present application; and

[0039] Fig.10 It is a schematic diagram of the composition of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] It should also be understood that expressions such as "include", "including", "have", "contain" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0042] In the description of the embodiments of the present application, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, the order in which the steps are described does not necessarily refer to the order in which the steps are performed, unless it can be determined from the context that the steps are performed sequentially.

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. Any one or more embodiments or elements thereof, or the arrangement or combination of some or all embodiments or elements thereof described herein are merely exemplary and are not intended to limit the scope of the present application.

[0044] The following is a description of the exemplary embodiments of the present application in conjunction with the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application.

[0045] According to an embodiment of the present application, a method for optimizing the structural parameters of an optical lens is provided, which can be used to simulate the imaging quality of an optical lens by means of image rendering, and use an image-level evaluation function to guide the design of the optical lens.

[0046] The method for optimizing the structural parameters of an optical lens according to an embodiment of the present application may include a plurality of forward pass stages for determining evaluation indicators of the structural parameters of the optical lens, and a backward pass stage for optimizing the structural parameters. The method may include:

[0047] In each forward propagation stage, determine and record only the intermediate gradient information corresponding to the current stage, and merge the intermediate gradient information into the target gradient information;

[0048] In the back propagation stage, the structural parameters are adjusted and optimized according to the target gradient information,

[0049] The target gradient information may include the gradient of the evaluation index relative to the structural parameter, and the intermediate gradient information may include the first gradient of the evaluation index relative to the point spread function of the optical lens in each field of view, and the second gradient of the point spread function relative to the structural parameter. Alternatively, the intermediate gradient information may also include more gradients for merging to obtain the target gradient information. More details about the intermediate gradient information and the target gradient information will be described in detail below.

[0050] Figure 1 FIG. 1 shows a processing flow of a method 100 for optimizing the structural parameters of an optical lens according to an exemplary embodiment of the present application. Figure 1 As shown, the method 100 as a whole may include:

[0051] Step 10, forward pass process; and

[0052] Step 20, Backward Pass process.

[0053] As an exemplary embodiment, the forward propagation includes k stages, that is, divided into k steps: steps 110, 120, ... 1k0. Among them, step 110 represents the first stage, step 120 represents the second stage, ... step 1k0 represents the kth stage. The value of k corresponds to the number of forward propagations, and k is a natural number greater than or equal to 2.

[0054] Combination Figure 1 , taking step 110 of the first stage as an example, the forward propagation of each stage may include the following sub-steps:

[0055] Step 111, import the current structural parameters of the optical lens. For example, taking an optical lens including m structural parameters as an example, it can be represented as a column vector Where m is a natural number. The structural parameters of the optical lens may include but are not limited to curvature, thickness, aspheric coefficient, etc.

[0056] Step 112, based on the current structural parameters of the imported optical lens, a PSF (Point Spread Function) is generated under each field of view of the optical lens based on a ray tracing algorithm. In step 112, the propagation process of light in the optical lens can be calculated based on a ray tracing algorithm. For example, based on the working wavelength and the number of fields of view of the optical lens, multiple rays of each wavelength under each field of view are traced to generate a point spread function under each field of view of the optical lens. n is the number of fields of view specified by the user. If the function f p represents the above PSF generation process, then the optical structure parameters With the point spread function The relationship between them is as follows:

[0057] Step 113, based on the point spread function of each field of view of the optical lens, generate a simulated image formed by the optical lens. In step 113, based on the PSF of each field of view of the optical lens, various image processing algorithms can be used to simulate the image formed by the optical lens to obtain a simulated image. Where M and N are the pixel sizes of the simulated image. If the function f I Represents a simulated image The generation process of PSF and simulation image The following relationship is satisfied:

[0058] Step 114, constructing an image level evaluation function of the simulation image to obtain an evaluation index of the current structural parameter. ε , and get the evaluation index ε. Evaluation index ε and simulation image Satisfies the following relationship: Among them, the evaluation function f ε It can be any image-level evaluation function, for example, L1 Loss, L2 Loss loss function, etc., which can be used to guide the optimization of structural parameters of optical lenses.

[0059] The forward propagation process of step 10 also includes determining and saving only the intermediate gradient information corresponding to the current stage in each stage, for example, by generating a computational graph, saving the intermediate gradient information in the computational graph, and then merging the saved intermediate gradient information according to the chain rule to obtain the target gradient information for guiding parameter optimization, and the target gradient information can be, for example, the gradient of the evaluation function relative to the structural parameters. The chain rule is a derivation rule in calculus, which is used to find the derivative of a composite function.

[0060] In an exemplary embodiment, the back propagation process of step 20 may include: adjusting and optimizing the structural parameters of the optical lens based on a gradient-based optimization algorithm according to the target gradient information so that the evaluation index ε is as small as possible until the evaluation index ε is reduced to below a preset target value.

[0061] Exemplarily, in step 20, a method based on gradient descent may be used to adjust the structural parameters of the optical lens, and the specific formula is as follows:

[0062]

[0063] In the formula, is the adjusted structural parameter, and α is the learning rate used to control the optimization step size.

[0064] It should be understood that, in addition to the above-mentioned gradient descent optimization algorithm, other gradient-based optimization algorithms can also be applied to the update optimization of optical lens structural parameters.

[0065] After adjusting the current structural parameters of the optical lens, return to the forward propagation process, re-import the adjusted structural parameters as the current structural parameters, and gradually calculate the evaluation index ε. If the evaluation index ε is less than the preset target value, it is considered that the iteration termination condition has been met, the parameter optimization process is terminated, and the current structural parameters are saved as the optimized target parameters; otherwise, continue to repeat the forward propagation process to iteratively calculate the evaluation index ε until the evaluation index ε is less than the preset target value, thereby obtaining the optical lens structure with optimized structural parameters.

[0066] In the structural parameter optimization method of optical lenses based on end-to-end design, obtaining the target gradient information is an important step in the optimization of optical lens parameters. As mentioned above, the AD algorithm is a commonly used derivation method in the current end-to-end design method of optical lenses. It solves the target gradient information based on the chain rule. The AD algorithm has higher gradient solution accuracy, but has higher requirements on the storage space of the computing device.

[0067] Figure 2 The process and mathematical expression of the traditional optical lens structural parameter optimization method are shown. Figure 2As shown, if the gradient of the evaluation function relative to the structural parameters is As the target gradient information, the target gradient information can be divided into the following three parts: the gradient of the evaluation function relative to the simulation image Gradient of simulated image relative to PSF and the gradient of the PSF with respect to the structural parameters According to the chain rule, the target gradient information can be obtained by combining the above gradient information The specific method is as follows:

[0068]

[0069] According to the traditional method of optimizing the structural parameters of optical lenses, in order to implement the AD algorithm, in addition to calculating the PSF, simulation image, and evaluation indicators in sequence during a forward propagation process, it is also necessary to generate a calculation graph to save the gradients calculated at each step. And according to the chain rule, the gradients of each step saved in the calculation graph are used to merge and calculate the target gradient information. Then, when the backward propagation is turned on, the target gradient information can be used to guide the optimization of the structural parameters.

[0070] However, the storage space of the computational graph is proportional to the amount of computation involved in the forward propagation process. The more parameters an optical lens has, the more light rays it has, and the higher the image resolution, the greater the amount of computation and the larger the storage space required for the computational graph. In short, the more complex the lens structure (the more parameters it has), the higher the storage space required. Therefore, in practical applications, for optical lens designs with more lens parameters, when using existing AD algorithms, the computational graph used to save the gradient information of each part will take up a lot of storage space.

[0071] In order to solve the above-mentioned problems, according to the method for optimizing the structural parameters of an optical lens provided in an embodiment of the present application, the forward propagation process is divided into multiple stages, and in each stage only the intermediate gradient information corresponding to the current stage is saved; and the intermediate gradient information of multiple stages saved is merged according to the chain rule to obtain the target gradient information for guiding parameter optimization, such as the gradient of the evaluation index relative to the structural parameter.

[0072] For example, the forward propagation process can be divided into k stages (i.e., k steps), where the value of k corresponds to the number of forward propagations, and the target gradient information It can be solved by the following formula:

[0073]

[0074] Wherein, k is a natural number greater than or equal to 2, and the size of k depends on the memory space of the GPU. Generally speaking, the smaller the memory space of the computing device, the larger k is.

[0075] According to the existing AD algorithm, only one forward propagation process is used to record all gradient information in the calculation graph, which greatly occupies the GPU video memory space. When the GPU video memory is xGB, only xGB of gradient information can be stored at most. However, the "step-by-step" gradient storage strategy of the above-mentioned implementation mode of the present application is adopted, and the forward propagation process is divided into k stages, and each stage only stores the intermediate gradient information corresponding to the current stage. On the whole, it is equivalent to storing xGB of gradient information in each step, and theoretically k*xGB of gradient information can be stored. In this way, more gradient information is saved through multiple forward propagations while the video memory space remains unchanged. Accordingly, the cost of introducing step-by-step gradient storage is only the increase in the number of forward propagations, that is, the increase in computing time. In practical applications, the increase in computing time brought about by this can be ignored.

[0076] In addition, the end-to-end design method of the optical lens generates the PSF based on the ray tracing algorithm during the forward propagation process. The ray tracing algorithm is used to calculate the intersection of the light and the optical surface of the optical lens, as well as the propagation direction of the light after refraction by the optical surface.

[0077] The basic principle of calculating the intersection of light and optical surface based on ray tracing algorithm is as follows: Figure 3 As shown, from the origin The propagation direction is The light form can be expressed as follows:

[0078]

[0079] If the ray intersects the surface F at point Then there must be a parameter t such that the surface equation The value of parameter t can be solved by Newton's method iteration. Exemplarily, the specific solution process may include:

[0080] (1) Assign an initial value t0 to the parameter t.

[0081] (2) When calculating t0, the surface equation The value of .

[0082] (3) Update the value of parameter t according to the following formula:

[0083]

[0084] Where F(t) is the surface equation, and F′(t) is the gradient of the surface equation F(t) with respect to the parameter t.

[0085] (4) Calculate the value of the surface equation F(t) at the current t value. If F(t) = 0, then the process ends and the current t value is the desired value. Otherwise, take the current t value as t0 and repeat the above steps (1) to (3) until a t value is obtained that makes F(t) = 0.

[0086] In the traditional end-to-end design method of optical lenses, when using a ray tracing algorithm to solve the intersection of light and the optical surface, the gradient information of all calculation steps in the entire iterative process will be fully recorded. Therefore, the intermediate iterative process will introduce a lot of useless gradient information, further aggravating the consumption of video memory.

[0087] In order to further reduce the video memory overhead of the end-to-end design method of the optical lens, thereby maximizing the complexity of the designable optical lens under limited video memory resources, the method for optimizing the structural parameters of the optical lens provided in accordance with the embodiment of the present application may also include: after solving the intersection of the light ray and the optical surface of the optical lens based on the ray tracing algorithm, clearing invalid gradient information in the ray tracing algorithm process.

[0088] The process of solving the parameter t is essentially an iterative solution process, and the number of iterations l depends on the initial value t0, where l is a natural number. If the initial value t0 can be directly taken on the optical surface, the final result can be obtained through only one iteration. In other words, the gradient information generated in the iterative process is only for finding the optimal t value.

[0089] Therefore, according to the method for optimizing the structural parameters of the optical lens provided in the embodiment of the present application, after the optimal value of the parameter t is determined, the invalid gradient information can be cleared, thereby saving video memory space. The number of iterative solutions can also be reduced from 1 to 1. It can be seen that according to the method for optimizing the structural parameters of the optical lens provided in the embodiment of the present application, by optimizing the gradient storage scheme in the ray tracing algorithm, the invalid gradient F′(t) in the intermediate process is discarded, thereby significantly reducing the video memory occupied space without changing the calculation result.

[0090] The features, principles and other aspects of the present application will be described in more detail below with reference to the accompanying drawings and in combination with specific embodiments.

[0091] Embodiment 1

[0092] The first embodiment of the present application provides a method for optimizing the structural parameters of an optical lens, which uses a "step-by-step" gradient storage strategy to divide the forward propagation process into k stages (k steps). As an exemplary implementation, in this embodiment, the forward propagation process is divided into two stages, that is, k=2.

[0093] like Figure 4As shown, the structural parameter optimization method 400 of the optical lens of the first embodiment may include the following main steps:

[0094] Step S1, recording the gradient information of the evaluation index relative to the PSF.

[0095] Step S2, recording the gradient information of the PSF relative to the structural parameters, and obtaining the gradient information of the evaluation index relative to the structural parameters.

[0096] Step S3, adjusting the structural parameters of the optical lens based on a gradient reduction optimization algorithm.

[0097] Step S4, perform iteration termination judgment according to the preset target value of the evaluation index ε. If the evaluation index ε is lower than the target value, terminate the iteration and end the optimization process; otherwise, return to step S1 and repeat steps S1-S3 to iteratively calculate the evaluation index ε until the evaluation index ε is reduced to the preset target value.

[0098] Combination Figure 4 and Figure 5 As shown, the structural parameter optimization method 400 of the optical lens of the first embodiment may include the following steps:

[0099] Step S1, record the gradient ΔI of the evaluation index relative to the PSF, This step S1 corresponds to the first stage of the forward propagation process. The gradient of the above evaluation index relative to the PSF recorded in this stage is also referred to as the first gradient below. That is, in this first stage, only the first gradient is recorded. Exemplarily, step S1 may include the following sub-steps:

[0100] Step S101, importing the current structural parameters of the optical lens

[0101] Structural parameters of optical lenses It may include parameters such as the lens's radius of curvature, thickness, and aspheric coefficient.

[0102] Step S102: according to the current structural parameters of the optical lens Generate point spread functions of optical lenses in various fields of view through ray tracing algorithms Exemplarily, during the calculation process of step S102, the calculation graph is closed and no gradient information is recorded. After completing step S102, the calculation graph is opened and gradient information is recorded in subsequent sub-steps.

[0103] Step S103, generating a simulated image of the optical lens through various image processing algorithms according to each field of view PSF At the same time, save the first sub-gradient, that is, the simulated image Gradient relative to PSF

[0104] Step S104, evaluating the function f at the image level ε Get the evaluation index ε, At the same time, save the second sub-gradient, that is, the evaluation index ε relative to the simulation image Gradient

[0105] Step S105, combining the first sub-gradient and the second sub-gradient in steps S103 and S104 according to the chain rule to obtain the gradient ΔI of the evaluation index relative to the PSF, specifically in the following manner:

[0106]

[0107] The obtained gradient ΔI of the evaluation index relative to the PSF is recorded as the first gradient.

[0108] Step S106, clearing the first sub-gradient in step S103 The second sub-gradient in step S104 After recording the first gradient, the computation graph may be cleared to remove useless gradient information (such as the first sub-gradient and the second sub-gradient), thereby freeing up video memory space.

[0109] Step S2, record the gradient of PSF relative to the structural parameters And get the gradient of the evaluation index relative to the structural parameters

[0110] This step S2 corresponds to the second stage of the forward propagation process. The gradient of the above PSF with respect to the structural parameters It is also referred to as the second gradient hereinafter. That is, in this second stage, only the second gradient is recorded.

[0111] Exemplarily, step S2 may include the following sub-steps:

[0112] Step S201, importing the current structural parameters of the optical lens At the same time, the computation graph can be opened, and the subsequent sub-steps begin to record gradient information.

[0113] Step S202: according to the current structural parameters The PSF of each field of view of the optical lens is generated by the ray tracing algorithm. Different from step S102, the gradient of PSF relative to the structural parameters needs to be recorded at this time And the gradient information is recorded as the second gradient.

[0114] Step S203, combining the second gradient recorded in step S202 and the first gradient recorded in step S1 according to the chain rule to obtain target gradient information, that is, the gradient of the evaluation index relative to the structural parameter The specific method is as follows:

[0115]

[0116] The target gradient information can be used to optimize the structural parameters.

[0117] Step S204: Clear the first gradient information ΔI and the second gradient information After obtaining the target gradient information, the calculation graph can be cleared, and the first gradient ΔI recorded in step S1 and the second gradient recorded in step S202 can be cleared. This frees up video memory space.

[0118] It should be noted that, since the main purpose of step S2 is to determine and record the second gradient, the sub-steps of generating a simulation image and constructing an image-level evaluation function can be omitted. Of course, the sub-steps of generating a simulation image and constructing an image-level evaluation function can also be retained, because compared with generating a calculation graph to record complete gradient information, the amount of computation of these sub-steps is very small for the memory resources occupied during the forward propagation process.

[0119] Step S3, adjusting the structural parameters of the optical lens based on a gradient-based optimization algorithm according to the target gradient information.

[0120] As an example, in step S3, a method based on gradient descent may be used to update the structural parameters, and the parameter updating formula is as follows:

[0121]

[0122] In the formula, is the current structural parameter of the optical lens, is the optimized structural parameter, and α is the learning rate used to control the optimization step size.

[0123] In step S3, there is no need to record the gradient information.

[0124] Step S4, judging whether to terminate the iteration according to the target value of the preset evaluation index ε.

[0125] Exemplarily, step S4 may include the following sub-steps:

[0126] Step S401, importing the structural parameters updated according to step S3 As an exemplary implementation, the computation graph may be closed at this time, and no gradient information needs to be recorded in subsequent sub-steps.

[0127] Step S402: according to the updated structural parameters Generate PSF based on ray tracing algorithm.

[0128] Step S403: According to the generated PSF, a simulation image may be generated based on an image processing algorithm.

[0129] Step S404, construct an evaluation function at the image level to obtain an evaluation index ε.

[0130] Step S405, determine whether the evaluation index ε is less than the preset target value. If the evaluation index ε is less than the target value, save the current structural parameters and end the optimization process; otherwise, return to step S1 and repeat steps S1-S4 until ε is less than the target value.

[0131] Compared with the traditional end-to-end design method of optical structure, the forward propagation process will retain all gradient information in the calculation graph, thus greatly occupying the video memory space. According to the structural parameter optimization method of the optical lens provided in the embodiment of the present application, by adopting a "step-by-step" gradient storage strategy, the forward propagation process is divided into k stages, corresponding to the number of forward propagations being k times, and only the gradient information of the current stage is stored each time, so that more gradient information is saved through multiple forward propagations while the video memory space remains unchanged. In other words, the same amount of gradient information can be stored in a smaller video memory space.

[0132] Embodiment 2

[0133] The second embodiment of the present application provides a method for optimizing the structural parameters of an optical lens, which can use a "step-by-step" gradient storage strategy to divide the forward propagation process into k stages (k steps), where the value of k depends on the capacity of the GPU of the computing device and the complexity of the lens. The more gradient information, the less GPU capacity, and the greater the number of stages k required.

[0134] If it is necessary to consider more light rays and design a more complex lens structure, the forward propagation process can be divided into more stages, that is, the value of k is larger, so as to avoid occupying too much video memory space under the condition of limited video memory resources, so that the structural parameter optimization solution can run smoothly.

[0135] According to this embodiment, the forward propagation includes three stages, that is, k=3. In each stage of the forward propagation, the intermediate gradient information corresponding to the current stage is determined and recorded.

[0136] Exemplarily, the intermediate gradient information in this embodiment may include: the gradient of the evaluation function relative to the simulation image Gradient of simulated image relative to PSF and the gradient of the PSF with respect to the structural parameters At each stage of the forward propagation, one of the above gradients is recorded separately, so that the above gradient information can be combined according to the chain rule to obtain the target gradient information The specific method is as follows:

[0137]

[0138] The specific process of the three-stage forward propagation in this embodiment and the implementation of the backward propagation stage can refer to the description in the aforementioned embodiment 1, and for the sake of brevity, they will not be repeated here.

[0139] In addition, the structural parameter optimization method of the optical lens provided in this embodiment also optimizes the processing flow of solving the intersection of light rays and optical surfaces in the ray tracing algorithm, and saves video memory space by timely clearing invalid gradient information generated during the iteration process.

[0140] Combination Figure 6 As shown, in the method for optimizing the structural parameters of an optical lens provided in this embodiment, the process 600 of using a ray tracing algorithm to solve the intersection of a ray and an optical surface includes the following steps:

[0141] Step S601, initializing parameter t, and assigning an initial value t0 to parameter t.

[0142] It should be understood that different types of optical surfaces may have different parameter t initialization methods, which are not described in detail in this application.

[0143] When initializing the parameter t, the computation graph can be opened to record the gradient information.

[0144] Step S602: Substitute the parameter t into the surface equation F(t), and solve the surface equation F(t) and the gradient F'(t) of the surface equation F(t) with respect to the parameter t.

[0145] Among them, the surface equation

[0146] Step S603: Update the parameter tn as follows:

[0147]

[0148] Among them, tn is the updated parameter t.

[0149] Step S604, substitute the parameter tn into the surface equation F(t) to calculate F(tn).

[0150] Step 605, determine whether the current value of F(tn) is equal to 0. If the value of F(tn) is equal to 0, execute the next step S606; otherwise, set the value of t equal to the updated tn, and return to step S602 to repeatedly iterate and calculate the parameter t.

[0151] Step S606: If the value of F(tn) is equal to 0, the current tn is the optimal solution for the parameter t, and the current value of tn is output as the target value tout of the parameter t.

[0152] Step S607, clearing redundant gradient information in the iterative process.

[0153] After step S606, the optimal solution tout of the parameter t has been obtained. The gradient in the iterative process has lost its effect. By clearing the calculation graph, these useless and redundant gradient information can be cleared in time, thereby freeing up the video memory space. The gradient information can be re-recorded later.

[0154] Step S608, using the obtained optimal solution tout as an initial value to initialize the parameter t.

[0155] Step S609: Substitute the value of parameter t into the surface equation F(t), and solve the surface equation F(t) and the gradient F'(t) of the surface equation F(t) with respect to the parameter t.

[0156] Step S610, updating parameter tn.

[0157] The step S610 may be updated according to the updating formula in the aforementioned step S603.

[0158] Step S611, find the optimal solution of parameter t and output it as the target value tout.

[0159] After step S606, since the parameter t is the optimal solution, the intersection can be calculated after only one round of calculation. The gradient information to be saved during the iteration process is also reduced from multiple rounds of iterations to one round of iteration, thereby greatly reducing the video memory space occupied by saving gradient information during the entire iterative update process.

[0160] Compared with the traditional ray tracing algorithm, when solving the intersection of line and surface, the gradient information of all calculation steps in the entire iteration process will be fully recorded, but the gradient information generated in the iteration process is only to help find the optimal t value. According to the structural parameter optimization method of the optical lens provided by the embodiment of the present application, by optimizing the gradient storage scheme in the ray tracing algorithm, the invalid gradients in the intermediate iteration process are discarded. Once the optimal value of the parameter t is solved, the invalid gradient information in the intermediate iteration process is cleared, thereby saving video memory space.

[0161] The advantages of the structural parameter optimization method of the optical lens provided in this application in saving video memory space will be described below in conjunction with specific application examples.

[0162] Application Example 1

[0163] In this application example, Figure 7 The mobile phone lens structure shown is optimized.

[0164] Combination Figure 7 As shown in the figure, the mobile phone lens includes 7 lenses with a total of 14 optical surfaces, of which the first 12 surfaces are high-order aspheric surfaces with 15 coefficients. The parameters that can be optimized in the entire lens include curvature, thickness, aspheric coefficients, etc., totaling more than 350. At the same time, the design of the lens needs to consider at least 12 fields of view and 6 bands.

[0165] Taking the optimization design of a mobile phone lens 6P (six lenses, including more than 350 parameters) as an example, when the data type is double and the image resolution is 256*256, the video memory consumed by tracing a single ray is about 0.5MB (and the more parameters and the more complex the structure of the optical lens, the greater the video memory consumption for a single ray). PSF simulation under a single wavelength and a single field of view requires 128*128 rays, consuming a total of 8192MB=8GB of storage space, while at least 12 fields of view and 6 fields of view need to be considered in the design process of a mobile phone lens, with a total of 72 types of PSF, consuming a total of 576GB of storage space. In order to improve computing efficiency, AD algorithms usually run on GPUs. Even the current high-end GPUs with 30GB of video memory space are obviously unable to meet the huge video memory requirements for storing computational graphs.

[0166] Table 1 below shows the relationship between the GPU memory space occupied and the number of rays.

[0167] Number of rays (10,000) Traditional solution video memory consumption (GB) Video memory consumption in this example (GB) 0.5 2.64 0.73 1 5.27 1.46 1.5 7.91 2.20 2 — 2.93 2.5 — 3.66 3 — 4.39 3.5 — 5.13 4 — 5.86 4.5 — 6.59

[0168] Table 1

[0169] Combining Table 1 and Figure 8 As shown, in the traditional end-to-end design method of optical lenses, since all gradient information needs to be retained in a forward propagation process, and the ray tracing algorithm is not optimized for video memory, the GPU video memory required for tracing a single ray is about 0.54MB, and the maximum video memory of the GPU set (8GB) is reached by tracing only 15,000 rays. According to the end-to-end design method of the present application, the target gradient information is decomposed into multiple intermediate gradient information for calculation through the number of steps k in the "step-by-step" gradient storage strategy. At the same time, the invalid intermediate gradients in the iterative process of the ray tracing algorithm are cleared in time. Taking k=2 as an example, the video memory required for a single ray can be reduced to 0.15MB, which is only 27.8% of the original solution. It is worth mentioning that if the value of the number of steps k in the "step-by-step" gradient storage strategy is increased, the video memory consumption can be further reduced.

[0170] In this application example, the image restoration network used in the image processing algorithm can be Fig. 9The ResUnet structure shown includes 3 down sampling layers (Down Sampling, referred to as DS) and 3 up sampling layers (Up Sampling, referred to as US), with a total of 4.6*10^6 parameters.

[0171] In this application example, the joint design of mobile phone lens and image post-processing algorithm runs on NVIDIA TeslaV100 PCle GPU, which has a single card memory capacity of 32GB.

[0172] If the traditional joint design scheme is adopted, the GPU memory consumption of a single ray is 0.54MB in the case of single wavelength and single field of view. In order to ensure the reliability of the scheme, at least 64*64=4096 rays need to be traced. Then the amount of video memory consumed by the traditional joint design scheme is 4096*0.54MB*12*6=155.52GB, where 12 is the number of fields of view and 6 is the number of bands, which obviously greatly exceeds the GPU memory limit.

[0173] According to the structural parameter optimization method of the optical lens provided in the embodiment of the present application, by optimizing the gradient storage scheme in the ray tracing algorithm and discarding the invalid gradients in the intermediate process, the video memory consumption of a single ray can be reduced to 0.15MB. At this time, if all gradient information is retained during a forward propagation process, 43.2GB of video memory space will be consumed, which still cannot meet the 32GB video memory limit. If the "step-by-step gradient storage strategy" of the present application is further combined and the entire forward propagation process is split into 2 steps, the amount of gradient information stored in a single step is roughly reduced to half of the original amount, that is, 21.6GB, which can meet the video memory limit of the GPU and enable the joint optimization solution to run smoothly with existing computing resources.

[0174] In addition, according to an embodiment of the present application, an electronic device 900 is also provided, such as Fig.10 As shown, the electronic device 900 may include: at least one processor and a memory, the memory is communicatively connected to the at least one processor and stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned optical lens structural parameter optimization methods 100 and 400, etc.

[0175] One embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned optical lens structural parameter optimization methods 100 and 400 are implemented.

[0176] Fig.10 is a schematic block diagram of an electronic device 900 according to some embodiments of the present application. Fig.10As shown, the electronic device 900 may be a device for implementing the various methods / steps described above. Fig.10 As shown, the electronic device 900 includes a processor 901, which can perform various appropriate steps and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The processor 901, ROM 902 and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0177] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a button, a touch screen, etc.; an output unit 907 to output various forms of signals; a memory 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The processor 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 901 may execute the method 100 and each processing flow described above. For example, in some embodiments, the above method and each processing flow may be implemented as a computer software program, which is stored in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or a communication unit 909. When the computer program is loaded into RAM 903 and executed by the processor 901, one or more steps in each method described above may be executed. Optionally, in other embodiments, the processor 901 may be configured to execute one or more steps in the above method by any other appropriate means (e.g., by means of firmware).

[0179] It should be further noted that the present application may include methods, devices, systems and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present application.

[0180] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used in this application is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0181] The computer-readable program instructions described in this application can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0182] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as C language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet). In certain embodiments, by using the state information of computer-readable program instructions to customize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0183] Various aspects of the present application are described in this application with reference to flowcharts of methods, devices (systems), and computer program products according to exemplary embodiments of the present application. It should be understood that each step of the flowchart and the combination of the steps in the flowchart can be implemented by computer-readable program instructions.

[0184] These computer-readable program instructions can be provided to a processor in a device, apparatus (system), a general-purpose computer, a special-purpose computer, or a processing unit of other programmable data processing devices, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / steps specified in one or more steps in the flowchart is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / steps specified in one or more steps in the flowchart.

[0185] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / steps specified in one or more steps in the flowchart.

[0186] The flowchart in the accompanying drawings shows the possible architecture, functions and operations of the equipment, methods and computer program products according to multiple embodiments of the present application. In this regard, each step in the flowchart can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions marked in the steps can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each step in the flowchart, and the combination of the steps in the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for optimizing structural parameters of an optical lens, comprising a plurality of forward propagation stages for determining evaluation indicators of structural parameters of the optical lens, and a backward propagation stage for optimizing the structural parameters, characterized in that: In each forward propagation stage, determining and recording only the intermediate gradient information corresponding to the current stage, and merging the intermediate gradient information into the target gradient information; In the back propagation stage, the structural parameters are adjusted and optimized according to the target gradient information, The target gradient information includes the gradient of the evaluation index relative to the structural parameter.

2. The method according to claim 1, wherein: The forward propagation at each stage includes: According to the current structural parameters of the optical lens, the propagation process of light in the optical lens is calculated based on a ray tracing algorithm to generate the point spread function of the optical lens in each field of view; generating a simulated image formed by the optical lens according to the point spread function; and Constructing an evaluation function at the image level of the simulation image to obtain an evaluation index of the current structural parameter; The back propagation phase includes: The current structural parameters are adjusted and optimized based on a gradient-based optimization algorithm, and the evaluation index is iteratively calculated by repeating the forward propagation stage until the evaluation index is reduced to a preset target value, and the current structural parameters are used as the optimized target parameters.

3. The method according to claim 2, wherein: The intermediate gradient information includes a first gradient and a second gradient; the multiple forward propagation stages include a first stage for determining and recording only the first gradient, and a second stage for determining and recording only the second gradient; The first gradient includes the gradient of the evaluation index relative to the point spread function; the second gradient includes the gradient of the point spread function relative to the evaluation index.

4. The method according to claim 2, wherein: The intermediate gradient information includes a gradient of the evaluation index relative to the simulation image, a gradient of the simulation image relative to the point spread function, and a gradient of the point spread function relative to the structural parameter.

5. The method according to claim 3, wherein: Determining and recording only the first gradient in the first stage comprises: While generating the simulated image, saving a first sub-gradient of the simulated image relative to the point spread function; While constructing the image-level evaluation function to obtain the evaluation index, saving a second sub-gradient of the evaluation index relative to the simulation image; and The first sub-gradient and the second sub-gradient are combined according to the chain rule to obtain a gradient of the evaluation index relative to the point spread function, and the gradient is recorded as the first gradient.

6. The method according to claim 5, wherein: The method further comprises: After recording the first gradient, clearing the saved first sub-gradient and the second sub-gradient.

7. The method according to claim 5, wherein: Determining and recording the second gradient in the second stage includes: While generating the point spread function, the gradient of the point spread function relative to the evaluation index is saved and recorded as the second gradient.

8. The method according to claim 7, wherein: The method further comprises: After obtaining the target gradient information, the recorded first gradient and second gradient are cleared.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having machine-executable instructions stored thereon, wherein when the machine-executable instructions are executed, the machine is caused to perform the method according to any one of claims 1 to 8.

Images