Minimal memory differentiation
Patent Information
- Application Number
- PCT/US2023/022126
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-05-13
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-14
AI Technical Summary
Grid-based simulators, such as Finite Difference Time Domain (FDTD) electromagnetic simulators, face high memory consumption when employed in reverse-mode automatic differentiation due to the need to store numerous grid positions and mathematical operations, limiting their deployment to small problems or large computer clusters.
The implementation of minimal memory reverse mode automatic differentiation, which leverages the time-reversible nature of Maxwell’s equations to reduce memory usage by storing only the final timestep's field values and using a time-reversed simulation to provide intermediate values during backpropagation, thereby eliminating redundant memory operations.
This approach significantly reduces memory requirements, achieving a 98% reduction in memory consumption while maintaining runtime performance, enabling the design of complex devices like bichromatic color sorters and expanding the applicability of inverse design topology optimization across fields.
Smart Images

Figure US2023022126_14112024_PF_FP_ABST
Abstract
Description
Atty. Dkt. No.: 098930-0387 HU 9011 MINIMAL MEMORY DIFFERENTIATION CROSS-REFERENCE TO RELATED PATENT APPLICATIONS
[0001] This application claims the benefit and priority of U.S. Provisional Patent Application No.63 / 341,959, filed on May 13, 2022, the entirety of which is incorporated by reference herein. GOVERNMENT LICENSE RIGHTS
[0002] This invention was made with government support under FA9550-19-1-0135 and FA9550-21-1-0312 awarded by U.S. Air Force Office of Scientific Research (AFOSR). The Government has certain rights in the invention. BACKGROUND
[0003] Nanostructured surfaces can be used to sense and manipulate the numerous degrees of freedom of light. Metalenses, for instance, can control the transmitted wavefront through an intricately structured surface to achieve diffraction-limited and aberration- corrected focusing. Lithography can enable the creation of features far smaller than the wavelength of visible light, resulting in a vast space of possible nanostructures. Identifying high-performing devices that can be fabricated within equipment constraints from this massive set of possible structures can be useful for various applications. Two main approaches to address this challenge can include forward design and inverse design. SUMMARY
[0004] The systems and methods of the present disclosure relate to a minimal memory differentiation (e.g., minimal memory automatic differentiation, minimal memory reverse mode automatic differentiation). Automatically differentiable models can enable efficient computation of parameter gradients, greatly expediting optimization and the solution of high-dimensional systems. However, when employed on a reverse-mode differentiable platform, grid-based simulators can exhibit high memory consumption due to the many grid positions and mathematical operations that need to be stored. The systems and method of the present disclosure can reduce the memory consumption of a differentiable Finite Difference Time Domain (FDTD) electromagnetic simulator by leveraging the time-reversible nature of Maxwell’s equations and eliminating redundant memory operations performed at each timestep. This architecture can be demonstrated in a 7 ൈ 5 ൈ 5 ^^ 3D FDTD, in which the memory requirements for calculating a 900-element gradient vector are reduced by 98% 1 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 (from 196 MB / timestep to 3.5 MB / timestep) without compromising runtime performance. The differentiable simulator can be applied to design a bichromatic color sorting device. This approach to automatically differentiate grid-based simulators can apply to a broad range of physics simulators, thereby enhancing the applicability of inverse design topology optimization across fields.
[0005] At least one aspect of the present disclosure is directed to a method. The method includes iterative grid-based modeling of a physical structure of a device to output a grid-based numerical model. The method includes determining a numerical relationship between one or more objective values of the device and one or more properties of the device. The method includes calculating a gradient of the one or more objective values with respect to one or more parameters corresponding to the one or more properties of the device by minimal memory reverse mode automatic differentiation. The minimal memory reverse mode automatic differentiation includes calculating field values on a grid by iteratively updating the field values for one or more steps. The minimal memory reverse mode automatic differentiation includes performing reverse iteration of the grid-based numerical model to output field values on the grid at earlier steps of the one or more steps. The minimal memory reverse mode automatic differentiation includes performing backpropagation based on the field values on the grid to compute intermediate gradients of the one or more objective values with respect to the one or more parameters.
[0006] Another aspect of the present disclosure is directed to a method. The method includes calculating one or more final field values, each of the one or more final field values corresponding to one or more coordinates. The method includes calculating a first gradient of one or more objective values based on the one or more final field values. The method includes calculating one or more first intermediate field values based on the one or more final field values, each of the one or more first intermediate field values corresponding to the one or more coordinates. The method includes calculating a first intermediate gradient of the one or more objective values based on the first gradient and the one or more first intermediate field values. The method includes calculating one or more second intermediate field values based on the one or more first intermediate field values, each of the one or more second intermediate field values corresponding to the one or more coordinates. The method includes calculating a second intermediate gradient of the one or more objective values based on the first intermediate gradient and the one or more second intermediate field values. The method includes calculating one or more initial field values based on the one or more second 2 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 intermediate field values, each of the one or more initial field values corresponding to the one or more coordinates. The method includes calculating a final gradient of the one or more objective values based on the second intermediate gradient and the one or more initial field values. The method includes calculating a gradient of an objective function based on the final gradient.
[0007] Those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices and / or processes described herein, as defined solely by the claims, will become apparent in the detailed description set forth herein and taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
[0009] FIG.1 illustrates reverse mode automatic differentiation.
[0010] FIG.2 illustrates minimal memory automatic differentiation, according to an embodiment.
[0011] FIG.3 illustrates a 3D FDTD simulation geometry, according to an embodiment.
[0012] FIG.4 illustrates validation of the computed gradients from minimal memory automatic differentiation (MMAD), according to an embodiment.
[0013] FIG.5 illustrates a comparison of runtime for MMAD and reverse mode automatic differentiation (RM AD), according to an embodiment.
[0014] FIG.6 illustrates a comparison of memory consumption for MMAD and RM AD, according to an embodiment.
[0015] FIG.7 illustrates a schematic of color sorting function, according to an embodiment.
[0016] FIG.8 illustrates the optimized permittivity distribution over the 30 x 60 pillar array, according to an embodiment. 3 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0017] FIG.9 illustrates the intensity distribution |Ex|2+|Ey|2+|Ez|2, according to an embodiment.
[0018] FIG.10 illustrates validation of the accuracy of MMAD FDTD field calculations against a FDTD suite, according to an embodiment.
[0019] FIG.11 illustrates the location of the target gaussian intensity profiles, according to an embodiment.
[0020] FIG.12 illustrates the variation of objective function values with iteration number, according to an embodiment.
[0021] FIG.13 illustrates the permittivity profiles of the pillar array at different iteration numbers, according to an embodiment.
[0022] FIG.14 illustrates a method for minimal memory differentiation, according to an embodiment.
[0023] FIG.15 illustrates a method for minimal memory differentiation, according to an embodiment.
[0024] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0025] Following below are more detailed descriptions of various concepts related to, and implementations of methods and apparatuses for minimal memory differentiation. The various concepts introduced above and discussed in greater detail below may be implemented in any of a number of ways, as the described concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes.
[0026] Forward design can involve constructing a large device from simpler components that have well-defined behavior. In flat optics, these components can include discrete subwavelength nanostructures called meta-atoms. Each meta-atom in a “library” (e.g., a collection) can include elementary shapes and can have its optical behavior (e.g., transmitted phase delay) individually simulated with electromagnetic simulators, such as Finite Difference Time Domain (FDTD), finite element, frequency-domain, or modal solvers. These meta-atoms can be tiled based on predefined spatial distributions, like a focusing hyperbolic phase profile. Forward design can systematically design large nanophotonic 4 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 devices for applications with requirements that can be met by a library of meta-atoms with a limited number of degrees of freedom.
[0027] However, inverse-design can be used for applications without a well-defined desired spatial parameter distribution or those demanding wider-ranging behaviors. One example application is designing different behaviors over a band of wavelengths, which can require control over the chromatic dispersion of the nanostructures and a much larger structure parameter space. Inverse design techniques can treat this as an optimization problem over a high-dimensional parameter space. A dielectric distribution can be approximated by pixelating a surface and assigning each pixel a binary fill factor, indicating the presence or absence of material. The behavior of that dielectric distribution can then be simplified to an objective function that quantifies the optical performance of a list of pixel fill factors. Inverse-designing such a device can be equivalent to finding the appropriate parameter vector that maximizes the objective value.
[0028] Most high dimensional optimization problem can employ gradient descent to discover high performing solutions. The most computationally intensive step can include calculating the objective function gradient. Finite difference approximations can, in principle, achieve this, but their computational complexity can scale poorly with respect to the number of degrees of freedom (e.g., at least 1 objective function call per degree of freedom).
[0029] The adjoint method is a technique that can be used to obtain gradients with the computational cost of just two objective function calls (e.g., two electromagnetic simulations), regardless of the number of degrees of freedom considered. Based on the field values obtained from these two simulations, Lorentz reciprocity can be exploited to compute the exact numerical derivative ^^ ^^ / ^^ ^^ of the objective function ^^ with respect to each pixel’s dielectric constant ε. Despite its power, it can be challenging to generalize the adjoint method, as it requires deep a priori knowledge of system symmetries to determine the source and boundary conditions of the adjoint simulation and the relationship between the objective function and adjoint field values.
[0030] Automatic differentiation (AD) can be a powerful and efficient technique for gradient calculation. In its most common form, reverse mode (RM), AD can maintain a record of all mathematical operations undertaken during a calculation and can compute the exact numerical gradients using the chain rule. Automatic differentiation can optimize arbitrary objective functions quantifying any continuous simulation result and can avoid the 5 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 use of mathematically complex reciprocity relations, particularly for complex systems involving non-reciprocal gyrotropic and nonlinear materials. However, applying AD to iterative grid-based simulation methods (e.g., FDTD) can require saving all intermediate values at every iteration step and pixel, along with every mathematical operation involved. Combined with the large number of space-time pixels in a typical FDTD calculation, this can lead to unmanageable memory requirements that limit the deployment of differentiable FDTDs to small problems or large computer clusters.
[0031] The systems and methods of the present disclosure can address this challenge by using minimal memory differentiation. For example, the systems and methods of the present disclosure can construct a differentiable 3D FDTD simulator that unifies minimal memory complexity with efficient computational complexity. The structure of the FDTD update equations can be exploited in two ways: first, since FDTD can reuse the same update equations for all timesteps, only one copy of the mathematical operations and parameter dependencies needs to be stored. Second, instead of storing intermediate parameter (e.g., field) values at each timestep, backpropagation can be performed by running the FDTD simulation backwards, starting with the final system state and time-stepping in reverse towards the initial conditions. Since the backpropagation chain rule calculation can proceed in this same reverse direction, the intermediate parameter values can be calculated precisely when they are needed at each timestep. This can be shown by applying the minimal memory FDTD architecture to design a simple photonic device that sorts incident illumination based on wavelength. The differentiable FDTD can lower the entry bar for gradient-based topological optimization and can expand the range of optimizable systems. These results can be generalizable to many grid-based simulators.
[0032] FDTD can include a grid-based, time-stepping method of solving partial differential equations. When applied to Maxwell’s equations and electromagnetism, FDTD can be used to simulate light-matter interactions in the time-domain. FDTD can be a highly generalizable simulation technique. FDTD can be well-suited for systems with multiple frequencies, time-varying properties, and non-linear materials. FDTD can be naturally integrated into multiphysics simulations. The electric and magnetic fields can be updated iteratively over a discretized simulation volume, for example, until the excitation fields are absorbed or escape the simulation volume. The space and time derivatives in Maxwell’s equations can be estimated using centered finite difference approximations. 6 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0033] FDTD can require a large number of spatial pixels and iteration timesteps. To minimize the effect of numerical dispersion, the spatial pixel size can be chosen to be less than λ / 10, which can be significantly smaller than the wavelength λ of interest. This can directly constrain the maximum timestep through the Courant stability condition, which can require that the ratio between the timestep ^^Δ ^^ and smallest spatial step Δ ^^ be smaller than^√ଷfor the 3D case with cubic pixels. The simulation then can be run for a sufficient length of time for the excitation fields to traverse the simulation volume or decay, which can take many calculation steps in the presence of resonances.
[0034] Although automatic differentiation can have two modes of operation (e.g., forward mode and reverse mode), it can be largely synonymous with the latter due to its scaling performance with respect to the number of input (Ninput) and output (Noutput) parameters. The forward mode computational cost can be independent of Noutput but can scale linearly with Ninput. Conversely, reverse mode automatic differentiation (RM AD) can scale linearly with Noutputand can be independent of Ninput. Most optimization problems can have a large Ninputdue the number of tunable parameters or fitting parameters, and a very small Noutput≈1 for the objective function. Therefore, the focus can be on reverse mode automatic differentiation.
[0035] Reverse mode automatic differentiation can rely on the computation tree, also known as a tape or Wengert list. The computation tree can contain all mathematical operations involved from the inputs to the outputs. Each node in the tree can represent an intermediate elementary operation with a well-defined derivative associated with its own inputs and output. There can be two passes in RM AD. The forward pass can traverse the computational tree from the inputs to the outputs, storing all the intermediate values obtained. The backward pass, also known as backpropagation, can perform the chain rule for differentiation from the outputs back towards the inputs, drawing upon the stored intermediate values. Backpropagation can be the foundation of modern machine learning, as it can provide the objective function gradients with respect to many tunable parameters (e.g., weights and biases in neural networks) for iterative model training and refinement.
[0036] These forward and backward passes can be envisioned through an example objective function ^^ including N scalar functions, ^^^to ^^ே, which can operate sequentially on an input x: 7 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 ^^^ ^^^ ൌ ^ ^^ே∘ ^^ேି^∘ ⋯∘ ^^^^^ ^^^ ൌ ^^ே^ ^^ேି^^⋯ ^^^^ ^^^⋯ ^^ (1)
[0037] This computation tree can be a single long line of operations from x through ^^^, ^^ଶ, … to ^^. Each operation can be a continuous elementary function of the previous variable with a stored functional derivative form. For example, ^^^can be a function of ^^^ି^such that the functional derivative ^^ ^^^ / ^^ ^^^ି^is well-defined in terms of ^^^ି^, and this information can be stored in the respective tree nodes. While the objective function and intermediate functions can be shown as single-valued for simplicity, the tree can be generalizable to multi-valued functions and multi-valued inputs. During backpropagation, the adjoint, ^^ ^^ / ^^ ^^, can be computed where ^^ ∈^^^, ^^ே, ^^ேି^, … , ^^^, ^^^represents every node in the tree. The adjoint can be defined as the sensitivity of the final objective function to a smallperturbation in the value of a node. The final adjoint value to be calculated is ^^ ^^ / ^^ ^^ ൌ^^ ^^ / ^^ ^^, the gradient of the objective function for descent optimization.
[0038] By definition, the first adjoint term can be ^^ ^^ / ^^ ^^ ൌ 1. If the adjoints are computed in reverse order from ^^ towards x, the adjoint value can be re-used (e.g., backpropagated) from the previous step to compute the subsequent adjoint value: ^^ ^^^^ ^^ ^^ ^^ ^^ ^^ேൌ^^ ^^^^ ^^ேฬிಿ
[0039] At every calculation step, the backpropagating adjoint term can be multiplied into the derivative ^^ ^^^ / ^^ ^^^ି^evaluated at ^^^ି^. The stored node values ^ ^^^, ^^^^ୀ^,...,ேduring the forward pass thus can provide the necessary information to calculate the derivatives during backpropagation.
[0040] This simple example can demonstrate how the computation complexity of RM AD is independent of Ninput. Suppose that ^^ ∈ ℝே^^^ೠ^is an Ninput-dimensional vector. There are far fewer input parameters than computation steps ^^^^^௨௧≪ ^^, and ^^^:ℝே^^^ೠ^↦ ℝ. All other operations ^^ଶ, … , ^^ே, ^^ can be kept the same so that the backpropagation algorithm 8 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 yields ^^ ^^ / ^^ ^^^with the same number of steps as described previously. Then the multi-valued gradient ^^^^^^ can be obtained from ^^ ^^ / ^^ ^^^and the functional form of ^^^through Ninputadditional elementary steps:
[0041] This gradient calculation thus may not scale with Ninput since the number of mathematical operations ^^ ≫ ^^^^^௨௧can be large and can dominate the computational cost. Instead of running the full computation tree once for each of the Ninput degrees of freedom, and performing N operations each time, as a single-sided finite difference technique may require, backpropagation can eliminate redundant calculations and can attain favorable scaling with respect to Ninput. Reverse mode automatic differentiation can provide the multivariate gradient in no more than five times the number of operations in the objective function (e.g., within five times the runtime of the forward pass), independent of the degrees of freedom. This scaling performance can be comparable to the adjoint method, which can also yield the multivariate gradient in a constant multiple of simulation runtimes (e.g., two runs).
[0042] Both gradient calculation methods can be applied to FDTD simulation of a structure (parametrized by vector ^^) in a simulation volume including NVpixels and NTtimesteps. In both cases, the desired output can be the objective function gradient with respect to the structure vector ^^ ^^ / ^^ ^^.
[0043] FIG.1 illustrates reverse mode automatic differentiation. The computation tree corresponding to an optimization calculation with an embedded FDTD calculation can be considered. The calculation can proceed from a structure parameter vector ^^, which can be the vector of pixel fill factors, to the electromagnetic fields over the NVspatial pixels at zero time F(0), through NTtimestep updates to the electromagnetic fields F(T) at the end time T, and finally to the objective function value f. The FDTD tree can be simultaneously wide (due to the number of spatial pixels) and deep (due to the number of iteration timesteps). During the forward pass, every mathematical operation and intermediate field value can be stored at considerable memory cost. This memory consumption can peak just after the objective value is computed. During backpropagation, the AD platform can compute the derivative of the objective function f with respect to every intermediate parameter. This process can proceed in reverse order from the objective value back towards the structure vector at the root of the tree 9 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 and can draw upon the stored operations and intermediate field values in reverse order. Reverse mode AD can have two steps: the forward pass and backpropagation. During the forward pass, all the field values F need to be stored (double box outline) for all space-time positions so that they can be re-used during the backpropagation process, leading to substantial memory consumption.
[0044] This layer-by-layer depiction of FDTD can be a simplification, but it can be valid even for objective functions that do not just depend on the fields at the final timestep F(T). In particular, to extract spectrally-resolved properties from FDTD, partial sums of the discrete Fourier transform can be explicitly accumulated during the time stepping updates. Thus, F can capture not only the electromagnetic fields over the simulation volume, but also partial Fourier transform sums, and it can be valid to write the FDTD as a cascaded set of layers (e.g., each layer being associated with one timestep) with the same update equations between each layer.
[0045] Table 1 displays the time and memory scaling behavior for various methods of gradient calculation applied to an FDTD system with Ninput tunable parameters, Noutput objective functions, NVpixels, NTtimesteps, and Nffrequency points. Scaling behaviors that are relatively favorable for high-dimensional inverse design (large Ninput, NV, NT, small Noutput) are labeled with asterisks (*). The simplest gradient calculation, finite differences, can exhibit poor time complexity as it can scale linearly with Ninput, but it can use very little memory as it only needs to store the electromagnetic field values at the current timestep. Forward mode AD can have favorable memory scaling that is independent of the number of timesteps. However, AD can also suffer from poor Ninputtime complexity. Reverse mode AD can have favorable time complexity independent of Ninput, but it can have unfavorable memory complexity due to the large number of space-time pixels ^^^^^்for which the field values must be stored during the forward pass. For example, a small ( ^^^ൌ 252 ൈ 180 ൈ 180) 3D FDTD simulation based on the FDTD framework running on RM AD can consume 196 MB of memory per timestep when using the open-source AD tool Autograd. The adjoint method can yield favorable time and memory complexity but can require significant mathematical manipulation for implementation.
[0046] 10 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011Table 1: Time and memory scaling complexities for gradient calculation modes in FDTD simulations. Asterisks (*) indicate favorable scaling for high-dimensional inverse design, relative to the other gradient calculation modes. Ninputand Noutputrepresent the number of input and output parameters, respectively. NV is the number of spatial pixels, NT is the number of timesteps, and Nfis the number of frequency points. ∂NVis the number of pixels on the recording layer at the spatial boundary of the simulation.
[0047] FIG.2 illustrates minimal memory automatic differentiation. The structure of the FDTD computational tree can motivate two key changes that substantially reduce the memory cost of a RM AD gradient calculation, forming the core of the minimal memory AD (MMAD) platform. First, the mathematical operations that update the field ^^^ ^^^, 1 ^ ^^ ^ ^^்from one timestep to the next can be the same at every timestep. Similarly, the backpropagation operations which propagate the parameter gradients from ^^ ^^ / ^^ ^^^ ^^^ to ^^ ^^ / ^^ ^^^^^ െ 1 / 2^using the chain rule can be time-independent. Thus, instead of storing an individual copy of the update equations (and backpropagation operations) for every timestep, it can suffice to store them once. This can lead to a factor of NTreduction in memory usage for storing the tree structure.
[0048] Second, it is observed that Maxwell’s equations are time-reversible and backpropagation during gradient calculation can proceed backwards from the objective function. The time-reversibility of Maxwell’s equations (not to be confused with time inversion symmetry ^^ → െ ^^) can arise from the conservation of information under time evolution. This can imply that the FDTD update equations can be rearranged to timestep fields in reverse, computing ^^^ ^^ െfrom ^^^ ^^^. Although FDTD on a finite domain may not conserve information due to the presence of lossy boundary conditions like Perfectly Matched Layers (PMLs), local information loss at boundaries can be prevented. 11 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0049] In MMAD, instead of storing all electromagnetic fields during the forward pass, only the fields at the final timestep ^^^ ^^^ need to be stored. This can reduce the peak memory required during the forward pass to that of a regular FDTD simulation. During backpropagation, a time-reversed simulation of the same FDTD domain (using ^^^ ^^^ as the initial conditions) can be run in parallel with the chain rule gradient calculations, beginning at the final timestep t = T and proceeding backwards in time. This time-reversed simulation can supply the required field values to the chain rule calculations at every timestep. Specifically, after a time-reversed simulation from ^^^ ^^^ to ^^^ ^^ െ 1 / 2^, ^^^ ^^ െ 1 / 2^ can be used to calculate can be found using the chain ruleడ^ డ ^^^௧^This method of generating the intermediate fields on-demand can account for the bulk of the memory saving in MMAD.
[0050] The calculation of the objective function derivative at the last timestep, ^^ ^^ / ^^ ^^^^^^, which is independent of the FDTD update equations, can be performed using RM AD if the objective function f is a complex functional of the field F(T).
[0051] In an infinitely large or periodically continued system, the information content can be conserved under the evolution dictated by Maxwell’s equations. It is possible to start at the final system configuration and perform reverse time-stepping to perfectly reattain the initial conditions. However, nanophotonic simulations are finite and can use idealizations to capture the essential behavior without unnecessary computational cost. After the light-matter interaction of interest, the source fields can be absorbed by infinitely lossy boundary conditions such as PMLs, which can serve as an approximation of fields exiting to free space. At first glance, such an “empty” final configuration cannot be time-reversed, as the field information incident at these infinitely lossy boundaries can be lost. To address this issue, a single-pixel-thick recording layer at the surface of these infinitely lossy boundaries can be introduced to capture the escaping time-dependent field. The system’s three-dimensional spatial initial state can be encoded in the two-dimensional simulation boundary and one time dimension. During the time-reversal simulation, these recorded fields can be replayed as field sources from the recording layers. For minimal memory AD, only the fields at one timestep need to be stored. During backpropagation, a reverse simulation can be run simultaneously to provide the necessary field values in the order required in backpropagation. The reverse iteration of the grid-based numerical model can occur in parallel with the backpropagation. 12 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0052] Table 1 exhibits the time and memory complexity of MMAD, demonstrating that it can have the same time complexity as RM AD and the adjoint method, but substantially less memory consumption than RM AD. In the limit of large NT, the peak memory usage of MMAD calculations can scale linearly with the number of simulation time steps. This memory consumption can arise from the storage operation of the recording boundary. Since the number of field positions associated with the recording boundary ∂NVcan scale with the surface area of the simulation volume, the additional memory consumption per timestep can scale with the square of the simulation box side length, instead of the cube as in RM AD. The memory saving thus can become more pronounced for larger simulation volumes.
[0053] The recording boundary can enclose a smaller subset of the simulation volume than the lossy boundaries, allowing the reverse time-stepping simulation to be performed over a smaller volume and thus consuming even less memory and computation time. Although this smaller volume does not need to enclose the FDTD field sources (since the information is already encoded on the recording boundary), the recording boundary may enclose the structure vector ^^ locations to preserve the updating of the gradient ^^ ^^ / ^^ ^^ there.
[0054] The MMAD approach can be validated by constructing a differentiable 3D FDTD and evaluating its gradient calculation accuracy, runtime, and memory scaling. Without loss of generality, a 3D simulation region of 252×180×180 pixels and spatial pixel size ∆ ^^ ൌ ∆ ^^ ൌ ∆ ^^ ൌ 13.75 nm can be chosen. FIG.3 illustrates a 3D FDTD simulationgeometry. The size is 6.5 ^^ ൈ 4.7 ^^ ൈ 4.7 ^^, ^^ ൌ 532 nm. The structure under testing can be a30 ൈ 30 ൌ 900 square pillar array (660 nm tall) on the yz plane. Each of the 900 pillars canhave an individual real-valued relative permittivity (normalized to the permittivity of free space ^^^) and the 30 × 30-element matrix of the relative permittivities ^^ can serve as the structure vector of tunable parameters. The pillars can be positioned on a glass substrate (n=1.44) in the yz plane and are surrounded by air (n=1). The source field can be a sinusoidal plane wave located within the glass substrate (vacuum wavelength ^^^ൌ 532 nm, 10 cycles, z-polarized, x-propagating). The objective function can be the phase ^^ of the complex electric field averaged over the monitor plane located 770 nm above the 900 pillars, which can correspond to the zeroth-order transmitted phase of the nanostructures. The objective function gradient can be, 1 ^ ^^ ^ 30, 1 ^ ^^ ^ 30. The source plane, nanostructures, andmonitor plane can be surrounded by Total Field Scattered Field boundaries, which can be 13 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 surrounded by PMLs on all sides. The recording boundary can be coincident with the PML boundaries. The accuracy of the FDTD simulation can be validated against an FDTD suite (e.g., Lumerical FDTD, Ansys Inc.).
[0055] The 252 ൈ 180 ൈ 180 pixel domain (6.5 ^^ ൈ 4.7 ^^ ൈ 4.7 ^^) can be partitioned into a glass volume and air volume. A single mode planar sinusoidal field at λ = 532 nm can be generated from the glass medium and can be incident on a pillar volume. The zeroth order phase ^^ of the transmitted field can be recorded. The simulation volume can be surrounded by a Total Field Scattered Field (TFSF) boundary and perfectly matched layers.
[0056] FIG.4 illustrates validation of the computed gradients from MMAD. The gradients obtained through the MMAD approach can be validated using 900 single-sided finite difference calculations for a structure with random permittivities and can find that the root-mean-squared absolute difference of the objective function gradient is negligible, on the order of 10−10rad. FIG.4 can show validation of the computed gradientsfrom MMAD against that of finite difference calculations, for different pillar positions on the yz plane. The marginal runtime dependencies on FDTD timesteps can be obtained by linear regression.
[0057] FIG.5 illustrates a comparison of runtime for MMAD and RM AD. Although MMAD can perform an additional time-reversal simulation during backpropagation compared to RM AD, runtime comparisons on a single CPU core using the same platform (Python) as a function of the number of FDTD timesteps shows similar scaling, 19 s / timestep and 13 s / timestep for MMAD and RM AD, respectively. Reverse mode AD and MMAD may not have significant runtime differences when run on the same single-core Python platform. A speedup can be obtained when MMAD is run on a compiled platform (C++) and also on a single core.
[0058] FIG.6 illustrates a comparison of memory consumption for MMAD and RM AD. In terms of peak memory consumption, MMAD can achieve a 98% memory usage reduction from 196 MB / timestep to 3.5 MB / timestep when compared to RM AD. MMAD can yield a 98% reduction in peak memory consumption per timestep compared to reverse mode AD, when run on the same single core Python platform. The marginal memory dependencies on FDTD timesteps can be obtained by linear regression.
[0059] Further gradient calculation speed-up can be demonstrated with similar peak memory consumption by implementing MMAD on a compiled language (C++) instead of an interpreted language (Python), even for single CPU core computations (1.4 s / timestep and 3.6 14 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 MB / timestep). This two order-of-magnitude improvement in memory consumption in moving from RM AD to MMAD can be consistent with the number of pixels along one spatial dimension of the 3D simulation volume.
[0060] With the performance and accuracy of the differentiable FDTD on MMAD established, the model can be deployed to inverse-design multifunctional nanophotonic devices. As a proof-of-concept, a passive structure that sorts incident illumination at two different wavelengthsൌ 488 nm, ^^ଶൌ 633 nm into two distinct spatial positions can be designed. The desired behavior of the device is illustrated in FIG.7: x-directed, z-polarized illumination can be incident on a glass substrate on which a compact 30 x 60 pixel array of pillars lies in the yz plane. The tunable parameters can include the dielectric permittivities of the pixelated pillar array. The objective function to be maximized for each wavelength can be the overlap integral of the transmitted electric field profile with a target Gaussian field profile. The target field profiles can be chosen to ensure that 488 nm and 633 nm light are deflected and focused in opposite z-directions, allowing the device to act as a simple color sorter for these two wavelengths. The total system objective function to be maximized can be the sum of these two individual wavelength objective functions. The objective function gradient can be computed with respect to the dielectric permittivities in the pixelated pillar array using the MMAD FDTD simulator. Gradient descent optimization can be performed at a fixed learning rate. FIG.7 illustrates a schematic of color sorting function in which incident illumination including a mixture of 488 nm and 633 nm wavelengths is spatially split into two distinct regions by a compact 30 x 60 array of pillars. The permittivity distribution over the pillar array can be determined by inverse-design using the 3D differentiable FDTD.
[0061] The optimized dielectric profile is plotted in FIG.8, which appears to have features split across two half planes, reflecting the symmetry of the starting permittivity distribution. Although the optimization was not explicitly set up to favor a binary pattern, the left half plane of permittivities can adopt a distribution that is close to binary. FIG.8 illustrates the optimized permittivity distribution over the 30 x 60 pillar array. The two color bars plot the frequency-dependent permittivity at the two wavelengths due to material dispersion.
[0062] The transmitted intensity profiles for both wavelengths are plotted in FIG.9, which demonstrates that the incident illumination is sorted into two spatial regions based on the wavelength. FIG.9 illustrates the intensity distribution |Ex|2+|Ey|2+|Ez|2at the monitor plane placed 50 pixels (687.5 nm) above the pillar array, under two illumination wavelengths, 15 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 showing how the different wavelengths are deflected to two different regions. The intensity profiles can be individually normalized to the maximum intensity on each plane.
[0063] The pixelated patterns with varying dielectric constants could be approximated experimentally using a phase change material. The inverse-design framework of the present disclosure can be an initial proof-of-concept employing a differentiable FDTD and can be augmented using other topology optimization tactics such as binarization schemes (to produce single-level designs with binary fill factors), the incorporation of fabrication tolerances, and level-set representations for curved structures.
[0064] The MMAD FDTD architecture can enable nanophotonic devices to be modelled differentiably at substantially lower memory cost compared to RM AD. This potentially allows these computations to be performed rapidly and in parallel on a single graphics processing unit (GPU) instead of dedicated large-memory enterprise computing clusters. The MMAD architecture can be more generally applied to any grid-like simulator that can be reverse time-stepped, such as quantum mechanical wavefunctions in space, low Reynolds number fluid dynamics, and dissipationless solid mechanics. MMAD can pave the way for the creation of differentiable simulators incorporating multiple coupled physical influences and performing high-dimensional optimization over the control parameters governing these dynamics.
[0065] The time and memory scaling relations for gradient calculation in different schemes (e.g., finite difference, forward mode, reverse mode, adjoint method, and minimal memory reverse mode) can be derived.
[0066] The complexity of one FDTD simulation is described below. Consider an FDTD simulation with ^^^pixels and over ^^்timesteps. FDTD can be used to obtain the steady-state frequency-domain response of a time-independent dielectric geometry. This can involve illuminating the dielectric geometry with an electromagnetic pulse and recording the time-domain field at the positions of interest. The pulsed source can be a superposition of temporal frequencies, allowing the steady-state frequency domain field at each pixel position to be obtained by Fourier transformation of the recorded time-domain field, such as in the case below for angular frequency ^^:16 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0067] The integration does not have to be performed all the way to infinity because ^^^ ^^^ decays over a finite time interval. This can allow the FDTD simulation to be halted at a maximum timestep ^^்and the Fourier transform can be approximated by a discrete form:
[0068] Frequency-domain field values can depend on the field values over the simulation duration. This can require that the time-domain field values be stored over the length of the simulation, which can require significant memory storage that scales as ^^^ ^^^^^்^ if the frequency-domain response of all FDTD pixels is desired. FDTD users can typically select a subset of the pixels for frequency-domain evaluation by introducing a monitor object, but this large storage requirement can be alleviated by only evaluating the Fourier transform over a small number of frequency points ^^^≪ ^^்that have been specified in advance. For each frequency of interest ^^, the FDTD program can record an additional value ^^^ ^^, ^^^ at each pixel of interest. ^^^ ^^, ^^^ is the partial sum at timestep ^^ of the discrete Fourier transform at that frequency.
[0069] The partial sum value can be updated at every timestep and converges to the discrete Fourier transform value by the end of the FDTD simulation. The memory required for this partial sum accumulation (assuming all ^^^pixels are treated in this manner) can thus scale as ^^^ ^^^^^^^, which is much better than the ^^^ ^^^^^்^ memory scaling required for storing all the time-domain information for Fourier transformation after simulation, since ^^^≪ ^^்typically.
[0070] The time complexity for a single FDTD run can scale as ^^^ ^^^^^்^^^^ since the ^^^pixels need to be updated for ^^்timesteps, and there are ^^^partial sums to be updated for each field value in each pixel.
[0071] The single-sided finite difference can evaluate the objective function at the position of interest, and then can evaluate the effect of Ninput small perturbations, one in each of the degrees of freedom. Thus, it can take Ninput+1 function calls to make Ninput first-order gradient approximations. Each function call can be equivalent to an FDTD simulation run that has ^^^ ^^^^^்^^^^ operations and a peak simulation memory usage of ^^^ ^^^^^^^. 17 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011simulation calls can result in a ^^൫ ^^^^^௨௧^^^^^்^^^൯ runtime complexity. The peak simulation memory usage can scale with Ninputif the simulations are run in parallel.
[0072] In the forward mode AD calculation, the chain rule calculations can be performed in the same order as in the forward computation of the objective function. The forward mode calculation can use the computation tree as a backbone. Each node or intermediate variable in the computation tree can be augmented to store the derivative of the intermediate variable with respect to every degree of freedom, of which there are Ninput. For example, for intermediate variable y, and input degrees of freedom^^^^^^ୀ^,…,ே^^^ೠ^, the node These derivatives can bepropagated forward as the computation progresses in the forward direction, until the computation reaches the objective function value f, at which time the algorithm yields the derivative values the desired gradient (e.g., gradient of thefunction).
[0073] During the forward mode AD process, the derivative values stored at previous nodes may not be re-used. This means that once the partial derivatives at each node are propagated to the next layer in the FDTD computation tree, which corresponds to an update timestep, the memory associated with the partial derivatives can be freed. Effectively, the forward mode AD can just keep track of the field values and its derivatives at one timestep. Thus, the peak memory complexity of forward mode AD in FDTD can scale as ^^^ ^^^^^௨௧^^^^^^^.
[0074] Since the elementary mathematical operations at each node can be duplicated by Ninput times, the time complexity of forward mode AD in FDTD can be Ninput times that of the single run FDTD, yielding a time complexity of ^^^ ^^^^^௨௧^^^^^்^^^^.
[0075] The reverse mode algorithm can require the entire computation tree and intermediate values to be stored for the backpropagation step. Thus, the peak memory complexity can scale as ^^൫ ^^^௨௧^௨௧^^^^^்^^^൯. For each objective function, the gradient calculation time complexity can scale as an integer multiplied by the time complexity of a single run, independent of Ninput. Thus, the time complexity of reverse mode AD in FDTD can be ^^൫ ^^^௨௧^௨௧^^^^^்^^^൯. 18 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0076] Adjoint optimization in nanophotonics can typically require two FDTD simulations: the forward and adjoint simulation. Both simulations can take place on the same simulation volume and over the same spectral range. Thus, the time complexity for the adjoint method can scale as that of an FDTD run that is performed Noutput times, one for each objective function: ^^൫ ^^^௨௧^௨௧^^^^^்^^^൯. The peak memory complexity can scale as that of a single FDTD run, ^^൫ ^^^௨௧^௨௧^^^^^^൯, assuming that the adjoint simulation for different objective functions are performed in parallel.
[0077] The peak memory requirements for MMAD can come from two contributions: storage of the field derivative values (with respect to each of the Noutputobjective functions) for one timestep of the simulation and storage of the time-domain field values at the recording boundary. The memory required to store the fields at one timestep can be ^^^ ^^^^^^^ and thus can be ^^൫ ^^^௨௧^௨௧^^^^^^൯ for Noutputfield derivative values. For the recording boundary, the peak memory required to store the time-domain field values can be proportional to the number of timesteps multiplied by the number of pixels along the recording boundary ^^ ^^^. Thus, the peak memory scaling for MMAD can be ^^൫ ^^^௨௧^௨௧^^^^^^^ ^^்^^ ^^^൯.
[0078] The time complexity of MMAD may not be the same as that of reverse mode AD, less ^^൫ ^^^^^்^^^൯ memory storage operations (no need to store the forward pass intermediate values) and with an additional ^^൫ ^^^^^்^^^൯ operations for the backward time- stepping FDTD. Thus, the time complexity of MMAD can also be ^^൫ ^^^௨௧^௨௧^^^^^்^^^൯.
[0079] The recording layer can serve as a simulation region boundary that records the electromagnetic field values in time during the forward pass, then plays back the field values during the time-reversal simulation. Specifically, it can include a discrete set of connected pixels^^^, ^^, ^^^∈ ^^ that forms a closed boundary. The pixels at which the topological optimization is performed (e.g., pillar pixels) may be enclosed within the interior of ^^. The electromagnetic D and H fields can be stored as a function of time during the forward sweep. The field storage operation can occur immediately after a boundary pixel field is updated to a newer value at each timestep. Due to the half-unit displaced positions within the FDTD YeeGrid scheme, for the pixel spanning the cubic cell^^^, ^^ ^ 1^ൈ^^^, ^^ ^ 1^ൈ ^ ^^, ^^ ^ 1^, thefollowing can be stored: 19 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0080] For the time-reversal simulation that runs in parallel with the back-propagation step, the simulation region can be restricted to the interior of ^^. For cases in which ^^ includes multiple non-intersecting recording boundaries, the disjoint interiors can be divided into multiple independent simulation regions and then they can be time-reversed independently and in parallel with multi-threading. There may not be additional boundary conditions (e.g., PMLs) required during the time-reversal simulation since the recording layer can form a closed boundary around the time-reversal simulation region.
[0081] In first order FDTD, only the pixels immediately adjacent to the pixel to be updated can be involved in the update equation. When pixels adjacent to the recording layer are time-stepped in reverse, they can extract the requisite field values from the recording layer to be substituted in the update equation.
[0082] The MMAD FDTD platform results can be validated against those obtained from a FDTD software suite, Lumerical FDTD 2021 R2 (Ansys Inc.) shown in FIG.10. The Lumerical FDTD can be used to simulate the identical geometry, pixel size, and Courant number as in FIG.3 for different pillar configurations. As in the MMAD FDTD, a Total Field Scattered Field (TFSF) source and boundary can be employed and the FDTD boundary conditions can all be PMLs. For the Lumerical FDTD, the simulation time can be set to 50 fs with an automatic shutoff threshold of 5 ൈ 10ି^ଶas the energy fraction below which the simulation will automatically terminate, effectively ensuring that the simulation does not end before the full time elapses. A staircasing mesh method can be used where the permittivity mesh coincides with the spatial Cartesian mesh. The electromagnetic field intensity can be recorded at the same monitor plane as in FIG.3. Three different pillar configurations can be simulated: a single square pillar, three non-identical rectangular pillars, and alternating air / material pixels in a checkerboard pattern, corresponding to each row of FIG.10, respectively. The normalized cross-correlation between the MMAD FDTD intensity pattern and the Lumerical FDTD intensity pattern can be 0.997, 0.997 and 0.999 for the three pillar configurations, respectively, indicating strong numerical agreement. For the single square 20 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 pillar, the zeroth order transmission phase is െ0.023 rad for MMAD FDTD and െ0.027 rad for the Lumerical FDTD. For the three rectangular pillars, the transmission phase is 0.160 rad for MMAD FDTD and 0.167 rad for the Lumerical FDTD. For the checkerboard pattern, the transmission phase is 0.129 rad for MMAD FDTD and 0.128 rad for the Lumerical FDTD. Thus, the transmission phase deviation obtained from the two simulators is on the order of 10ିଷπ. The slight deviation between the two simulators can be due to the different PML implementation.
[0083] FIG.10 illustrates validation of the accuracy of MMAD FDTD field calculations against a FDTD suite (e.g., Lumerical FDTD). Three pillar structures can be simulated (first column: a, e, and i) and the transmitted field intensities in the YZ plane using the MMAD FDTD and Lumerical FDTD are plotted in the second (b, f, j) and third (c, g, k) columns. The fourth column (d, h, k) plots the difference in intensity obtained between the two simulation platforms.
[0084] The full 3D FDTD region for the passive color sorter can have a geometry similar to that of FIG.3, with an increased ^^ ൈ ^^ ൈ ^^ extent of 184 ൈ 50 ൈ 70 pixels (2530 nm ൈ 687.5 nm ൈ 962.5 nm) excluding the 10 PML layers surrounding all six boundary surfaces. The TFSF region can have an extent of 152 ൈ 38 ൈ 68 pixels (2090 nm ൈ 522.5 nm ൈ 935 nm) so that its X extents are 6 pixels away from the PML boundaries in the X direction. The substrate can be glass with a fixed refractive index of 1.44. The structure to be optimized can be a compact 30 ൈ 60 array of pillars on the YZ plane.Each pillar can have an ^^ ൈ ^^ ൈ ^^ dimension of 72 ൈ 1 ൈ 1 pixels (990 nm ൈ 13.75 nm ൈ13.75 nm) so that the full array has an extent of 72 x 30 x 60 pixels (990 nm x 412.5 nm x825 nm). The optimization tunable parameters can be the permittivities of these 30 ൈ 60 ൌ1800 pillars. To avoid optimizing with parameter bounds, a latent space can be chosen forthe normalized permittivity ^^ to be on the full real line, then the real line can be mapped monotonically to a bounded relative permittivity ^^^range using the hyperbolic tangent function. To account for material dispersion, the relative permittivity bounds can correspond to that of real materials: air and TiO2. Thus, for a pillar with normalized permittivity ^^, its relative permittivities at the two wavelengths of interest ^^^, ^^ଶare: ^^^^^^, ^^^^ൌ ^^^,்^ைమ^^^^^⋅1 ^^ 1 ^^ (8)2^tanh2 ^ 1^ ^ ^^^,^^^^ ^^^^ ⋅ ^1 െ2 ^tanh2 ^ 1^൨^^ ^ ^^, ^^ ^ ൌ ^ ^1 ^^ 1 ^^^ଶ ^^^,்^ைమ ^^ଶ ⋅2 ^tanh2 ^ 1^ ^ ^^^,^^^^ ^^ଶ^ ⋅ ^1 െ2 ^tanh2 ^ 1^൨21 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0085] The numerical values used are ^^^,்^ைమ^488 nm^ ൌ 2.7312ଶand ^^^,்^ைమ^633 nm^ൌ 2.3893ଶ. This method can incorporate material dispersion. More complex versions which incorporate analytic approximations can be used as well. The device can be illuminated by a z-polarized plane wave propagating in the x-direction. The objective function ^^^,ଶcan be defined to be maximized for each wavelength to be the overlap between the transverse intensities | ^^௭|ଶ^ ^^, ^^^ recorded at a 30 ൈ 60 pixel (412.5 nm ൈ 825 nm)monitor plane placed 50 pixels (687.5 nm) above the pillars and a desired intensity profile.| ^^௭|ଶ can be used as a proxy for the total intensity because the incident polarization is z-directed and the total transmitted intensity is dominated by the | ^^௭|ଶcomponent. The desired intensity profile can be selected for each wavelength to be gaussians with different center positions ^ ^^^,ଶ, ^^^,ଶ^ and width W=8 pixels (110 nm).
[0086] The optimization details for a color sorter are shown in FIGS.11-13. The gaussian center positions and widths can be plotted over the optimized intensity profile in FIG.11. The total objective function to be maximized for the system ^^ can be the sum of the two individual wavelength objective functions ^^ ൌ ^^^^ ^^ଶ. FIG.11 illustrates the location of the target gaussian intensity profiles. The target gaussian intensity profiles are centered on the black crosses and widths (dotted line) are shown overlaid on the converged intensity profiles at each wavelength after optimization.
[0087] The gradient of ^^ with respect to each of the 1800 permittivity values can be computed using MMAD and the optimization of ^^ can be performed using simple gradient descent with a constant learning rate. The objective function for each of ^^^,ଶa function of iteration is plotted in FIG.12. FIG.12 illustrates the variation of objective function values with iteration number. FIG.12 illustrates the variation of objective function values with iteration number for the individual wavelength objective functions ^^^,ଶ. A Gaussian smoothing filter can be applied to the right-hand side of the permittivity profile after iteration 599, resulting in a transient decrease in the objective function
[0088] The starting permittivity profile (in the normalized latent space units) is chosen to be uniformly 0 on the left and a random uniform distribution from 0 to 5 inclusive on the right (FIG.13). This starting configuration was chosen as the optimized structure 22 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 significantly outperformed that of other starting configurations (uniform distribution, two halves with uniform distributions). To reduce the grainy permittivity pattern, a Gaussian kernel of five-pixel width can be applied to the right-hand side of the latent space permittivity pattern after step 599 of optimization. FIG.13 plots the permittivity pattern before and after this Gaussian smoothing step was performed. The permittivity profile after 954 optimization iterations is plotted in FIG.8. FIG.13 illustrates the permittivity profiles of the pillar array at different iteration numbers. Iteration 0 is the initial configuration, and iteration 954 is the final configuration. The permittivities before and after the iteration 599 smoothing step are also shown.
[0089] The performance of the inverse-designed device can be validated by simulating it in a commercial FDTD suite (Ansys Lumerical 2022 R2.3) in addition to the minimal memory FDTD. A staircasing mesh method can be used where the permittivity mesh coincides with the spatial Cartesian mesh. The commercial FDTD simulation can be run for 100 fs without an automatic early shutoff. The transmitted intensity plots in FIG.9 and FIG. 11 are of the Lumerical FDTD electric field profiles.
[0090] FIG.14 illustrates a method 1400 for minimal memory differentiation. The method 1400 can include modeling a physical structure (BLOCK 1405). The method 1400 can include determining a numerical relationship (BLOCK 1410). The method 1400 can include calculating a gradient (BLOCK 1415). Minimal memory differentiation can be used for inverse design of a device. Inverse design can be an optimization problem in which given a particular behavior or performance metric, a device is sought after that achieves this behavior or performance metric. The method 1400 can include a method of generating a device (e.g., high performance device). The method 1400 can include a minimal memory differentiable FDTD for photonic inverse design. The method 1400 can include a method for obtaining the gradient. For example, the method 1400 can include obtaining the gradient for gridded physical simulators (e.g., gridded systems). A gridded system can be a system in which field values are defined on various coordinates (e.g., spatial coordinates, frequency coordinates, or other parameter coordinates). The system (e.g., gridded system) can have the property of time-reversibility. In a time-reversible system, the amount of information can be conserved going from one time-step to another time-step. Given the state of the system at a future time, the system can be reconstructed at a past time. The method 1400 can include inverse-designing a device based on minimal memory differentiation. The method 1400 can 23 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 apply to a physics simulator that uses a grid. The method 1400 can apply to Maxwell solvers for electromagnetism.
[0091] The method 1400 can include modeling a physical structure (BLOCK 1405). Modeling the physical structure can include iterative grid-based modeling of the physical structure. Modeling the physical structure can include iterative grid-based modeling of the physical structure of a device. The device can include a differentiable nanophotonic simulator. The simulator can achieve efficient gradient computation using reverse mode automatic differentiation. The iterative grid-based modeling of the physical structure of the device can output a grid-based numerical model. Grid-based modeling can include modeling a physical system by storing information at different positions in space. Grid-based modeling can include modeling the physical system by storing information on a coordinate system. The device can represent a region of unknown geometry. The device can represent a region of known scattering properties.
[0092] The method 1400 can include determining a numerical relationship (BLOCK 1410). The method 1400 can include determining a numerical relationship between one or more objective values of the device and one or more properties of the device. The numerical relationship can include a finite difference time domain simulation configured to operate in one or more of one spatial dimension, two spatial dimensions, or three spatial dimensions. The numerical relationship can include finite element discretization. The numerical relationship can be represented as a differentiable computational tree.
[0093] The one or more objective values can correspond to one or more performance efficiencies. The one or more objective values can correspond to one or more computational accuracies. The one or more objective values correspond to one or more transmitted phases, transmitted amplitudes, transmitted temporal frequencies, transmitted spatial frequencies, transmitted polarization states, transmitted coherence parameters, reflected phases, reflected amplitudes, reflected temporal frequencies, reflected spatial frequencies, reflected polarization states, reflected coherence parameters, calculated material contrasts, material impedances, or scattering properties. The one or more objective values can represent a quantitative similarity between the known scattering properties and calculated scattering properties.
[0094] The one or more properties of the device can include at least one of the physical structure of the device, one or more electromagnetic properties of the device, one or 24 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 more photonic properties of the device, one or more mechanical properties of the device, or one or more acoustic properties of the device. The at least one of the physical structure of the device, the one or more electromagnetic properties of the device, the one or more photonic properties of the device, the one or more mechanical properties of the device, or the one or more acoustic properties of the device can be tunable parameters. The at least one of the physical structure of the device, the one or more electromagnetic properties of the device, the one or more photonic properties of the device, the one or more mechanical properties of the device, or the one or more acoustic properties of the device can be one or more of refractive index, dielectric permittivity, magnetic permeability, birefringence, diattenuation, retardance, attenuation, phase delay, group delay, group delay dispersion, electrical conductivity, electrical impedance, loss, gain, elastic modulus, damping, reflectivity, transmittivity, absorptivity, emissivity, roughness.
[0095] The physical structure of the device to be optimized can be parametrized by one or more of geometrical sizes, rotation angles, tilts, fill factors, grid size, lattice vectors. The physical structure of the device can include at least one of a metasurface, a spatially- variant phase plate, a spatial light modulator, a spatially-variant waveplate, an acoustic transducer, a water pump, a pneumatic actuator, or a magnetic lens.
[0096] The method 1400 can include calculating a gradient (BLOCK 1415). For example, the method 1400 can include calculating a gradient (e.g., final gradient) of the one or more objective values with respect to one or more parameters corresponding to the one or more properties of the device by minimal memory reverse mode automatic differentiation. The gradient of the one or more objective values with respect to a geometry of the device can be used to reconstruct the unknown geometry. The method 1400 can include calculating an approximate solution to an inverse scattering problem. For example, the method 1400 can include calculating an approximate solution to an inverse scattering problem based on the gradient. The inverse scattering problem can include the problem of imaging through scattering media (e.g., media that scatters light).
[0097] The minimal memory reverse mode automatic differentiation can include calculating field values on a grid by iteratively updating the field values for one or more steps. A physical system can be modeled by storing information at different positions in space (e.g., on the grid). The physical system can have a behavior which can be recorded in time by iteratively updating the field values. 25 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0098] The minimal memory reverse mode automatic differentiation can include performing reverse iteration (e.g., reverse simulation, reverse calculation) of the grid-based numerical model to output field values on the grid at earlier steps of the one or more steps. Reverse iteration can include stepping the system backwards in time. The state of the system at each timestep can contain the same amount of information as the state of the system at the previous timestep. The state of the system at each timestep can contain the same amount of information as the state of the system at future timestep.
[0099] The minimal memory reverse mode automatic differentiation can include performing backpropagation based on the field values on the grid to compute intermediate gradients of the one or more objective values with respect to the one or more parameters. The intermediate gradients can include, for example, ^^ ^^⁄ ^^^ ^^ ൌ 1^ or ^^ ^^⁄ ^^^ ^^ ൌ 1⁄ 2 ^ . The intermediate gradients can include the gradients calculated after ^^ ^^⁄ ^^^ ^^ ൌ ^^்^ and before ^^ ^^⁄ ^^^ ^^ ൌ 0^ . The intermediate gradients can include the gradients calculated after ^^ ^^⁄ ^^^ ^^ ൌ ^^்^ and before ^^ ^^ / ^^ ^^. The reverse iteration of the grid-based numerical model can occur in parallel with the backpropagation. The reverse iteration of the grid-based numerical model can occur in the same direction as the backpropagation.
[0100] The memory used for storing the field values on the grid can be released after the one or more steps. For example, the memory used for storing the field values on the grid can be released after or more timesteps, but the field values are not stored for the entire sequence of iterations. The field values can be stored for one or more iteration steps at or near grid positions at which information loss occurs. The gradient can include the first partial derivative of the objective values with respect to one or more tunable parameters. The gradient can include the second partial derivative of the objective values with respect to one or more tunable parameters. The gradient can include the third partial derivative of the objective values with respect to one or more tunable parameters.
[0101] The field values can include electromagnetic field values. The method 1400 can include storing the electromagnetic field values proximate to perfectly matched layer (PML) boundary conditions. A PML can include an artificial absorbing layer for wave equations. The MPL can be used to truncate computational regions in numerical methods to simulate problems with open boundaries. The method 1400 can include storing the electromagnetic field values proximate to lossy surfaces. Lossy surfaces can include dissipative surfaces or media. The method 1400 can include storing the electromagnetic field values proximate to perfect electrical conductor (e.g., perfect conductor, perfect electric 26 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 conductor) surfaces. A perfect electric conductor can include an idealized material exhibiting infinite electrical conductivity or zero resistivity.
[0102] FIG.15 illustrates a method 1500 for minimal memory differentiation. The method 1500 can include calculating field values (BLOCK 1505). The method 1500 can include calculating a gradient of objective values (BLOCK 1510). The method 1500 can include calculating a gradient of an objective function (BLOCK 1515). The method 1500 can include a method of generating a device (e.g., high performance device). The method 1500 can include a minimal memory differentiable FDTD for photonic inverse design. The method 1500 can include a method for obtaining the gradient of the objective function. The method 1500 can include inverse-designing a device based on minimal memory differentiation. The method 1500 can apply to a physics simulator that uses a grid. The method 1500 can apply to Maxwell solvers for electromagnetism.
[0103] The method 1500 can include calculating field values (BLOCK 1505). For example, the method 1500 can include calculating one or more field values. The method 1500 can include calculating one or more final field values (e.g., ^^^^^ ൌ ^^்^). The method 1500 can include calculating one or more intermediate field values. For example, the method 1500 can include calculating one or more first intermediate field values (e.g., ^^^^^ ൌ 1^). The method 1500 can include calculating one or more second intermediate field values (e.g., ^^^ ^^ ൌ 1⁄ 2 ^). The method 1500 can include calculating one or more initial field values (e.g., ^^^^^ ൌ 0^). Calculating the one or more final field values, the one or more first intermediate field values, the one or more second intermediate field values, and the one or more initial field values can occur during reverse iteration (e.g., reverse simulation, reverse calculation). Reverse iteration can include stepping the system backwards in time.
[0104] The method 1500 can include calculating a gradient of objective values (BLOCK 1510). For example, the method 1500 can include calculating a first gradient (e.g., ^^ ^^⁄ ^^^ ^^ ൌ ^^்^ ) of one or more objective values. The method 1500 can include calculating a first intermediate gradient (e.g., ^^ ^^⁄ ^^^ ^^ ൌ 1^ ) of the one or more objective values. The method 1500 can include calculating a second intermediate gradient (e.g., ^^ ^^⁄ ^^^ ^^ ൌ 1⁄ 2 ^ ) of the one or more objective values. The method 1500 can include calculating a final gradient (e.g., ^^ ^^⁄ ^^^ ^^ ൌ 0^ ) of the one or more objective values.
[0105] The method 1500 can include calculating a gradient of an objective function (BLOCK 1515). The method 1500 can include calculating a gradient (e.g., ^^ ^^ / ^^ ^^) of an 27 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 objective function based on the final gradient. ^^ can represent control parameters in the simulation. Calculating the first gradient, the first intermediate gradient, the second intermediate gradient, the final gradient, and the gradient of the objective function can occur during backpropagation. The reverse iteration and the backpropagation can occur in parallel. The reverse iteration of the grid-based numerical model can occur in the same direction as the backpropagation.
[0106] The method 1500 can include calculating one or more final field values, each of the one or more final field values corresponding to one or more coordinates. The method 1500 can include calculating a first gradient of one or more objective values based on the one or more final field values. The method 1500 can include calculating one or more first intermediate field values based on the one or more final field values, each of the one or more first intermediate field values corresponding to the one or more coordinates. The method 1500 can include calculating a first intermediate gradient of the one or more objective values based on the first gradient and the one or more first intermediate field values. The method 1500 can include calculating one or more second intermediate field values based on the one or more first intermediate field values, each of the one or more second intermediate field values corresponding to the one or more coordinates. The method 1500 can include calculating a second intermediate gradient of the one or more objective values based on the first intermediate gradient and the one or more second intermediate field values. The method 1500 can include calculating one or more initial field values based on the one or more second intermediate field values, each of the one or more initial field values corresponding to the one or more coordinates. The method 1500 can include calculating a final gradient of the one or more objective values based on the second intermediate gradient and the one or more initial field values. The method 1500 can include calculating a gradient of an objective function based on the final gradient.
[0107] The method 1500 can include calculating, during a forward pass, the one or more initial field values prior to the one or more final field values. The method 1500 can include calculating, during the forward pass, the one or more first intermediate field values prior to the one or more final field values. The method 1500 can include calculating, during the forward pass, the one or more second intermediate field values prior to the one or more first intermediate field values. The method 1500 can include calculating, during the forward pass, the one or more initial field values prior to the one or more second intermediate field values. 28 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0108] As used herein, the singular terms “a,” “an,” and “the” may include plural referents unless the context clearly dictates otherwise. Spatial descriptions, such as “above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,” “vertical,” “horizontal,” “side,” “higher,” “lower,” “upper,” “over,” “under,” and so forth, are indicated with respect to the orientation shown in the figures unless otherwise specified. It should be understood that the spatial descriptions used herein are for purposes of illustration only, and that practical implementations of the structures described herein can be spatially arranged in any orientation or manner, provided that the merits of embodiments of this disclosure are not deviated by such arrangement.
[0109] As used herein, the terms “approximately,” “substantially,” “substantial” and “about” are used to describe and account for small variations. When used in conjunction with an event or circumstance, the terms can refer to instances in which the event or circumstance occurs precisely as well as instances in which the event or circumstance occurs to a close approximation. For example, when used in conjunction with a numerical value, the terms can refer to a range of variation less than or equal to ±10% of that numerical value, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%. For example, two numerical values can be deemed to be “substantially” the same if a difference between the values is less than or equal to ±10% of an average of the values, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%.
[0110] Additionally, amounts, ratios, and other numerical values are sometimes presented herein in a range format. It is to be understood that such range format is used for convenience and brevity and should be understood flexibly to include numerical values explicitly specified as limits of a range, but also to include all individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly specified.
[0111] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular can include implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein can include implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their 29 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.
[0112] Any implementation disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternate implementation,” “various implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.
[0113] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Elements other than ‘A’ and ‘B’ can also be included.
[0114] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods.
[0115] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.
[0116] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein. 30 4894-8793-0977Atty. Dkt. No.: 098930-0387 HU 9011
[0117] While the present disclosure has been described and illustrated with reference to specific embodiments thereof, these descriptions and illustrations do not limit the present disclosure. It should be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the true spirit and scope of the present disclosure as defined by the appended claims. The illustrations may not be necessarily drawn to scale. There may be distinctions between the artistic renditions in the present disclosure and the actual apparatus due to manufacturing processes and tolerances. There may be other embodiments of the present disclosure which are not specifically illustrated. The specification and drawings are to be regarded as illustrative rather than restrictive. Modifications may be made to adapt a particular situation, material, composition of matter, method, or process to the objective, spirit and scope of the present disclosure. All such modifications are intended to be within the scope of the claims appended hereto. While the methods disclosed herein have been described with reference to particular operations performed in a particular order, it will be understood that these operations may be combined, sub-divided, or re-ordered to form an equivalent method without departing from the teachings of the present disclosure. Accordingly, unless specifically indicated herein, the order and grouping of the operations are not limitations of the present disclosure. 31 4894-8793-0977
Claims
Atty. Dkt. No.: 098930-0387 HU 9011 WHAT IS CLAIMED IS:
1. A method, comprising: iterative grid-based modeling of a physical structure of a device to output a grid-based numerical model; determining a numerical relationship between one or more objective values of the device and one or more properties of the device; and calculating a gradient of the one or more objective values with respect to one or more parameters corresponding to the one or more properties of the device by minimal memory reverse mode automatic differentiation, the minimal memory reverse mode automatic differentiation comprising: calculating field values on a grid by iteratively updating the field values for one or more steps; performing reverse iteration of the grid-based numerical model to output field values on the grid at earlier steps of the one or more steps; and performing backpropagation based on the field values on the grid to compute intermediate gradients of the one or more objective values with respect to the one or more parameters.
2. The method of claim 1, wherein the one or more properties of the device comprises at least one of the physical structure of the device, one or more electromagnetic properties of the device, one or more photonic properties of the device, one or more mechanical properties of the device, or one or more acoustic properties of the device.
3. The method of claim 1, wherein the numerical relationship comprises a finite difference time domain simulation configured to operate in one or more of one spatial dimension, two spatial dimensions, or three spatial dimensions.
4. The method of claim 1, wherein the numerical relationship comprises finite element discretization.
5. The method of claim 1, wherein the one or more objective values correspond to one or more performance efficiencies or computational accuracies. 32 -8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 6. The method of claim 1, wherein the one or more objective values correspond to one or more transmitted phases, transmitted amplitudes, transmitted temporal frequencies, transmitted spatial frequencies, transmitted polarization states, transmitted coherence parameters, reflected phases, reflected amplitudes, reflected temporal frequencies, reflected spatial frequencies, reflected polarization states, reflected coherence parameters, calculated material contrasts, material impedances, or scattering properties.
7. The method of claim 1, wherein the physical structure of the device comprises at least one of a metasurface, a spatially-variant phase plate, a spatial light modulator, a spatially-variant waveplate, an acoustic transducer, a water pump, a pneumatic actuator, or a magnetic lens.
8. The method of claim 1, wherein the numerical relationship is represented as a differentiable computational tree.
9. The method of claim 1, wherein memory used for storing the field values on the grid is released after the one or more steps.
10. The method of claim 1, wherein the field values are stored for one or more iteration steps at or near grid positions at which information loss occurs.
11. The method of claim 1, wherein the field values are electromagnetic field values, the method further comprising: storing the electromagnetic field values proximate to perfectly matched layer boundary conditions.
12. The method of claim 1, wherein the field values are electromagnetic field values, the method further comprising: storing the electromagnetic field values proximate to lossy surfaces.
13. The method of claim 1, wherein the reverse iteration of the grid-based numerical model occurs in parallel with the backpropagation.
14. The method of claim 1, wherein: 33 -8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 the device represents a region of unknown geometry and known scattering properties; the one or more objective values represent a quantitative similarity between the known scattering properties and calculated scattering properties; the gradient of the one or more objective values with respect to a geometry of the device is used to reconstruct the unknown geometry; and an approximate solution to an inverse scattering problem is calculated based on the gradient.
15. A method, comprising: calculating one or more final field values, each of the one or more final field values corresponding to one or more coordinates; calculating a first gradient of one or more objective values based on the one or more final field values; calculating one or more first intermediate field values based on the one or more final field values, each of the one or more first intermediate field values corresponding to the one or more coordinates; calculating a first intermediate gradient of the one or more objective values based on the first gradient and the one or more first intermediate field values; calculating one or more second intermediate field values based on the one or more first intermediate field values, each of the one or more second intermediate field values corresponding to the one or more coordinates; calculating a second intermediate gradient of the one or more objective values based on the first intermediate gradient and the one or more second intermediate field values; calculating one or more initial field values based on the one or more second intermediate field values, each of the one or more initial field values corresponding to the one or more coordinates; calculating a final gradient of the one or more objective values based on the second intermediate gradient and the one or more initial field values; and calculating a gradient of an objective function based on the final gradient.
16. The method of claim 15, comprising: calculating, during a forward pass, the one or more initial field values prior to the one or more final field values. 34 -8793-0977Atty. Dkt. No.: 098930-0387 HU 9011 17. The method of claim 15, comprising: calculating, during a forward pass, the one or more first intermediate field values prior to the one or more final field values; calculating, during the forward pass, the one or more second intermediate field values prior to the one or more first intermediate field values; and calculating, during the forward pass, the one or more initial field values prior to the one or more second intermediate field values.
18. The method of claim 15, wherein calculating the one or more final field values, the one or more first intermediate field values, the one or more second intermediate field values, and the one or more initial field values, occurs during reverse iteration.
19. The method of claim 15, wherein calculating the first gradient, the first intermediate gradient, the second intermediate gradient, the final gradient, and the gradient of the objective function, occurs during backpropagation.
20. The method of claim 15, wherein: calculating the one or more final field values, the one or more first intermediate field values, the one or more second intermediate field values, and the one or more initial field values, occurs during reverse iteration; calculating the first gradient, the first intermediate gradient, the second intermediate gradient, the final gradient, and the gradient of the objective function, occurs during backpropagation; and the reverse iteration and the backpropagation occur in parallel. 35 -8793-0977
Citation Information
Patent Citations
Stereoscopic display method and device, electronic equipment and storage medium
CN113012270A
Method for the automatic optimization of a natural gas transport network
US20070260333A1
Generating a smooth grid for simulating fluid flow in a well system environment
US20150066445A1
Waveguide including at least one photonic crystal region for directing signals propagating therethrough
US6804446B1
Computer aided generative design with overall thickness control to facilitate manufacturing and structural performance
WO2021231027A1
