System and method for machine-learning enabled micro-assembly control with the aid of a digital computer
Patent Information
- Application Number
- JP2021201707
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-12-14
- Filing Date
- 2021-12-13
- Publication Date
- 2024-12-20
- Estimated Expiration
- Not applicable · inactive patent
Abstract
Description
Technical Field
[0001] (Rights of the Invention) This invention was made with government support under Contract No. HR00111990027 awarded by the Defense Advanced Research Projects Agency. The government has certain rights in this invention.
[0002] (Field of the Invention) This application generally relates to microassembly control, and more particularly to systems and methods for machine learning-enabled microassembly control using AID of digital computers.
Background Art
[0003] Microscale and nanoscale particle manipulation has attracted the interest of many researchers. The degree to which control can be exerted over the assembly of micro-objects, which are objects measured in microns, and nano-objects, which are objects measured in nanometers, can make a significant difference in many technologies, including microfabrication, biology, and medicine. For example, by accurately controlling the positions of micro-objects such as capacitors and resistors to enable the manufacture of circuits with desired behavior, the manufacture of reconfigurable electrical circuits can be improved. Similarly, the production of photovoltaic solar cell arrays can benefit from being able to place photovoltaic cells with specific qualities at specific positions on the array. Such cells are too small to be placed in their desired arrangement via human or robotic manipulation and require another type of transport mechanism. The microassembly and nanoassembly of particles can also be used to manipulate the microstructure of materials such as biological cells assembled in tissues. There are many other technical fields where increasing the control of the assembly of micro and nano-objects can provide significant advantages.
[0004] Existing technologies cannot control the movement of micro and nano objects with the required precision. For example, uncontrolled mechanical agitation is typically used for directed particle assembly. However, this technique cannot achieve the nearly 100% yield required for certain industrial applications such as electronic device assembly.
[0005] Previous work has also attempted to use electric fields to direct the movement of microscopic objects. For example, in Matei et al., "Micro-scale chiplets position control," Journal of Microelectromechanical Systems, 28(4):643-655, Aug 2019, and Matei et al., "Towards printing as an electronics manufacturing method: Micro-scale chiplet position control," 2017 American Control Conference (ACC), pages 1549-1555, May 2017, the disclosures of which are incorporated by reference, a microscopic object control policy based on a one-step model predictive control approach is disclosed. The disclosed one-dimensional model is capacitance-based, but the actuating mechanism used is based on spiral electrodes, which limits the number of simultaneously actuated electrodes that can be used by the model. Furthermore, the size of the particles with controlled movement is assumed to be small enough not to interfere with the electric field, which may limit the applicability of the described technology to larger particles.
[0006] Similarly, Wang et al., "Dielectrophoretic manipulation of cells with spiral electrodes," Biophysical Journal, 72(4):1887-1899, 199, describes studying the effect of dielectrophoresis on cancer cells. This study assumes that the microparticles involved are spherical and small enough so that their presence does not disrupt the electric field, which limits the applicability of the described technology.
[0007] Control schemes for the control of individual and ensemble colloids are described in Tara D. Edwards and Michael A. Bevan, "Controlling colloidal particles with electric fields," Langmuir, 30(36):10793-10803, 2014. PMID:24601635 ("Edwards"), and their disclosures are incorporated by reference. In particular, Edwards demonstrates how heterogeneous electric fields are used to manipulate individual aggregates of colloidal particles (1 μm to 3 μm in diameter) in water and sodium hydroxide solutions by electrophoresis and electroosmosis. The relative size of the colloids to the electrodes used to generate the electric field, the medium in which the particles are immersed, and the resulting mathematical models do not enable the use of the described techniques in specific industrial applications. In particular, the described techniques are not suitable for assembling micro-objects slightly larger than those described in Edwards' paper. Furthermore, the control schemes used involve high-frequency signals (MHz), which further limits the applicability of such techniques.
[0008] Similarly, Qian et al., “On-demand and location selective particle assembly via electrophoretic deposition for fabricating structures with particle-to-particle precision,” Langmuir, 31(12):3563-3568, 2015. PMID:25314133, whose disclosure is incorporated by reference, demonstrated single-particle precision and location-selective particle deposition where electrophoretic force is the primary driving force for particle (2 μm polystyrene bead) manipulation. The control scheme used was based on constructing a large energy well close to the desired position of the nanoparticles. However, the described technique cannot adequately select and position individual objects. Furthermore, the described technique cannot adequately manipulate asymmetric objects such as semiconductor chips that require orientation control used to construct electronic systems.
[0009] For example, several studies, including Xue et al., "Optimal design of a colloidal self-assembly process," IEEE Transactions on Control Systems Technology, 22(5):1956-1963, Sept 2014, and Xue et al., "Mdp based optimal control for a colloidal self-assembly system," American Control Conference (ACC), 2013, pages 3397-3402, June 2013, which incorporate disclosures by reference, describe driving systems to desired highly crystalline states and controlling stochastic colloidal assembly processes based on Markov decision process optimal control policies. The dynamic models are based on actuator-parameterized Langevin equations. However, in these studies, individual particles are not directly manipulated, and it is unclear how this approach can be used when assembling electrical circuits. Furthermore, the size of the particles used (diameter) is also a factor.
[0010]
number
[0011] For example, the disclosure is incorporated by reference in Grzelczak et al., "Directed self-assembly of nanoparticles," ACS Nano, 4(7):3591-3605, 2010. PMID:20568710; Paulson et al., "Control of self-assembly in micro- and nano-scale systems," Journal of Process Control, 27:38-49, 2015; Mastrangeli et al., "Automated real-time control of fluidic self-assembly of microparticles," Robotics and Automation (ICRA), 2014 IEEE International Conference on pages 5860-5865, May 2014, and Paulson et al., "Control of self-assembly in micro- and nano-scale systems," Journal of Process Control. Other self-assembling control approaches, such as those described in Control, 27:38-49, 2015, do not, by their very nature, allow for easy scaling of the number of objects being moved.
[0012] For example, aqueous solutions into which particles are immersed are a common choice of control medium, as described by Edman et al., "Electric field directed assembly of an InGaAs LED onto silicon circuitry," IEEE Photonics Technology Letters, 12(9):1198-1200, Sept 2000, and Tolley et al., "Dynamically programmable fluidic assembly," Applied Physics Letters, 93(25), 2008, the disclosure of which is incorporated by reference. In such cases, both electrophoretic force and electroosmotic fluid motion are used to drive the particles. However, since water does not behave like a dielectric, it cannot generate an electric field, resulting in a low force for moving objects, which thus significantly limits the size of objects that can be moved using this setting.
[0013] Precise control of cells, quantum dots, and nanowire bases on electroosmosis is employed in Mathi et al., "Simultaneous positioning and orientation of single nano-wires using flow control," RSC Adv., 3:2677-2682, 2013 and Probst et al., "Flow control of small objects on chip: Manipulating live cells, quantum dots, and nanowires," IEEE Control Systems, 32(2):26-53, April 2012, whose disclosures are incorporated by reference. The authors use a linear model of electrode potential, and the effect of particles on the electric field distribution is negligible. However, the presented linearity in electrode potential is not maintained when the driving force is primarily dielectrophoresis, thus limiting the applicability of these techniques. Furthermore, the presented linearity may not be maintained with objects at the microscale.
[0014] Finally, Zemanek et al., “Feedback-controlled dielectrophoretic micromanipulation,” 2018 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS), July 2018, whose disclosure is incorporated by reference, describes feedback in dielectrophoretic micromanipulation. The authors use a simulated annealing technique to solve the optimal control problem and utilize the spherical shape of the particles when constructing the system model. However, this technique limits the applicability of the technique because it depends on the spherical shape of the moving particles.
[0015] Therefore, there is a need for an extensible method to control the assembly of micro-objects of various shapes that can be implemented in real time. [Overview of the project]
[0016] Systems and methods for utilizing machine learning algorithms to move micro-objects to desired positions are described. Sensors, such as high-speed cameras or arrays of capacitive sensors, track the object's position. Arrays of programmable electrodes can be individually controlled by integrated electronic devices such as active matrix arrays, or optical patterns can be projected to address optical switches or optical transistors to dynamically control each electrode, thus generating a dynamic potential energy landscape for manipulating the object. Dielectrophoretic (DEP) and electrophoretic (EP) forces are examples of forces that can be generated to move micro-objects. Micro-objects can be semiconductor chips, integrated circuits, particles, nanodevices, or structures, or any object that an array of electrodes can physically manipulate. One or more computing devices are used to process sensing data, such as images captured by a camera, to estimate the initial position of the micro-object; to generate a continuous representation of a dynamic model for the movement of the micro-object by electrode potentials generated by at least some of the electrodes; to derive the optimal potentials generated by the electrodes to move the micro-object to desired positions using automatic differentiation and Gaussian quadrature on the dynamic model; and to map the newly calculated electrode potentials to an electrode array. In the case of an electrode array with integrated active matrix addressing electronics, the potential on each electrode of the array is updated electronically. In the case of an electrode array controlled by a video projector illuminating an array of optical transistors or optical switches connected to each electrode, the potential on each electrode is updated by updating the image projected onto the array using the video projector. The projected image activates or deactivates the electrodes as indicated by the control input. The system and method enable real-time control of micro-object positioning in a scalable and computationally efficient manner.
[0017] In one embodiment, a system and method for machine learning-enabled microassembly control using a digital computer are provided. The system comprises one or more processors implementing one or more machine learning algorithms, the one or more processors comprising: obtaining one or more parameters of the system for positioning a micro-object; modeling the capacitance between the micro-object and the electrodes using the parameters; estimating the position of the micro-object with the sensors; receiving the further position of the micro-object; defining using capacitance as a continuous representation of a dynamic model for the movement of the micro-object by electrode potentials generated by at least some of the electrodes, the dynamic model being associated with constraints expressed as expected values; applying one or more of the machine learning algorithms to at least part of the continuous representation of the dynamic model to optimize the electrode potentials generated by at least some of the electrodes to move the micro-object from the estimated position to a further position; and using the optimized electrode potentials to activate at least some of the electrodes to induce the movement of the micro-object from the estimated position to a further position.
[0018] Further embodiments of the present invention will be readily apparent to those skilled in the art from the following detailed description, and embodiments of the present invention will be described by illustrating the best mode intended for carrying out the invention. As realized, other different embodiments of the present invention are possible, and some of its details can be modified in various obvious ways, all without departing from the spirit and scope of the invention. Accordingly, the drawings and detailed description should be considered as illustrative and not limiting. [Brief explanation of the drawing]
[0019] [Figure 1]This is a block diagram showing a system for machine learning-enabled microassembly control using a digital computer. [Figure 2] This figure shows the interaction between the video projector and the optical transistor array in Figure 1, according to one embodiment. [Figure 3A] Several diagrams of an example of the optical transistor array shown in Figure 1 according to one embodiment are shown. [Figure 3B] Several diagrams of an example of the optical transistor array shown in Figure 1 according to one embodiment are shown. [Figure 3C] Several diagrams of an example of the optical transistor array shown in Figure 1 according to one embodiment are shown. [Figure 4A] This is an example of a micro-object pattern realized using the system shown in Figure 1. [Figure 4B] This is an example of a micro-object pattern realized using the system shown in Figure 1. [Figure 5] This figure shows a capacitance-based interaction between a single micro-object and five electrodes in the system of Figure 1, according to one embodiment. [Figure 6] This figure shows a COMSOL model of two conductors (a spherical micro-object and an electrode) for capacitance calculation according to one embodiment. [Figure 7] This shows a capacitance function with training data generated by COMSOL simulation attached, according to one embodiment. [Figure 8] This figure shows a 2D capacitance function determined from a 1D model using symmetry characteristics according to one embodiment. [Figure 9] This figure shows the control policy obtained by solving equation (2) according to one embodiment. [Figure 10] This figure shows a control policy for moving a micro-object to a desired position of two dimensions, according to one embodiment. [Figure 11] This flowchart illustrates a method for controlling machine learning-enabled microassemblies using a digital computer. [Figure 12]This flowchart shows a routine for modeling the motion of a micro-object based on capacitance, for use in the method shown in Figure 11, according to one embodiment. [Figure 13] This figure shows a bimodal control policy according to one embodiment, where x is the position of a minute object and the voltage polarity changes by x+b. [Figure 14] This figure shows the force distribution when Vmax = 1 and σ = 0.1, as a function of b. [Figure 15] This figure shows a bimodal control policy according to one embodiment, where x is the position of a minute object and the potential is non-zero, starting at x+b. [Figure 16] This figure shows the force distribution as a function of b when V_max=1 and σ=0.1, using the control policy of equation (6). [Modes for carrying out the invention]
[0020] The systems and methods described below provide an efficient and scalable method for finding control policies and electrode potential assignments to move a micro-object toward a desired position. The analysis involves transforming a micro-model (a dynamic model with variance parameters) into a macro-model (a dynamic model with a continuous distribution) and finding the control policy using machine learning-based optimization. Transforming the micro-model into a macro-model significantly increases computational efficiency. Furthermore, the control policy can be one-dimensional or two-dimensional, and rotational transformations allow for the deriving of an optimal two-dimensional control policy using a one-dimensional control policy.
[0021] Figure 1 is a block diagram showing a system 10 for machine learning-enabled microassembly control using a digital computer. The system 10 enables cooperative assembly of numerous objects 11, such as micro-objects. The size of the objects 11 varies between the nanoscale (<1 μm) and the microscale (hundreds of μm), but other sizes are possible. The objects 11 can be spherical, but other shapes such as rectangles are also possible, and even more shapes are possible. In one embodiment, the diameter of a spatial object 11 is 20 μm to 50 μm, but other diameters are possible. In one embodiment, the dimensions of a rectangular object 11 may be 200 μm x 300 μm x 100 μm, but other dimensions are possible. The objects 11 are immersed in a dielectric fluid (not shown) contained within a housing (not shown), and a layer of film 38 is beneath the dielectric fluid contained within the enclosure. In one embodiment, the dielectric fluid is Isopar® M manufactured by ExxonMobil Chemical Company in Spring, Texas, but other dielectric fluids 17 are also possible. The dielectric fluid may contain one or more additives, such as di-2-ethylhexyl sulfosuccinate (AOT) charge-inducing molecules, but other additives are also possible. Object 11 can be made of aluminum oxide (AlOx), but other materials are also possible. Each of object 11 may be a semiconductor chip, integrated circuit, particle, nanodevice, or structure, or any object that can be physically manipulated, or an array of electrodes. In the following description, object 11 will be referred to as micro-object 11, but other ways of naming object 11 are possible. In one embodiment, the film may be a 50 μm thick perfluoroalkoxy (PFA) film, but other types of films of other thicknesses are possible.
[0022] Below the suspended micro-object 11, multiple programmable electrodes 12 form an array 35, the electrodes 12 configured to generate a dynamic potential energy landscape for manipulating the object with both dielectrophoretic ("DEP") and electrophoretic ("EP") forces. A film is laminated on the electrodes 12. In one embodiment, the electrodes may be square in shape and made of copper, but other shapes and materials are also possible. In one embodiment, the dimensions of the square electrode 12 may be 16 μm wide and 100 nm thick, but other dimensions are possible in further embodiments. The array 35 may include multiple rows of electrodes 12, each row containing multiple electrodes 12.
[0023] The potential generated by electrode 12 is controlled by an array of optical transistors 13, which includes an active-matrix optical transistor backplane set on glass. A number of optical transistors on the backplane form the array 13, and each optical transistor in the array 13 controls the potential generated by a single electrode 12. In particular, each optical transistor in the array 13 is attached to one electrode 12. The array of optical transistors 13 may have additional properties, such as those described in Rupp et al., "Chiplet micro-assembly printer," 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), pages 1312-1315, May 2019, whose disclosure is incorporated by reference.
[0024] The array 13 is optically addressed by a video projector 14 to enable simultaneous dynamic control of the electrostatic energy potential and manipulation of the positions of numerous micro-objects 11. In particular, the video projector 14 is used to address each optical transistor control electrode 12, as shown in Figure 2, allowing for easy changes in the zoom factor and stitching to a larger array. Figure 2 shows the interaction between the video projector 14 and the optical transistor array 13 of Figure 1 according to one embodiment. The video projector 14 acts on the electrodes 12 by projecting a predetermined pattern, an image 37, which is generated based on a control input of the potential that the electrodes should generate to achieve the desired motion of the micro-objects 11, as will be further described below. The pixel light projected by the projector 14 that constitutes the image charges each individual optical transistor 13 in the array to the required extent so that the electrodes 12 are charged by their optical transistors, thereby generating the desired potential.
[0025] Figures 3(A) to 3(C) show several diagrams of an example of the optical transistor array 13 in Figure 1 according to one embodiment. The optical transistor array 13 shown with reference to Figures 3(A) to 3(B) is constructed using manufacturing techniques similar to those used to construct display transistor backplanes, but adapted to a large area (measuring meter). Voltages up to + / -200V are controlled by these optical transistors to display a dynamically changing complex potential terrain. A single voltage signal is synchronized with optical addressing to charge storage capacitors on each pixel. The dynamic electric field pattern is generated by refreshing the projected image pattern (control input). Arrays 13 with pitches of 50 μm, 10 μm, and even 3 μm are constructed, where the pitch is the distance between the centers of the electrodes to which the optical transistors are mounted. A dielectric fluid 17 medium (isopar M) and various additives are used to control the charge.
[0026] Returning to Figure 1, the system 10 further includes a high-speed camera 15 that tracks the position of the moving micro-object 11. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16, which can control the electrodes 12 via the projector 14 to induce the formation of a desired micro-object 11 pattern. Figures 4(A) and 4(B) are examples of micro-object 11 patterns realized using the system 10 of Figure 1. Other micro-object 11 patterns are also possible. In further embodiments, the position of the micro-object can be measured and tracked using a different type of sensor (such as utilizing capacitive sensing). In yet another embodiment, the electrode array 35 may have an integrated active matrix addressing electronic device, and the potential on each electrode of the array is electronically updated when the electrode potential required for the movement of the desired micro-object is generated.
[0027] The connection between one or more computing devices 16 may be direct, for example, via wire or wireless connection to physically close the one or more computing devices 16, or the connection may be via internet work to physically distance the one or more computing devices 16 via the internet or cellular network. One or more computing devices 16 include one or more processors. In one embodiment, the computer processor may be a central processing unit (CPU), a graphics processing unit (GPU), or a mixture of a CPU and a GPU, but other types of processors or mixtures of processors are also possible. In one embodiment, two or more of the processors may perform processing in parallel, as further described in Matei et al., U.S. Patent Application No. 17 / 098,816, filed November 16, 2020, titled System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer, the disclosure of which is incorporated by reference. Additional efficiency and improved scalability can be obtained by parallel computing capability for tracking a large number of particles and for voltage pattern generation.
[0028] While one or more computing devices 16 are shown as servers, other types of computer devices are possible. The computing device 16 may include one or more modules for performing the embodiments disclosed herein. A module may be implemented as a computer program or procedure written as source code in a conventional programming language and presented as object or bytecode for execution by a processor. Alternatively, a module may also be implemented as an integrated circuit or in hardware burned into a read-only memory component, and each of the computing devices 16 may function as a specialized computer. For example, if a module is implemented as hardware, that particular hardware is specialized to perform the calculations and communications described above, and cannot be used by other computers. Furthermore, if a module is burned into a read-only memory component, the computer storing the read-only memory is specialized to perform the operations described above that other computers cannot. Various implementations of source code, object code, and bytecode can be stored on computer-readable storage media such as floppy disks, hard drives, digital video disks (DVDs), random access memory (RAM), read-only memory (ROM), and similar storage media. Other types of modules and module functions, as well as other physical hardware components, are possible. For example, computing device 16 may include, but is not limited to, input / output ports, network interfaces, and other components found in programmable computing devices, such as non-volatile storage devices. In embodiments where computing device 16 is a server, the server may also be cloud-based or a dedicated server.
[0029] One or more computing devices 16 interface to storage 17 and run a dynamic modeler 18 that acquires parameters 19 of system 10, stores the parameters 19 in storage 17, and models the model capacitance between the electrode 12 and the micro-object. The parameters 19 may include the diameter of the micro-object 11, the dimensions of the electrode 12, the dielectric fluid constant (e.g., ε=2), the fixed position of the electrode 12, the materials of the micro-object 11 and the electrode 12, and the vertical distance between the micro-object 11 and the electrode 12 (the "height" of the micro-object 11). Other parameters 19 are still possible.
[0030] The control policy design is based on a dynamic model of the motion of a micro-object under the influence of the electric field generated by the electrode array. The application of potential to electrode 12 results in the generation of electrophoretic force acting on the micro-object 11. Viscous drag, proportional to velocity, opposes the motion of the micro-object (i.e., when the Reynolds number is small, the drag is proportional to the velocity of the non-turbulent flow). Since the mass of the micro-object is negligible, acceleration is negligible. Therefore, the dynamic model 20 for the motion of the micro-object can be described as follows:
[0031]
number
[0032]
number
[0033]
number
[0034]
number
[0035] Feedback control design requires an explicit expression for the capacitance between the micro-object and the electrode. The capacitance modeler 18 creates the model 20 using high-fidelity simulations with COMSOL Multiphysics® software developed by COMSOL, Inc. in Burlington, Massachusetts (“COMSOL”), although other types of simulations using other types of software are also possible. For symmetric micro-objects (e.g., beads) assuming the electrodes are symmetric, the capacitance is estimated by simulating a two-dimensional electrostatic COMSOL model. This means that the capacitance function is of the form C i (η)=C(∥η - η i ∥), where η T = [x,y] indicates the micro-object position,
[0036]
Number
[0037]
number
[0038]
number
[0039] Figure 7 shows the capacitance function with training data generated by COMSOL simulation attached, according to one embodiment. Figure 7 shows C(ξ), i.e., the capacitance between the micro-object and the electrode, as a function of the horizontal position of the micro-object, and the numerical values are fitted to the error function parameterization. The 1D model is transformed
[0040]
number
[0041]
number
[0042] One or more computing devices 16 further perform a position tracker 36, analyzing images 21 captured by a high-speed camera 15 (or another type of sensor) to determine the position 22 of a micro-object 11. The position 22 and dynamic capacitance model 20 are used to determine a control policy 24, which includes a control input 27 (potential) that needs to be generated by the electrode 12, guiding the electrode to the desired position.
[0043] The determination of the control policy 24 is performed using an electrode potential modeler 25, which is run by one or more computing devices 16. As described above, the determination of the control policy 24 involves transforming the micromodel (dynamic model 20 with variance parameters) into a macromodel (dynamic model 20 with a continuous distribution) and finding the control policy using machine learning-based optimization. Transforming the micromodel into a macromodel significantly increases computational efficiency. Furthermore, the control policy 24 can be one-dimensional or two-dimensional, and rotational transformations allow for the derivation of an optimal two-dimensional control policy using a one-dimensional control policy. To enable traceability, an average field-of-view-like approximation of the dynamic model 20 is used, assuming a continuous distribution instead of considering a discrete distribution of array potentials. Under this assumption, a bimodal control policy in which the voltage distribution shifts polarity just before the microobject actually guides the motion of the microobject in the desired direction. The following description begins with the one-dimensional (1D) case and shows how the approach is extended to the two-dimensional (2D) case. In addition, as described below, the policy is derived by solving a regression problem with respect to the voltage of electrode 12, where machine learning algorithms, in particular automatic differentiation, are used to calculate the gradient of the micro-object potential function.
[0044] The intuition behind the transition from discrete to continuous representation can be seen in the emergence of electrical potentials in minute objects.
[0045]
number
[0046]
number
[0047]
number
[0048]
number
[0049]
number
[0050]
number
[0051]
number
[0052]
number
[0053]
number
[0054]
number
[0055] The electrode potential modeler 25 models the function of determining potential assignment as a neural network (NN) with two hidden layers of size 60 and tanh as the activation function, i.e., V(y) = V(y;β), where β is a vector of parameters encompassing the weights and biases of the NN. The following optimization problem can be formulated.
[0056]
number
[0057] The optimal solution (1) of the electrode potential modeler 25 maximizes the infinitesimal object solution in the positive direction. For the negative direction, the sign of the cost function representing the partial derivative of the potential energy U can be changed. The electrode potential modeler uses Gaussian quadrature, a method commonly found in the theory of generalized chaotic polynomials, and its disclosures are incorporated by reference to: A. O'Hagan, "Polynomial Chaos: A Tutorial and Critique from a Statistician's Perspective," University of Sheffield, UK, May 2013; N. Wiener, "The Homogeneous Chaos," Am.J.Math., vol.60, no.4, pp.897-936, 1938; D. Xiu, D. Lucor, C.-H. Su, and G. Em Karniadakis, "Performance Evaluation of Generalized Polynomial Chaos," in Computational Science - ICCS 2003: International Conference, Melbourne, Australia and St. Petersburg, Russia, June 2-4, 2003 Proceedings, Part This is described in IV, Berlin, Heidelberg, 2003, pp. 346-354. Gaussian quadrature provides a tool for efficiently evaluating conditional expectations. Therefore, the expectation value of a function of a random variable using a Gaussian distribution can be accurately approximated using Gauss-Hermit quadrature. Thus, the expectation value of V(Y;β) is given by:
[0058]
number
[0059]
number
[0060]
number
[0061] Therefore, the optimization problem (1) is as follows:
[0062]
number
[0063] The electrode potential modeler 25 performs optimization according to equation (2) using a machine learning algorithm, auto-differentiation. In particular, the electrode potential modeler 25 utilizes the auto-differentiation feature of the Autograd module of PyTorch®, developed by Facebook, Inc. in Menlo Park, California, whose disclosure is incorporated by reference in Maclaurin et al., "Autograd: Effortless gradients in numpy," ICML 2015 AutoML Workshop, 2015, as a function of NN weights and biases.
[0064]
number
[0065] As described below, experimentally, the optimization problem was solved for x=0 and a sequence of sample point numbers n∈{10,20,30,40,50}. All of these yielded the same type of control policy 24 for n=30, as shown with reference to Figure 9. Figure 9 shows the control policy obtained by solving equation (2) according to one embodiment. The resulting control policy is a bimodal control policy that exhibits a sudden change in polarite immediately before the micro-object. Due to the translational symmetry of the micro-object dynamics, it is sufficient to solve the optimization problem only for x=0.
[0066] Due to physical limitations on the addressability of the electrode array, special cases of bimodal control policies are considered by the electrode potential modeler-25.
[0067]
number
[0068] The optimal parameters for the policy are determined by the electrode potential modeler 25. In particular, when b = 0.4307σ, the micro-object moves at maximum velocity in the positive direction (i.e.,
[0069]
number
[0070]
number
[0071] It is not meaningful to activate all electrodes 12, but only those within the interval [x+b, 4σ] are activated, and therefore, cutting off electrode activation after 4σ does not affect the generated force. Thus, this interval imposes a specific length on the image pattern. The discretized control policy is y i For ∈[x+b,4σ] V i =V max Given by, otherwise not assigned, y i is the position of electrode i, which is precisely the control scheme used experimentally. Therefore, the discretized control policy 24 generated to move the micro-object in one dimension is y i For ∈[x+b,4σ] V i =V max Given by, otherwise not assigned (0), y i This is the position of electrode i.
[0072] The above results can also be extended to move the micro-object 11 in two dimensions. The position of the micro-object 11 is represented by x=(x1,x2),
[0073]
number
[0074]
number
[0075]
number
[0076]
number
[0077]
number
[0078] All results determined in the 1D case regarding the selection of b to generate the maximum force are retained in the 2D case as well. It should be noted that discretizing the ideal control policy introduces some error because the micro-object may not be precisely on top of the electrode. Therefore, the experimental policy is based on a pattern including adjacent electrodes in case the electrodes are not precisely aligned in the direction toward the desired position.
[0079] By using the rotational symmetry of the model, the electrode potential modeler 25 determines a 2D control policy for any direction of motion. (0) Let x be the current position of the micro-object. (d) Let be the desired position of the minute object. Next, the control policy 24 is given by the following:
[0080]
number
[0081]
number
[0082] Therefore, by taking the dynamic model 20, the initial microobject position 22, and the desired microobject position 23 as input, the electrode potential modeler 25 can derive a control policy for moving the microobject 11 from the initial position 22 to the desired microobject position 23. The control policy 24 is designed to move the position of one microobject at a time, since the microobjects of this design do not come close to each other enough to cause significant microobject-to-microobject interactions (avoiding interactions that need to be considered, which must be determined according to the shape and potential of the microobjects, at a precise distance that one microobject 11 needs to move away from another microobject; generally, for microobject-to-microobject interactions to occur, the distance between two microobjects must be the same as the distance from the microobject 11 to the electrode 12). The control policy 24 generated by the electrode potential modeler is used by a projector controller 28, which is run by one or more computing devices 16, to map the control policy to an image 37 provided to a video projector 14, and then project the image 37 onto an optical transistor that controls the electrodes 11, thereby acting on the movement of the micro-objects 11. The mapping can be performed as described in U.S. Patent Application Publication 2020 / 0207617, published on July 2, 2020, entitled “Micro-assembler system for manipulating micro-objects,” the disclosure of which is incorporated by reference. In a further embodiment, if the electrode array 35 has an integrated active matrix addressing electronic device, the potential on each electrode of the array is electronically updated to the generated control policy 24.
[0083] By transforming the dynamic model from a discrete formulation to a continuous formulation, the constraints of the model can be rewritten as expected values. This then allows the model to be rerun in a much more efficient way, enabling the deriving of a more effective optimal potential using automatic differentiation. Figure 11 is a flowchart illustrating Method 50 for multi-object microassembly control using a digital computer. Method 50 can be performed using the system in Figure 1. System parameters are obtained, such as the diameter of the micro-objects, the dimensions of the electrodes, the dielectric fluid constants, the initial position and material of one or more micro-objects to be moved (which can be obtained using a high-speed camera or another type of sensor), and the height of the micro-objects, but other parameters are possible (Step 51). Using the parameters, a capacitance-based dynamic model of the motion of the micro-objects is created, as further described below with reference to Figure 10 and as described above with reference to Figure 1 (Step 52). A desired position of at least one of the micro-objects is obtained (Step 53). The control policy is determined using the initial position of one of the micro-objects and the desired position of that micro-object, as described above with reference to Figure 1 (Step 54). In short, the method converts a dynamic model with variance parameters into a continuous representation of the dynamic model and uses automatic differentiation along with Gaussian quadrature to optimize the problem to derive the electrode potentials required to move a micro-object to a desired position in either one or two dimensions. The derived electrode potentials are mapped to an image for display by a video projector (step 54), and electrodes in the electrode array are controlled to guide the movement of the micro-object to the desired position (step 55). If additional movement is required (step 57), if other micro-objects need to be moved, or if the moved micro-object does not reach the desired position (determined using a high-speed camera or another type of sensor), the method returns to step 54. If no additional movement is required (step 57), the method terminates.
[0084] In a further embodiment, if the electrode array 35 has an integrated active matrix addressing electronic device, the potential on each electrode of the array is electronically updated to a control policy 24 generated in place through an image projected by the projector in step 54.
[0085] The system and method use only one micro-object at a time, and since the micro-objects are not close enough to generate micro-object interactions, the dynamic model needs to consider only the interaction between one micro-object and the electrode. A capacitance-based model of the motion of the micro-objects considers both the capacitance between the micro-object and the electrode and the capacitance between two different micro-objects. Figure 12 is a flowchart showing a routine 60 for modeling the motion of micro-objects based on capacitance for use in method 50 of Figure 11, according to one embodiment. A simulation, such as a COMSOL simulation, is performed for the capacitance between the electrode and the micro-objects over a number of distances, as described above with reference to Figure 1 (step 61). A function describing the dependency between the capacitance between the electrode and the micro-objects is defined based on the simulation for the capacitance between the micro-objects and the electrode, as described above with reference to Figure 1 (step 62), and routine 60 ends.
[0086] The system and method described above utilize a bimodal control policy derived using machine learning algorithms and Gaussian quadrature to derive the required electrode potentials. Further mathematical reasoning and experimental details following the derivation and application of the control policy are provided below. The reasoning and details are provided for illustrative purposes only and are not limiting.
[0087] Control design: 1D case As mentioned above, the intuition from discrete representation to continuous representation can be observed in the emergence of electrical potentials in minute objects.
[0088]
number
[0089]
number
[0090]
number
[0091]
number
[0092]
number
[0093]
number
[0094]
number
[0095]
number
[0096]
number
[0097]
number
[0098] The following results demonstrate that the bimodal potential distribution is sufficient to move a micro-object toward a desired position.
[0099] Proposal 1: Let's choose the function that describes the assignment of potentials as follows:
[0100]
number
[0101]
number
[0102]
number
[0103] Proof. As defined by equation (4), the graph of the function V(y) can be seen with reference to Figure 13. Figure 13 shows a bimodal control policy according to one embodiment, where x is the position of a small object and the voltage polarity changes with x+b. The Gaussian distribution of Y is,
[0104]
number
[0105]
number
[0106] To evaluate the partial derivative of the potential energy, we evaluate the energy function at points x and x+ε, and then the limit
[0107]
number
[0108]
number
[0109] The perturbed mean voltage is given by the following:
[0110]
number
[0111]
number
[0112]
number
[0113] Equation (5) is,
[0114]
number
[0115]
number
[0116]
number
[0117] Previous results have shown that the modal voltage distribution across electrode 11 does indeed ensure that the micro-object moves in the desired direction. More importantly, and not unexpectedly, the results show that only a limited number of electrodes are relevant, namely those located in the interval [-4σ, 4σ]. The control policy in the continuous representation of the electrodes is actually implemented by discretizing the control policy, resulting in the following:
[0118]
number
[0119] Control policy As described above, based on projecting an image including a specific pattern, a micro-object is pushed in a desired direction using an experimentally designed control policy. The following explanation analytically shows the reason why the control policy actually functions and formally rediscovers the pattern shape. In the previous section, the 1D case was considered and then extended to the 2D case. Consider the control policy given below.
[0120]
Number
[0121] In the experimentally designed control policy, the voltage is not kept constant for all points, that is, for y ≥ x + b and y < x + b, the electrode potential floats freely. Here, it is assumed that when the frequency at which the image is projected onto the array is sufficiently small, the capacitors associated with the electrode photodiodes are fully discharged when they are not activated (i.e., black is projected onto them). The following results summarize the effect of this new policy on the movement of the micro-object.
[0122] Proposal 2. The function describing the potential assignment is selected as in (6), where V max is the maximum absolute value of the potential. Then, when b = 0.4307σ, the micro-object moves in the positive direction at the maximum speed (i.e.,
[0123]
Number
[0124]
number
[0125] Proof. The proof follows the same steps as the proof in Proposal 1. The main difference is that only the integral over the interval [x+b,∞) is performed. As before, the closed-form formula for the potential energy can be calculated using the Gaussian distribution of Y. In particular, the potential energies at points U(x) and U(x+ε) are given by the following, respectively:
[0126]
number
[0127]
number
[0128] The force as a function of variable b is shown with reference to Figure 16. Figure 16 shows the control policy of equation (6) for V max This figure shows the force distribution as a function of b for the cases where =1 and σ=0.1. Figure 16 also shows the two critical points that maximize the velocity in the desired direction. There are two important differences compared to the policy introduced in equation (4). The critical points are b * Given by ±0.4307σ, the point is numerically estimated, and the magnitude of the force is significantly reduced (a reduction of over 80%).
[0129] Note that activating all electrodes is pointless, but activating only those within the interval [x+b, 4σ], and therefore cutting off electrode activation after 4σ, will not affect the generated force. The discretized control policy is y iFor ∈[x+b,4σ] V i =V max Given by, otherwise not assigned, y i This is the position of electrode i, which is precisely the control scheme used experimentally. The experimentally designed control policy was implemented by taking into account several physical limitations of the operating mechanism, which cannot simultaneously set positive and negative polarity with respect to the electrode potential.
[0130] Optimal control design for machine learning Building upon a formal proof that a bimodal control policy is sufficient to induce the motion of a micro-object in the desired direction, the intriguing question is whether such a policy is optimal. The following explanation does not analytically prove optimality, but leverages a deep learning optimization framework (its disclosure is incorporated by reference, as noted, Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Effortless gradients in numpy. In ICML 2015 AutoML Workshop, 2015; and Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017) to solve a regression problem where the potential distribution of the array is provided. As seen below, the optimization results reinforce the inference that the control policy derived via equation (4) is indeed optimal.
[0131] The function that determines the assignment of potentials is modeled as a neural network (NN) with two hidden layers of size 60 and tanh as the activation function, i.e., V(y) = V(y;β), where β is a vector of parameters encompassing the weights and biases of the NN. The following optimization problem is formulated and previously provided in equation (1).
[0132]
number
[0133] The optimal solution in (1) maximizes the infinitesimal object solution in the positive direction. In the negative direction, the sign of the cost function describing the partial derivative of the potential energy U can be changed. Note that, unlike when the policy of equation (4) is used, the conditional expectation can no longer be evaluated analytically. To address this challenge, we use Gaussian quadrature, which, as mentioned above, is often found in the theory of generalized chao polynomials, and its disclosure is incorporated by reference to: Anthony O'Hagan. Polynomial chaos: A tutorial and critique from a statistician's perspective. Technical report, University of Sheffield, UK, May 2013; Ralph C. Smith. Uncertainty Quantification: Theory, Implementation, and Applications. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2013; Norbert Wiener. The Homogeneous Chaos. American Journal of Mathematics, 60(4):897-936, 1938; Dongbin Xiu, Didier Lucor, C.-H. Su, and George Em Karniadakis. Performance evaluation of generalized polynomial chaos. In Computational Science-ICCS This is described in the 2003 International Conference, Melbourne, Australia and St. Petersburg, Russia, June 2-4, 2003, Proceedings, Part IV, pages 346-354, Berlin, Heidelberg, 2003. The Gaussian quadrature method provides a tool for efficiently evaluating conditional expectations.Therefore, the expected value of a function of a random variable using a Gaussian distribution can be accurately approximated using the Gauss-Hermit quadrature method. Thus, the expected value of V(Y;β) is given by the following:
[0134]
number
[0135]
number
[0136]
number
[0137] Therefore, the optimization problem (1) becomes as given by equation (2) earlier.
[0138]
number
[0139] Using the automatic gradient function of the Autograd Module in the PyTorch® software developed by Facebook, Inc. in Menlo Park, California (described in Maclaurin et al., "Autograd: Effortless gradients in numpy," ICML 2015 AutoML Workshop, 2015), the NN weights and biases are expressed as follows:
[0140]
number
[0141] Extension to 2D control policies This section shows how the previous results can be extended to the 2D case. x=(x1, x2) is the position of the infinitesimal object,
[0142]
number
[0143]
number
[0144]
number
[0145]
Number
[0146] The following bimodal control policy is used.
[0147]
Number
[0148] The micro-object potential is calculated as the mean of the potential function V(y), that is, as follows.
[0149]
Number
[0150] The potential energy can be calculated in closed form and is given by the following.
[0151]
Number
[0152] To evaluate the gradient of the potential energy, the partial derivative of U(x) is calculated. Therefore, it is as follows.
[0153]
Number
[0154]
Number
[0155]
Number
[0156] Similarly, the partial derivative of U(x1, x2) is calculated with respect to x2, which becomes zero, i.e., as follows.
[0157]
Number
[0158] It should be noted that the control policy generates force only on the y1 axis and has exactly the same formula as the force generated in the 1D case. All the results determined in the 1D case regarding the selection of b to generate the maximum force are also retained in the 2D case. It should be noted that the discretization of the ideal control policy introduces some errors because the micro-object may not be exactly above the electrode. Therefore, the experimental policy is based on a pattern including adjacent electrodes for the case where the electrodes are not exactly aligned with the direction towards the desired position.
[0159] A general 2D control policy can be summarized in the following results.
[0160] Proposal 3 x (0) is taken as the current micro-object position, and x (d) is taken as the desired micro-object position. In that case, the control policy is as follows,
[0161]
Number
[0162]
Number
[0163] Proof. The proof is
[0164] [Number] by applying a rotation transformation so that
[0165] [Number] and applying it to policy (11). Also,
[0166] [Number] is orthogonal on the straight line connecting x (0) and x (d) and note that the point x (0) at a distance b from x (b) also passes through the same straight line.
[0167] Although the present invention has been particularly shown and described with reference to its embodiments, those skilled in the art will understand that forms and details thereof, as well as other modifications, can be made therein without departing from the spirit and scope of the present invention.
Claims
1. 1. A system for machine learning enabled microassembly control using a digital computer, comprising: One or more processors implementing one or more machine learning algorithms, the one or more processors comprising: acquiring one or more parameters of a system for positioning a micro-object, the system including a plurality of electrodes, each of the electrodes being programmable, the electrodes being configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes when one or more electrical potentials are generated by one or more of the electrodes; modeling a capacitance between the micro object and the electrode using the parameters; Estimating the position of the micro object with a sensor; receiving a further position of the small object; defining, using the capacitance as a continuous representation of a dynamic model for the motion of the micro-object due to an electrode potential generated by at least one of the plurality of electrodes, the dynamic model being associated with constraints expressed as expected values; applying one of the machine learning algorithms to at least a portion of the continuous representation of the dynamic model to perform an optimization of an electrode potential generated by at least one of the plurality of electrodes to move the micro object from the estimated position to the further position; and using the optimized electrode potential to actuate at least one of the plurality of electrodes to induce the movement of the micro object from the estimated position to the further position.
2. The system of claim 1 , wherein one of the constraints includes a potential energy of the minute object, the potential energy being approximated during the optimization using Gaussian quadrature.
3. The one or more processors: The system of claim 2 , further configured to represent the function that determines the optimized electrode potential assignments as a neural network associated with weights and biases.
4. the one machine learning algorithm is automated differentiation, the one or more processors:
4. The system of claim 3, further configured to use the automated differentiation to calculate partial derivatives of the potential energy as a function of the weights and biases associated with the neural network.
5. The system of claim 3 , wherein the neural network includes two hidden layers and uses tanh as an activation function.
6. The further location is separated from the estimated location by a distance in a direction, and the one or more processors: receiving an additional location separated from the estimated location by the distance in a direction opposite to the direction in which the further location is separated from the estimated location; The system of claim 1 , further configured to: change a sign of the function to optimize the potential for moving the micro object from the estimated position to the additional position.
7. the movement from the estimated position to the further position is in one dimension, receiving an additional location separated from the estimated location in two dimensions; The system of claim 1 , further configured to apply a rotational transformation to the optimized electrode potential to derive the electrode potential to be produced by the electrode to move the micro object from the estimated position to the additional position.
8. The one or more processors: performing a plurality of simulations of the capacitance between the electrode and the micro object; 2. The system of claim 1, further configured to: define a function describing the capacitance between the micro object and each of the electrodes as a function of a distance between the micro object and that electrode.
9. The system of claim 1 , wherein one or more of the processors comprises one or more of at least one central processing unit (CPU) and at least one graphics processing unit (GPU).
10. The system of claim 1 , wherein at least one of the plurality of electrodes is rectangular.
11. 1. A method for machine learning enabled microassembly control using a digital computer, comprising: acquiring, by one or more processors implementing one or more machine learning algorithms, one or more parameters of a system for positioning a micro-object, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes when one or more electrical potentials are generated by one or more of the electrodes; modeling by the processor one or more of the capacitances between the micro-objects and the electrodes using the parameters; estimating, by one or more of the processors, a position of the micro-object based on at least one sensor measurement; receiving, by one or more of the processors, a further position of the micro object; defining, by one or more of the processors, using the capacitance as a continuous representation of a dynamic model for the movement of the micro-object due to an electrode potential generated by at least one of the plurality of programmable electrodes, the dynamic model being associated with constraints expressed as expected values; applying, by one or more of the processors, one of the machine learning algorithms to at least a portion of the continuous representation of the dynamic model to perform an optimization of an electrode potential generated by at least one of the plurality of programmable electrodes to move the micro object from the estimated position to the further position; and using the optimized electrode potential to actuate at least one of the plurality of programmable electrodes to induce the movement of the micro object from the estimated position to the further position.
12. The method of claim 11 , wherein one of the constraints includes a potential energy of the minute object, the potential energy being approximated during the optimization using Gaussian quadrature.
13. 13. The method of claim 12, further comprising expressing the function that determines the optimized electrode potential assignments as a neural network associated with weights and biases.
14. the one machine learning algorithm is automated differentiation; 14. The method of claim 13, further comprising: using the automated differentiation to calculate partial derivatives of the potential energy as a function of the weights and biases associated with the neural network.
15. The method of claim 13 , wherein the neural network includes two hidden layers and uses tanh as an activation function.
16. the further location is separated from the estimated location by a distance in a direction; receiving an additional location separated from the estimated location by the distance in a direction opposite to the direction in which the further location is separated from the estimated location; The method of claim 11 , further comprising: changing a sign of the function to optimize the potential for moving the micro object from the estimated position to the additional position.
17. the movement from the estimated position to the further position is in one dimension, receiving an additional location separated from the estimated location in two dimensions; The method of claim 11 , further comprising: applying a rotational transformation to the optimized electrode potential to derive the electrode potential to be produced by the electrode to move the micro object from the estimated position to the additional position.
18. performing a plurality of simulations of the capacitance between the electrode and the micro object; The method of claim 11 , further comprising: defining a function describing the capacitance between the micro object and each of the electrodes as a function of the distance between the micro object and that electrode.
19. The method of claim 11 , wherein one or more of the processors comprises one or more of at least one central processing unit (CPU) and at least one graphics processing unit (GPU).
20. The method of claim 11 , wherein at least one of the plurality of programmable electrodes is rectangular.