System and method for machine learning-enabled control of micro-object density distribution with the help of a digital computer.

Abstract

A system and method for forming a micro-object density distribution (how densely micro-objects are assembled in a particular spatial region) are provided. [Solution] A high-speed camera tracks the existing object density distribution. An array of phototransistor-controlled electrodes is used to generate a dynamic potential energy terrain for manipulating objects with both DEP and EP forces, and a video projector is used to actuate the array. One or more computing devices are used to process images captured by the camera, estimate the existing density distribution of objects, receive a desired density distribution of micro-objects, define a model that describes the variation in micro-object density over time due to capacitance-based interactions, generate an array of electrode potentials that minimizes the error between the existing density distribution and the desired density distribution, and use the array of electrode potentials to actuate the electrodes.

Description

[Technical Field]

[0001] This application relates, in general, to micro-assembly control, and more specifically, to a system and method for machine learning-enabled micro-object density distribution control with the help of a digital computer. [Background technology]

[0002] The manipulation of micro and nanoscale particles has attracted the interest of many research fields. The degree of control that can be exercised over the assembly of micro-objects (objects whose dimensions are measured in microns) and nano-objects (objects whose dimensions are measured in nanometers) can make a significant difference in many technologies, including microfabrication, biology, and medicine. For example, the manufacture of reconfigurable electrical circuits can be improved by enabling the precise control of the position of micro-objects such as capacitors and resistors in order to fabricate circuits with desired behavior. Similarly, the production of photovoltaic solar cell arrays can benefit from the ability to place photovoltaic cells of specific qualities at specific positions on the array. Such cells are too small to allow for the desired placement of cells by human or robotic operation, requiring a different type of transport mechanism. Micro-assemblies and nano-assemblies of particles can also be used to design the microstructure of materials such as biological cells assembled in tissues. There are many other technological fields in which increasing control over the assembly of micro and nano-objects can offer significant advantages.

[0003] A specific challenge associated with microassemblies is the generation of a specific desired density of micro-objects in a given spatial region. Once achieved, this density distribution can serve as the basis for individual micro-object operations that can achieve desired micro-object manipulations. However, simultaneous control and tracking of individual micro-objects is computationally expensive, and existing techniques cannot control the density distribution of micro-objects with the degree of control and computational efficiency desired for many practical applications.

[0004] For example, uncontrolled mechanical agitation is typically used in directional particle assemblies. However, this technique cannot achieve the nearly 100% yield required for certain industrial applications, such as electronic equipment assemblies.

[0005] Previous studies have also attempted to direct the movement of micro-objects using electric fields. For example, Matei et al., "Micro-scale chiplets position control," Journal of Microelectromechanical Systems, 28(4):643-655, August 2019, and Matei et al., "Towards printing as an electronics manufacturing method: Micro-scale chiplet position control," 2017 American Control Conference (ACC), pp. 1549-1555, May 2017. These disclosures are incorporated by reference and disclose a microchiplet control policy based on a one-stage model predictive control approach. The disclosed one-dimensional model is capacitance-based, but the operating mechanism used is based on helical electrodes, which limited the number of simultaneously operated electrodes that could be used by the model. Furthermore, the size of the particles whose movement is controlled is assumed to be small enough not to interfere with the electric field, which may limit the applicability of the described technique to larger particles.

[0006] Similarly, Wang et al., “Dielectrophoretic manipulation of cells with spiral electrodes,” Biophysical Journal, 72(4):1887-1899, 199, describe studying the effects of dielectrophoresis on cancer cells. This study assumes that the particles involved are spherical and small enough that their presence does not interfere with the electric field, thus limiting the applicability of the described technique.

[0007] Control schemes for individual and ensemble control of colloids are described by Tara D. Edwards and Michael A. Bevan, "Controlling colloidal particles with electric fields," Langmuir, 30(36):10793~10803, 2014, PMID:24601635 ("Edwards"), and their disclosure is incorporated by reference. In particular, Edwards shows how heterogeneous electric fields are used to manipulate individual and ensemble colloidal particles (1 μm to 3 μm in diameter) in water and sodium hydroxide solutions via electrophoresis and electroosmosis. The relative size of the colloids to the electrodes employed to generate the electric field, the medium in which the particles were immersed, and the resulting mathematical models prevent the described techniques from being used for specific industrial applications. In particular, the described techniques are not suitable for assembling micro-objects slightly larger than those considered in Edwards' paper. Furthermore, the control scheme used involves high-frequency signals (MHz), which further limits the applicability of such technology.

[0008] Similarly, Qian et al., "On-demand and location selective particle assembly via electrophoretic deposition for fabricating structures with particle-to-particle precision," Langmuir, 31(12):3563~3568, 2015, PMID:25314133, whose disclosure is incorporated by reference, demonstrated single-particle precision and location-selective particle deposition where electrophoretic force is the primary driving force for particle (2 μm polystyrene bead) manipulation. The control scheme employed was based on constructing a large energy well near the desired location of the nanoparticles. However, the described technique does not enable proper sorting and placement of individual objects. Furthermore, the described technique does not enable proper manipulation of asymmetric objects such as semiconductor chips that require orientation control used to construct electronic systems.

[0009] Several studies, such as Xue et al., "Optimal design of a colloidal self-assembly process," IEEE Transactions on Control Systems Technology, 22(5):1956-1963, September 2014, and Xue et al., "Mdp based optimal control for a colloidal self-assembly system," American Control Conference (ACC), 2013, pp. 3397-3402, June 2013, have incorporated their disclosures by reference, explaining how to drive a system to a desired highly crystalline state and control a probabilistic colloidal assembly process based on an optimal control policy of a Markov decision process. The dynamic model is based on the Langevin equations with actuator parameterization. However, in these studies, individual particles are not directly manipulated, and it is unclear how this approach can be used when assembling electrical circuits. Furthermore, the size of the particles used (approximately 3 μm in diameter) is small enough to hardly interfere with the electric field perfectly shaped by the operating potential. Moreover, the timescale for achieving the desired state makes it difficult to achieve the goal of high throughput using this approach.

[0010] Other self-assembly control approaches, such as those described by Grzelczak et al., "Directed self-assembly of nanoparticles," ACS Nano, 4(7):3591~3605, 2010, PMID:20568710; Paulson et al., "Control of self-assembly in micro-and nano-scale systems," Journal of Process Control, 27:38~49, 2015; Mastrangeli et al., "Automated real-time control of fluidic self-assembly of microparticles," Robotics and Automation (ICRA), 2014 IEEE International Conference, pp. 5860~5865, May 2014; and Paulson et al., "Control of self-assembly in micro-and nano-scale systems," Journal of Process Control, 27:38~49, 2015, do not incorporate these disclosures by reference and do not enable them to easily scale the number of objects being moved.

[0011] The water-based solution in which the particles are immersed is a common choice of control medium, as described by Edman et al., "Electric field directed assembly of an ingaas led onto silicon circuitry. IEEE Photonics Technology Letters, 12(9):1198 - 1200, September 2000, and Tolley et al., "Dynamically programmable fluidic assembly", Applied Physics Letters, 93(25), 2008, the disclosures of which are incorporated by reference. In such cases, both the electrophoretic force and the electroosmotic flow of fluid motion are used to drive the particles. However, since water does not behave like a dielectric, it cannot generate an electric field, and the force to move an object decreases, thus significantly limiting the size of the objects that can be moved using this setup.

[0012] The precise control of electroosmosis-based cells, quantum dots, and nanowires has been used in Mathai et al., "Simultaneous positioning and orientation of single nano-wires using flow control", RSC Adv., 3:2677 - 2682, 2013, and Probst et al., "Flow control of small objects on chip: Manipulating live cells, quantum dots, and nanowires", IEEE Control Systems, 32(2):26 - 53, April 2012, the disclosures of which are incorporated by reference. The authors used a linear model of the electrode potential and ignored the particle effect on the electric field distribution. However, the presented linearity in the electrode potential is not maintained when the driving force is mainly dielectrophoresis, thus limiting the applicability of these techniques. Furthermore, the presented linearity may not hold with objects at the microscale.

[0013] Finally, Zemanek et al., "Feedback-controlled dielectrophoretic micromanipulation", 2018 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS), July 2018, the disclosure of which is incorporated by reference, describes dielectrophoresis-based feedback for microsphere manipulation. The authors use a simulated annealing approach to solve the optimal control problem and utilize the spherical shape of the particles when constructing the system model. However, since this technique depends on the spherical shape of the moving particles, it limits the applicability of the technique.

[0014] Therefore, there is a need for an easily extensible and computationally efficient approach for controlling the density distribution of micro-objects. SUMMARY OF THE INVENTION

[0015] A system and method are provided that enable the formation of a micro-object density distribution (how densely micro-objects are assembled in a particular spatial region). A high-speed camera tracks the existing object density distribution. An array of phototransistor control electrodes is used to generate a dynamic potential energy landscape for manipulating the objects with both DEP and EP forces, and a video projector is used to activate the array. One or more computing devices are used to process the images captured by the camera, estimate the existing density distribution of the objects, receive the desired density distribution of the micro-objects, define a model that describes the variation of the micro-object density over time due to capacitance-based interactions, generate an array of electrode potentials that minimizes the error between the existing density distribution and the desired density distribution when generated, and map the array of electrode potentials to the image projected onto the array using the video projector. The projected image activates or deactivates the electrodes as indicated by the electrode potentials. Machine learning algorithms such as automatic differentiation are used to increase the efficiency of optimizing the array of electrode potentials.

[0016] In one embodiment, a system and method for micro-object density distribution control with the help of a digital computer are provided. One or more parameters of the system for positioning a plurality of micro-objects are obtained by one or more processors configured to execute computer-executable code, the system includes a plurality of programmable electrodes, the electrodes configured to induce the movement of micro-objects when a micro-object approaches an electrode when one or more of the electrodes generate one or more potentials. A model describing the variation in micro-object density over time, based on capacitance-based interactions, is defined by one or more of the processors based on the system parameters. The density distribution of micro-objects is estimated by one or more of the processors using at least one sensor. A target density distribution of micro-objects is received by one or more of the processors. Model predictive control (MPC) optimization is performed by one or more of the processors to minimize the error between the estimated density distribution and the target density distribution by deriving an array of electrode potentials generated by at least some of the electrodes to move at least some of the micro-objects, based on the density model. At least some of the electrodes are actuated by one or more of the processors to generate an array of electrode potentials.

[0017] Further embodiments of the present invention will be readily apparent to those skilled in the art from the following detailed description, and embodiments of the present invention will be described by illustrating the best mode intended for carrying out the invention. As will be understood, other different embodiments of the present invention are possible, and some of its details can be modified in various obvious ways, all without departing from the spirit and scope of the invention. Accordingly, the drawings and detailed description should be considered as illustrative and not restrictive. [Brief explanation of the drawing]

[0018] [Figure 1]This figure illustrates a control target in which micro-objects initially have a uniform distribution, and then converge to a multivariate Gaussian distribution centered on zero and a given covariance matrix. [Figure 2] This is a block diagram showing a system for machine learning-enabled micro-object density distribution control with the help of a digital computer. [Figure 3] This figure illustrates the interaction between the video projector and the phototransistor array shown in Figure 2, according to one embodiment. [Figure 4] This diagram shows individual grids superimposed on a continuous potential function, with the center of each grid representing an electrode. [Figure 5] This figure illustrates a capacitive-based interaction between one chiplet and five electrodes in the system shown in Figure 2, according to one embodiment. [Figure 6] This figure shows a COMSOL model of two conductors (spherical tiplet and electrode) for capacitance calculation according to one embodiment. [Figure 7] The capacitance function fitted with training data generated by COMSOL simulation according to one embodiment is shown. [Figure 8] This figure shows a 2D capacitance function determined from a 1D model using symmetry characteristics according to one embodiment. [Figure 9] This flowchart illustrates a method for machine learning-enabled micro-object density distribution control with the help of digital computers. [Figure 10] This flowchart shows a routine for setting up a dynamic model that governs the variation in chiplet density over time for use in the method shown in Figure 9, according to one embodiment. [Figure 11A] This figure shows the chiplet density distribution for the first example at different time periods. [Figure 11B] This figure shows the chiplet density distribution for the first example at different time periods. [Figure 11C] This figure shows the chiplet density distribution for the first example at different time periods. [Figure 11D]This figure shows the chiplet density distribution for the first example at different time periods. [Figure 11E] This figure shows the chiplet density distribution for the first example at different time periods. [Figure 11F] This figure shows the chiplet density distribution for the first example at different time periods. [Figure 12A] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 12B] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 12C] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 12D] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 12E] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 12F] Figures 11A to 11F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 13A] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 13B] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 13C] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 13D] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 13E] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 13F]These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 11A to 11F. [Figure 14A] This figure shows the chiplet density distribution for the second example at different time points. [Figure 14B] This figure shows the chiplet density distribution for the second example at different time points. [Figure 14C] This figure shows the chiplet density distribution for the second example at different time points. [Figure 14D] This figure shows the chiplet density distribution for the second example at different time points. [Figure 14E] This figure shows the chiplet density distribution for the second example at different time points. [Figure 14F] This figure shows the chiplet density distribution for the second example at different time points. [Figure 15A] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 15B] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 15C] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 15D] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 15E] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 15F] Figures 14A to 14F show the evolution of electrode potential over time to create the chiplet density distribution. [Figure 16A] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Figure 16B] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Figure 16C]These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Figure 16D] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Figure 16E] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Figure 16F] These figures illustrate the force vector fields over time used to create the chiplet density distributions shown in Figures 14A to 14F. [Modes for carrying out the invention]

[0019] Simultaneous control and tracking of individual chiplets is computationally expensive. The computational cost of moving multiple micro-objects simultaneously can be reduced by first performing micro-object control at the macro level, i.e., by tracking and controlling the micro-object density and changing the electrode potential to shape the density to a desired form. An example of a control objective is illustrated in the graph with reference to Figure 1. Figure 1 illustrates a control objective in which the micro-objects initially have a uniform distribution and then converge to a multivariate Gaussian distribution centered on zero and a given covariance matrix. Such a control objective can be achieved using the system and method described below, which utilizes an array of electrodes that generate an electric field that moves the micro-objects. First, a macro-level mathematical model is generated based on the physical laws governing the behavior of micro-objects under an electric field in order to design the array of electrode potentials generated by the electrodes necessary to achieve the desired density distribution. This macro-model explains the evolution of the micro-object density f(x, t) as a result of changing the electric field by the electrode potential. The objective of density control is to bring the micro-object density to a desired pattern f d The goal is to form a (x, t) (e.g., Gaussian) function. The loss function is the Kullback-Leibler divergence between the current microobject density and the desired microobject density over several periods in a Model Predictive Control (MPC) problem.

[0020] Figure 2 is a block diagram showing a system 10 for machine learning-enabled micro-object density distribution control with the help of a digital computer. System 10 enables the cooperative assembly of multiple objects 11, such as micro-objects. The size of the objects 11 varies between nanoscale (less than 1 μm) and microscale (hundreds of μm), but other sizes are possible. The objects 11 can be spherical, but other shapes such as rectangles are also possible, and yet other shapes are also possible. In one embodiment, the diameter of a spherical object 11 can be 20 μm to 50 μm, but other diameters are also possible. In one embodiment, the dimensions of a rectangular object 11 can be 200 μm × 300 μm × 100 μm, but other dimensions are also possible. The objects 11 are immersed in a dielectric fluid (not shown) contained within a casing (not shown), and a layer of film 38 is beneath the dielectric fluid contained within the casing. In one embodiment, the dielectric fluid (not shown) is Isopar® manufactured by ExxonMobil Chemical Company in Spring, Texas, but other dielectric fluids are also possible. The dielectric fluid may contain one or more additives, such as di-2-ethylhexyl sulfosuccinate (AOT) charge-inducing molecules, but other additives are also possible. Object 11 can be made from aluminum oxide (AlOx), but other materials are also possible. Each of the objects 11 may be a semiconductor chip, integrated circuit, particle, nanodevice or structure, or any object that the array 35 of electrodes 12 can physically manipulate. In the following description, object 11 will be interchangeably referred to as chiplet 11 or microobject 11, but other ways of referring to object 11 are possible. In one embodiment, the film 38 may be a 50 μm thick perfluoroalkoxy (PFA) film, but other types of films of other thicknesses are also possible.

[0021] Below the suspended chiplet 11 are a plurality of electrodes 12 forming an array 35, the electrodes configured to generate a dynamic potential energy landscape for manipulating objects with both dielectrophoretic ("DEP") and electrophoretic ("EP") forces. The film is laminated on the electrodes 12. In one embodiment, the electrodes may be square in shape and made of copper, but other shapes and materials are also possible. In one embodiment, the dimensions of the square electrode 12 may be 16 μm wide and 100 nm thick, but in further embodiments, other dimensions are also possible. The array 35 may include a plurality of rows of electrodes 12, each row containing a plurality of electrodes 12.

[0022] The potential generated by electrode 12 is controlled by an array of phototransistors 13, the array including an active-matrix phototransistor backplane set on glass. Multiple phototransistors on the backplane form the array 13, and each phototransistor in the array 13 controls the potential generated by a single electrode 12. In particular, each phototransistor in the array 13 is attached to one electrode 12. The array of phototransistors 13 may have additional properties, such as those described in Rupp et al., "Chiplet micro-assembly printer," 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), pp. 1312-1315, May 2019, which are incorporated by reference.

[0023] The array 13 is optically addressed by a video projector 14 to enable simultaneous dynamic control of the electrostatic energy potential and manipulation of the positions of multiple chiplets 11. In particular, the video projector 14 is used to address each phototransistor control electrode 12, as illustrated in Figure 3, allowing for easy changes in zoom ratio and stitching to larger arrays. Figure 3 illustrates the interaction between the video projector 14 and the phototransistor array 13 of Figure 2 according to one embodiment. The video projector 14 acts on the electrodes 12 by projecting a predefined pattern, i.e., an image 37 generated based on the control input. That is, as will be further explained below, the potential is generated by the electrodes to achieve the desired movement of the chiplets 11. The pixelated light projected by the projector 14 that constitutes the image charges the individual phototransistors 13 in the array to the required extent, causing the electrodes 12 charged by those phototransistors to generate the desired potential. Additional information relating to the phototransistor array 13 is described in U.S. Patent Application No. 17 / 098,816 to Matei et al., filed November 16, 2020, entitled "System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer," the disclosures of which are incorporated by reference.

[0024] In a further embodiment, the electrode array 35 of the electrode 12 may be programmable to generate the desired potential array by means other than through the user of the phototransistor array 13, for example, through the use of a memory-based array in which the electrodes are operated via specific memory addresses.

[0025] Returning to Figure 2, the system 10 further includes a high-speed camera 15 which can be used to track the position of the moving chiplets 11 and to estimate the density distribution of the chiplets 11. In further embodiments, other types of sensors can be used to estimate and track the position of the chiplets 11 and the chiplet density distribution. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16 which can control the electrodes 12 via the projector 14 to induce the formation of a desired chiplet density distribution. Other chiplet 11 patterns are also possible.

[0026] The connection between one or more computing devices 16 may be direct, such as a wired or wireless connection, to physically close the computing devices 16, or the connection may be via an internetwork, such as via the internet or a cellular network, to physically distance the computing devices 16. One or more computing devices 16 include multiple computer processors specialized for performing data processing in parallel. In one embodiment, the computer processors may be graphics processing units (GPUs). In a further embodiment, the computer processors may be tensor processing units (TPUs) developed by Google, LLC in Mountain View, California, which are particularly specialized for neural network machine learning. In yet another embodiment, the computer processors may include both GPUs and TPUs. In yet another embodiment, the computer processors may include other types of processors specialized for parallel processing.

[0027] As will be further explained below, in detail, one or more computing devices 16 execute optimization-based machine-learning (ML) algorithms using mathematical macro models and cost functions to compute the chiplet density f(x, t) and a continuous function V(x, t) that assigns electrode potentials, where x is the 2D position and t is time. The computing devices 16 form the electric field by the discrete geometric shape of the electrodes (e.g., a grid structure). Given the discretization of the continuous plane (i.e., the grid of electrodes), the computing devices 16 can compute the potentials of discrete points by integrating the potential field near the discrete points based on the discretization scheme, using an example of such discretization imposed on the continuous potential function V(x, t) as seen with reference to Figure 4. Figure 4 is a diagram showing individual grids superimposed on the continuous potential function, with the center of each grid representing electrode 12. The grayscale portion of Figure 4 points to low electrode potential values, and the darker the grayscale color of a portion of Figure 4, the lower the magnitude of the potential in that portion of Figure 4. The dotted areas in Figure 4 indicate high potential values, and the darker the dots in a part of Figure 4, the greater the potential in that part of Figure 4. The electric field E(x, t) = ∇V(x, t) can be calculated using automatic differentiation. Then, point x i Potential at the center of the grid rectangle X i teeth,

[0028]

number

[0029] While one or more computing devices 16 are shown as servers, other types of computer devices are possible. The computing device 16 may include one or more modules for performing the embodiments disclosed herein. A module may be implemented as a computer program or procedure written as source code in a conventional programming language and presented as object or bytecode for execution by a processor. Alternatively, a module may also be implemented as an integrated circuit or in hardware burned into a read-only memory component, and each of the computing devices 16 may function as a specialized computer. For example, if a module is implemented as hardware, that particular hardware is specialized to perform the calculations and communications described above, and no other computer can perform them. Furthermore, if a module is burned into a read-only memory component, the computer storing the read-only memory is specialized to perform the operations described above that no other computer can. Various implementations of source code, as well as object code and bytecode, can be stored on computer-readable storage media such as floppy disks, hard drives, digital video disks (DVDs), random access memory (RAM), read-only memory (ROM), and similar storage media. Other types of modules and module functions, as well as other physical hardware components, are possible. For example, computing device 16 may include other components found in programmable computing devices, such as input / output ports, network interfaces, and non-volatile storage devices, but other components are possible. In embodiments where computing device 16 is a server, the server may also be cloud-based or a dedicated server.

[0030] One or more computing devices 16 interface to a storage device 17 and run a capacitance modeler 18 that acquires parameters 19 of the system 10, stores the parameters 19 in the storage device 17, and models the capacitance between the electrode 12 and the chiplet 11, as well as between the chiplet 11 itself. The parameters 19 may include the diameter of the chiplet 11, the dimensions of the electrode 12, the dielectric constant (e.g., ε=2), the fixed position of the electrode 12, the materials of the chiplet 11 and the electrode 12, and the vertical distance between the chiplet 11 and the electrode 12 (the "height" of the chiplet 11). Other parameters 19 are still possible.

[0031] Capacitance modeler 18 creates a capacitance-based dynamic model 20 for chiplet density. Capacitance modeler 18 starts with model 31 induced by a discrete (e.g., grid) electrode array 35. Next, capacitance modeler 18 changes from this discrete representation of the potential to model 32 using a continuous one, assuming that the potential is continuous. The final step is to convert the continuous single-chiplet model 32 to the chiplet density dynamic model 20 by mean-field approximation theory.

[0032] A 2D model 31 for the motion of the chiplets 11 under the influence of the potential field induced by the electrode array 12 is described below. The model considers only one chiplet 11 and omits possible interactions when the chiplets 11 approach each other (the exact distance at which one chiplet 11 must be separated from another chiplet 11 so as not to cause interactions that need to be considered, as these are determined by the shape and potential of the microobjects; in general, for a chiplet-to-chiplet interaction to occur, the distance between the two chiplets must be of the same order as the distance from the chiplet 11 to the electrode 12).

[0033] The result of applying a potential to electrode 12 is the generation of a DEP force acting on the chiplet 11. The viscous drag force proportional to the velocity opposes the movement of the chiplet when the drag force is proportional to the velocity of the non-turbulent flow, i.e., when the Reynolds number is small. Since the mass of the chiplet can be ignored, the acceleration is ignored. Therefore, the dynamic model for the movement of the chiplet 11 can be described as follows.

[0034] [Number] where (x, y) represents the chiplet position measured at the center of the chiplet mass, μ is the fluid-dependent viscosity coefficient, and F x (x, y) and F y (x, y) are the projections of the forces induced by the potential field on the x and y axes, respectively. The forces F x and F y . The capacitance model 18 calculates the potential energy by using a capacitive-based electrical circuit that lumps the interaction between the electrodes and the chiplet. Such a circuit is shown in reference to FIG. 5, and only one row having five electrodes 12 of the array 35 is depicted. FIG. 5 is a diagram illustrating the capacitive-based interaction between one chiplet 11 and five electrodes of the system of FIG. 2 according to an embodiment. The chiplet 11 and the electrodes 12 act as metal plates. Therefore, the capacitance of these capacitors depends on the chiplet position. As expected, the maximum value is achieved when the position of the chiplet 11 maximizes the overlap with the electrode 12. To simplify the analysis, the analysis is limited to only the low-frequency region, and the dielectric constant is not frequency-dependent.

[0035] The forces F x and F y are

[0036] [Number] It can be formally expressed as follows, where U(x, y) is the potential energy of the tiplet 11, and is given by the following:

[0037]

number

[0038]

number

[0039] The feedback control design requires an explicit representation of the capacitance between the chiplet 11 and the electrode 12. The capacitance modeler 18 creates a chiplet motion model using high-fidelity simulation with COMSOL Multiphysics® software developed by COMSOL, Inc. (“COMSOL”) in Burlington, Massachusetts, although other types of simulations using other types of software are also possible. For symmetric chiplets (e.g., beads) where the electrodes are also assumed to be symmetric, the capacitance is estimated by simulating a two-dimensional electrostatic COMSOL model. This assumes the capacitance function is C i (η) = C(∥η-η) i ||) This means that it is of a certain form, and in the formula, η T =[x, y] indicates the chiplet position.

[0040]

number

[0041] Figure 6 shows a COMSOL model of two conductors (spherical chiplets and electrodes) for capacitance calculation according to one embodiment. As shown with reference to Figure 6, in the COMSOL model, a copper plate with a width of 16 μm and a thickness of 100 nm, and a spherical aluminum oxide (AlOx) object with a diameter of 10 μm are surrounded by a dielectric having the same properties as the isopar-M solution.

[0042] The quasi-static model is calculated by the capacitance modeler 18 in the form of an electromagnetic simulation using partial differential equations, with the ground boundary (zero potential) as the boundary condition. The capacitance matrix components are calculated from the resulting charge in each conductor when a potential is applied to one of them and the other is set to ground. The COMSOL simulation reflects the field distortion as the tiplet 11 approaches the electrode 12. The COMSOL electrostatic model has as parameters the diameter of the sphere, the electrode dimensions, the dielectric fluid constant (ε=2), the position and material of the sphere and electrode. The height of the tiplet 11 is fixed (z=5μm), and the position of the tiplet 11 on the x-axis is in the interval [-1mm, 1mm]. Note that the fringe effect (distortion of the electric field at the edge) is significant due to the size of the tiplet 11 versus the size of the electrode 12. Offline simulations are performed for a range of tiplet positions, i.e., x∈[-1mm, 1mm] and z=5μm. The capacitance between the electrode and the tiplet 11 is evaluated for all considered positions. The capacitance function is the error function:

[0043]

number

[0044]

number

[0045] Figure 7 shows the capacitance function fitted with training data generated by a COMSOL simulation according to one embodiment. Figure 7 depicts the capacitance between the chiplet and the electrode as a function of the horizontal position of the chiplet, C(ξ), and the numerical values ​​are fitted to the error function parameterization. The capacitance modeler 18 transforms

[0046]

number

[0047]

number

[0048] The underlying intuition that progresses from Model 31 to the discrete representation of Model 32, which has a continuous representation, is the equation for the potential of the chiplet 11.

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054]

number

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059] The chiplet positions are shown by x=(x1, x2),

[0060]

number

[0061]

number

[0062]

number

[0063]

number

[0064]

number

[0065] Set of chiplets x i (t) is considered. Here, each chiplet 11 is (5), i.e.,

[0066]

number

[0067]

number

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072] By expanding the time derivative on the left-hand side of the previous equation, we obtain the following:

[0073]

number

[0074]

number

[0075] subsequence

[0076]

number

[0077]

number

[0078] The PDE preceding equation (7) is set by the capacitive modeler 18 as a dynamic model 20 for controlling the chiplet density.

[0079] One or more computing devices 16 further perform a position tracker 36 that analyzes the image 21 (or data from another sensor) captured by the high-speed camera 15 to estimate the existing (initial) density distribution 22 of the chiplets 11. That is, it estimates how densely the chiplets 11 are distributed across various spatial regions within the housing. The position tracker 36 further receives a desired target density distribution 23 (e.g., from the user). That is, it receives how densely the chiplets 11 should be distributed across different spatial regions within the space within the disclosure. The current chiplet density distribution 22 and target density distribution 20 are used by an MPC optimizer 25, which is performed by one or more of the computing devices 16 to determine a control scheme 24 including control inputs 27, and the array of potentials that need to be generated by the electrodes 12 guides the chiplets 11 to form the desired density. The decision is made using Model Predictive Control ("MPC") optimization by an MPC optimizer 25 executed by one or more computing devices 16, which is described, for example, by Camponogara et al., "Distributed model predictive control," Control Systems, IEEE, 22(1):44-52, February 2002, and by Garcia et al., "Model predictive control: Theory and practice—A survey," Automatica, 25(3):335-348, 1989, these disclosures are incorporated by reference, and MPC proceeds to an approach for controlling nonlinear systems, based on solving constrained nonlinear programs.

[0080] One or more computing devices use feedback control to set the current density function (initial density function that describes the current density distribution 22 of the chiplet 11) f0(x) = f(x, 0) to the target density function f d The MPC optimizer 25 is executed to form (x), and the target density function describes the target distribution 23 of the chiplet 11. The error between the current density function and the target density function is,

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086] To calculate the potential function, the MPC optimizer 25 uses a multivariable, Gaussian conditional distribution, i.e.,

[0087]

number

[0088]

number

[0089]

number

[0090]

number

[0091]

number

[0092] The final step in the effort to arrive at a more manageable optimization formula is to parameterize both the solution f(x, t) of the PDE in equation (7) and the potential assignment map V(x, t). That is, two smoothing functions f(x, t) = f(t, x; θ) f ) and V(x, t) = V(x, t; θ v ) is the combined parameterization

[0093]

number

[0094]

number

[0095] The MPC optimizer can solve the optimization surrogate for equation (12), where the PDE constraint and initial boundary conditions are treated as weighted cost functions. In particular, the following three loss functions are defined:

[0096]

number

[0097]

number

[0098] The box type of constraint 26 considers the function V(x, t; β). That is, V = [V min , V max The ] and potential assignment functions are defined as follows:

[0099]

number

[0100]

number

[0101]

number

[0102] While solving the optimization problem of equation (16), we need to evaluate the four derivatives / gradients:

[0103]

number

[0104] The control input 27 generated by the MPC optimizer is used by a projector controller 28, which is run by one or more computing devices 16, to map the control input 27 to an image 37 provided to a video projector 14, and then project the image 37 onto a phototransistor that controls the electrode 11, thereby activating the electrode 12 and inducing the movement of the chiplet 11. The mapping can be performed as described in U.S. Patent Application Publication No. 2020 / 0207617 to Plochowietz et al., entitled "Micro-assembler system for manipulating micro-objects," published on July 2, 2020.

[0105] In a further embodiment, to enhance processing scalability, the potential assignment map required to achieve the desired chiplet density can be derived as a function of the current chiplet distribution, instead of using MPC optimization.

[0106] Once the desired density distribution is achieved, the individual particles can be moved using the system, as described in U.S. Patent Application No. 17 / 121,411 to Matei et al., filed December 14, 2020, entitled "System and Method for Machine-Learning Enabled Micro-Assembly Control With the Aid of a Digital Computer," the disclosure of which is incorporated by reference.

[0107] Using model predictive control, a desired chiplet distribution pattern can be achieved in a computationally efficient manner by computing an array of potential assignments that form the chiplet distribution into a desired pattern. Furthermore, the use of Gauss-Hermitian quadrature to accurately approximate the expected values ​​and automatic differentiation for computing the derivatives of physical quantities (e.g., potential energy, chiplet distribution) and the gradient of the KL loss function provides additional ease of use to the approach. Figure 9 is a flowchart showing Method 50 for machine learning-enabled micro-object density distribution control with the help of a digital computer. Method 50 can be performed using the system in Figure 2. System parameters, such as the diameter of the chiplet, the dimensions of the electrodes, the dielectric constant, the initial chiplet density distribution (which can be obtained using a high-speed camera), and the materials of the chiplet and electrodes, as well as the height of the chiplet, are obtained, but other parameters are possible (Step 51). A dynamic model governing the temporal variation of the chiplet density is set up, as further described below with reference to Figure 10 and as described above with reference to Figure 1 (Step 52). The target density distribution of the chiplets is received (Step 53). Referring to Figure 1 and as further described above, MPC optimization is performed to derive the control input, and an array of electrode potentials is obtained that minimizes the difference between the current chiplet density distribution and the target chiplet density distribution (step 54). The control input is mapped to an image for display by a video projector (step 55). The electrodes in the electrode array are controlled to guide the movement of the chiplets to the desired positions (step 56). If the desired density distribution is achieved (as can be observed using camera 15), no further chiplet movement is required, and method 50 terminates. If the target density distribution is not achieved and the chiplets still need to be moved (step 57), method 50 returns to step 54.

[0108] As described above, in further embodiments, the density distribution can be estimated using at least one sensor other than a high-speed camera. Similarly, in further embodiments, other methods are possible that activate programmable electrodes in addition to the phototransistor array. In these further embodiments, method 50 is modified accordingly.

[0109] A capacitance-based model for dynamic control over chiplet density can be derived through the use of a mean-field approximation. Figure 10 is a flowchart showing a routine 60 for setting up a dynamic model governing the variation in chiplet density over time for use in method 50 of Figure 9, according to one embodiment. Simulations, such as COMSOL simulations, are performed for the capacitance between the electrode and the chiplet at multiple distances, as described above with reference to Figure 1 (step 61). A discrete representation of the model for moving one of the chiplets based on the capacitance between the chiplet and the electrode is defined based on the simulations, as described above with reference to Figure 1 (step 62). The discrete representation of the model is converted to a continuous representation, as described above with reference to Figure 1 (step 63). A mean-field approximation is applied to the continuous representation of the model for dynamic control over chiplet density (step 64), and routine 60 is completed.

[0110] For illustrative purposes only, not limiting, two experimental examples of the use of the above system and method to manipulate the chiplet density distribution are provided. In the examples, normalized representations of the chiplet position and potential quantities on the electrode grid are given. These are similar to their physical counterparts beyond the scaling factor. A 3x3 grid was used, and the sampling period was 0.025. The chiplet capacitance was given by mean and chiplet position x and covariance matrix σ. 2The PDE solution f(x, t) and the potential assignment function V(x, t) are multivariate, Gaussian distributions, where σ = 0.05. A time range of 5 seconds is considered, with a sampling period of 0.1 seconds. The PDE solution f(x, t) and the potential assignment function V(x, t) were modeled as a neural network with one hidden layer of size 30 and tanh as the activation function. Each function has an input of size 3 and an output of size 1. The step size of the Adam optimization algorithm was 0.001. The number of epochs varied based on the quality of the solution. Using 30 points, the expected values ​​of equations (10) and (11) were approximated by using Gauss-Hermitian quadrature (where the term "n" is used in these equations). This results in a total of 30 × 30 points.

[0111] In the first example of the change in chiplet density distribution achieved using the system in Figure 2 and the method in Figure 9, the objective is to generate an array of potential assignments that start with a "flat" distribution of chiplets 11 and concentrate chiplets near the origin. The initial distribution is σ=2, with zero mean and covariance matrix σ 2 This is a two-dimensional Gaussian distribution with I. Figures 11A to 11F show the chiplet density distribution of the first embodiment at different time points. The chiplet distribution starts at t=0 seconds and has a nearly flat distribution, achieving the desired distribution at t=5 seconds. Figures 12A to 12F show the evolution of the electrode potential over time to create the chiplet density distribution in Figures 11A to 11F. The grayscale areas in Figures 12A to 12F indicate low electrode potential values; the darker the grayscale color in a part of the figure, the lower the potential magnitude in that part of the figure. The dotted areas in Figures 12A to 12F indicate high potential values; the denser the dots in a part of the figure, the greater the potential value in that part of the figure. In Figures 12A to 12F, the x and y axes are the first (x1) and second (x2) axes. The evolution of the force vector field is shown with reference to Figures 13A to 13F. Figures 13A to 13F illustrate the force vector fields over time used to create the chiplet density distributions in Figures 11A to 11F. In Figures 13A to 13F, the arrows at the points indicate the direction in which the chiplet is being pushed at that point, and the color intensity is proportional to the magnitude of the force.

[0112] In a second example of the change in chiplet density distribution achieved using the system in Figure 2 and the method in Figure 9, the chiplets were split into two piles. The example uses a zero-mean, Gaussian-distributed covariance matrix σ 2 Starting from I, with σ=2, two distinct piles were created. The final distribution is a mixture of two Gaussian distributions, and the two distributions are, respectively,

[0113]

number

[0114] While the present invention has been shown and described in particular with reference to its embodiments, those skilled in the art will understand that the aforementioned and other modifications of its forms and details can be made therein without departing from the spirit and scope of the invention.

Claims

1. A system for controlling the micro-object density distribution with the help of a digital computer, A system comprising one or more processors configured to execute computer-executable code, wherein the processors To obtain one or more parameters of a system for positioning multiple micro-objects, wherein the system includes multiple programmable electrodes, the electrodes are configured to induce the movement of the micro-objects when they approach the electrodes when one or more of the electrodes generate one or more potentials, To define a model that describes the changes in micro-object density over time caused by capacitance-based interactions based on the parameters of the system, To estimate the density distribution of the micro-objects using at least one sensor, The process involves receiving the target density distribution of the aforementioned micro-objects, In order to minimize the error between the estimated density distribution and the target density distribution, model predictive control (MPC) optimization is performed to derive, based on the density model, the array of electrode potentials generated by at least some of the electrodes in order to move at least some of the micro-objects. A system configured to activate at least some of the electrodes to generate the array of electrode potentials.

2. Define a discrete representation of a model for moving one of the micro-objects based on the aforementioned parameters, Converting the discrete representation of the aforementioned moving model into a continuous representation, The system according to claim 1, further comprising applying a mean-field approximation to the continuous representation of the model to obtain the density model.

3. Performing multiple simulations of the capacitance between the electrode and one of the micro-objects, The system according to claim 2, further comprising defining a function included in the movement model that describes the capacitance between each of the micro-objects and the electrodes as a function of the distance between the micro-objects and the electrodes.

4. The system according to claim 3, wherein the movement model describes the movement in at least one of one-dimensional and two-dimensional dimensions.

5. The system according to claim 1, wherein the error between the estimated density distribution and the target density distribution is expressed using Kullback-Leibler divergence.

6. The system according to claim 1, wherein performing the MPC optimization includes performing automatic differentiation to calculate multiple gradients.

7. The system according to claim 1, wherein performing the MPC optimization includes evaluating at least one expected value using the Gauss-Hermitian quadrature method.

8. The system according to claim 1, wherein the processor includes at least one of one or more graphics processing units (GPUs) and one or more tensor processing units (TPUs).

9. The system according to claim 8, wherein two or more of the processors operate in parallel to perform the MPC optimization.

10. The electrode is controlled by a phototransistor, and one or more processors Mapping the electrode potentials within the aforementioned array to multiple images, The system according to claim 1, wherein a video projector is controlled to project the image onto the phototransistor, the phototransistor is further configured to control the electrodes to generate the array of electrode potentials based on the projected image.

11. A method for controlling the micro-object density distribution with the help of a digital computer, Acquiring one or more parameters of a system for positioning multiple micro-objects by one or more processors configured to execute computer-executable code, wherein the system includes multiple programmable electrodes, the electrodes configured to induce the movement of the micro-objects when the micro-objects approach the electrodes when one or more of the electrodes generate one or more potentials, One or more of the aforementioned processors define a model that describes the changes in the micro-object density over time caused by capacitance-based interactions based on the parameters of the system, One or more of the aforementioned processors estimate the density distribution of the micro-objects using at least one sensor, One or more of the aforementioned processors receive the target density distribution of the micro-objects, One or more of the processors perform model predictive control (MPC) optimization to derive, based on the density model, an array of electrode potentials generated by at least some of the electrodes to move at least some of the micro-objects, in order to minimize the error between the estimated density distribution and the target density distribution. A method comprising: using one or more of the processors to activate at least some of the electrodes to generate the array of electrode potentials.

12. Define a discrete representation of a model for moving one of the micro-objects based on the aforementioned parameters, Converting the discrete representation of the aforementioned moving model into a continuous representation, The method according to claim 11, further comprising applying a mean-field approximation to the continuous representation of the model to obtain the density model.

13. Performing multiple simulations of the capacitance between the electrode and one of the micro-objects, The method according to claim 12, further comprising defining a function included in the movement model that describes the capacitance between each of the micro-objects and the electrodes as a function of the distance between the micro-objects and the electrodes.

14. The method according to claim 13, wherein the movement model describes the movement in at least one of one-dimensional and two-dimensional directions.

15. The method according to claim 11, wherein the error between the estimated density distribution and the target density distribution is expressed using Kullback-Leibler divergence.

16. The method according to claim 11, wherein performing the MPC optimization includes performing automatic differentiation to calculate a plurality of gradients.

17. The method according to claim 11, wherein performing the MPC optimization includes evaluating at least one expected value using the Gauss-Hermitian quadrature method.

18. The method according to claim 11, wherein the processor includes at least one of one or more graphics processing units (GPUs) and one or more tensor processing units (TPUs).

19. The method according to claim 18, wherein two or more of the processors operate in parallel to perform the MPC optimization.

20. The electrode is controlled by a phototransistor, and one or more processors Mapping the electrode potentials within the aforementioned array to multiple images, The method according to claim 11, wherein a video projector is controlled to project the image onto the phototransistor, the phototransistor is further configured to control the electrodes to generate the array of electrode potentials based on the projected image.

Images