Systems and methods for machine learning-based position estimation for use in digital computer-based microassembly control

Abstract

To provide a system and method for machine-learning-based position estimation for use in microassembly control with the aid of a digital computer.SOLUTION: Control loop latency is taken into account in predicting positions of micro-objects being moved by using a hybrid model that includes both a physics-based model and machine-learning models. The models are fitted based on residuals calculated during a previous stage using gradient boosting and based on comparison with training data. A loss function for each stage is selected based on the model being created. The hybrid model is evaluated on data extrapolated and interpolated from the training data to prevent overfitting and to have a sufficient predictive ability. By including both the physics-based model and the machine-learning model, the hybrid model can account for both deterministic and stochastic components involved in the movement of the micro-object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application relates generally to micro-assembly control, and more particularly to a system and method for machine learning based position estimation for use in micro-assembly control using a digital computer. [Background technology]

[0002] Microscale and nanoscale particle manipulation has attracted the interest of many researchers. The degree to which we can control the assembly of micro- and nano-objects, which are objects with dimensions on the order of microns and nanometers, can make a significant difference in many technologies, including micromanufacturing, biology, and medicine. For example, the fabrication of reconfigurable electrical circuits can be improved by precisely controlling the position of micro-objects such as capacitors and resistors to fabricate circuits with desired behavior. Similarly, the generation of photovoltaic solar cell arrays can benefit from being able to place photovoltaic cells with specific qualities at specific locations on the array. Such cells are too small to allow for desired placement of the cells via human or robotic manipulation; another type of transport mechanism is required. Micro- and nanoassembly of particles can also be used to manipulate the microstructure of materials, such as biological cells, which assemble into tissues. There are many other technological fields where increased control over the assembly of micro- and nano-objects could provide significant benefits.

[0003] Many techniques used for microparticle and nanoparticle manipulation use electrodes to generate an electrode potential to move a particle of interest in a desired direction. The particle's initial position is determined using a sensor, and an electrode potential is generated based on the desired position and the initial position. However, the throughput and accuracy of such systems are significantly limited by control loop latency, which is the delay between when a particle is at a specific location and when an electrode potential is generated based on the particle's location. The exact length of the delay depends on the software and hardware of the system used to manipulate the particle. The delay has several components. One component is the actuation delay, which is the delay between when a sprite (a control signal for an electrode) is generated and when the electrode generates the electrode potential specified by the sprite. Another component of the delay is the sensing delay, which is the time between the sensor observing a particle occupying a specific location and the time. Analysis of the sensor's signal is completed to obtain the particle's location. Such delays include the time it takes to stream and store the captured image in memory, the time it takes to transmit the image data for image analysis to determine its location, and the time it takes to run the image analysis algorithm. These delays can be as much as 80ms-100ms, and up to 300ms or more, depending on various latencies and processor load, and can take up to five frames at frame rates below 60hz.

[0004] Predicting delays and subsequently predicting the actual locations of micro- or nano-sized particles that need to be moved could improve the accuracy and efficiency of micro- and nanoassembly. Such modeling typically utilizes physics-based models, which are models that utilize known physical properties of the operational setup, the environment (e.g., whether the assembly is in air, liquid, ground, space, or another environment), and the particles being moved. However, physics-based models of closed-loop systems do not take into account dynamic changes in control latencies that occur during micro- or nanoparticle operations, such as particle rotation, target assignment, and transfers that occur for small or large subsets of particles. Such physics-based models also do not take into account dynamic changes between different sets of operational conditions, such as the use of different types of particles or different lighting conditions. Such physics-based models are also not robust when modeling specific variables within a particular operational setup, such as particle stiction or particle oscillating motion, or when modeling parameters with unknown values, such as the average number of pixels a particle transitions through during a control cycle.

[0005] Therefore, what is needed is a method to accurately account for control loop latency to predict the actual position of a particle being moved due to a generated electrode potential. Summary of the Invention

[0006] Control loop latency can be taken into account in predicting the position of a micro-object by using a hybrid model that includes at least one physics-based model and a machine learning model. The models are combined using gradient boosting, and the model created during at least one of the stages is adapted based on residuals calculated during previous stages based on comparison with training data. The loss function for each stage is selected based on the model being created. The hybrid model is evaluated with extrapolated and interpolated data from the training data to prevent overfitting and ensure the hybrid model has sufficient predictive power. By including both a physics-based model and a machine learning model, the hybrid model can consider both deterministic and stochastic components involved in the movement of the micro-object, thereby increasing the accuracy and throughput of microassembly.

[0007] In one embodiment, a system and method for machine learning-enabled microassembly control using a digital computer is provided. One or more parameters of a closed-loop system for positioning one or more micro-objects are acquired, the system including a plurality of programmable electrodes configured to induce movement of the micro-objects when the micro-objects are in proximity to the electrodes upon actuation of the electrodes, each actuation including generation of one or more electrical potentials by one or more of the electrodes. Training data is acquired, the training data including states associated with previous movements of one or more of the micro-objects due to previous ones of the actuations. One or more physics-based models are defined for predicting a position of at least one of the micro-objects after one of the actuations. A hybrid model for predicting a position of at least one of the micro-objects after one of the actuations is constructed using the training data via gradient boosting, the hybrid model including a plurality of models including one or more physics-based models and one or more machine learning models. The position of one or more of the micro-objects after one of the actuations is predicted based on at least one sensor measurement. One or more further positions of the micro-objects are received. Another one of the actuations is performed using the predicted and further positions.

[0008] Still other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, which describes embodiments of the present invention by illustrating the best mode contemplated for carrying out the invention. As will be realized, the present invention is capable of other and different embodiments, and its several details can be modified in various obvious respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature, and not restrictive. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram illustrating a system for machine learning-based position estimation for use in microassembly control using a digital computer. [Figure 2] FIG. 1 is a flow diagram illustrating a method for machine learning-based position estimation for use in microassembly control using a digital computer. [Figure 3] 3 is a flow diagram illustrating a routine for generating training data for use in the method of FIG. 2, according to one embodiment. [Figure 4] FIG. 3 is a flow diagram illustrating a routine for constructing a hybrid model for use in the method of FIG. 2, according to one embodiment. [Figure 5A] 5A and 5B show examples of positions taken by micro-objects in recorded training data (shown with reference to FIG. 5A) and positions of micro-objects extrapolated from the recorded training data (shown with reference to FIG. 5B). [Figure 5B] 5A and 5B show examples of positions taken by micro-objects in recorded training data (shown with reference to FIG. 5A) and positions of micro-objects extrapolated from the recorded training data (shown with reference to FIG. 5B). [Figure 6] FIG. 10 shows the results of testing Estimator 1.0 comparing the use of a hybrid model that includes an ARIMA model as the machine learning model with the use of a hybrid model that includes an RNN model as the machine learning model. [Figure 7] FIG. 10 shows the results of using Estimator 1.0 for ARIMA and RNN models in the extrapolation test dataset. [Figure 8] Figure 1 shows a simulation (approximately 50 minutes) of the entire extrapolation dataset using an unsupervised trained System ID model. [Figure 9-1] Figure 10 shows a simulation of the extrapolated dataset using the unsupervised trained System ID model. [Figure 9-2] Figure 10 shows a simulation of the extrapolated dataset using the unsupervised trained System ID model. [Figure 9-3]Figure 10 shows a simulation of the extrapolated dataset using the unsupervised trained System ID model. [Figure 10A] Figure 1 shows the test estimator 2.0 (hybrid model) and linear, dense, and ARIMA models implemented in microassembler software using frugally-deep (cpp). [Figure 10B] Figure 1 shows the test estimator 2.0 (hybrid model) and linear, dense, and ARIMA models implemented in microassembler software using frugally-deep (cpp). [Figure 11] FIG. 1 illustrates gradient boosting, according to one embodiment. [Figure 12] FIG. 1 is a diagram of the relationship between recorded, extrapolated and interpolated data when defined for a complex hull. DETAILED DESCRIPTION OF THE INVENTION

[0010] The systems and methods described below allow for control loop latency to be taken into account when generating control signals, thereby improving the accuracy of positioning achieved using micro- or nano-assemblies. As described below, the systems and methods utilize a hybrid model for predicting the position of a micro-object after actuation, where the hybrid model includes one or more physics-based models and one or more machine learning models. While physics-based models have the advantages of not requiring any data to build, allowing extrapolation beyond the data used to fit parameters, and high interpretability of these predictions, they can only capture a small portion of the behavior of the system used to implement the micro-assembly. The complete behavior of the system 10 includes unmodeled degrees of freedom, and the use of machine learning models can capture these unmodeled degrees of freedom. The total system behavior, given by Y, is the physical model f(X,θ), where X is the previous history and measurements, θ is the parameter, and the noise N(φ) is the noise parameter of the stochastic noise model. The neural network function NN(X,W) is a function of the input and trainable weights W and captures the unmodeled degrees of freedom according to the following equation:

[0011]

number

[0012] The use of hybrid models allows combining the advantages of physics-based models (interpretability, stability, and extrapolation capabilities) with the flexibility of machine learning models and their ability to capture unmodeled degrees of freedom, resulting in more accurate predictions of micro-object positions and therefore improved throughput of micro-assembly. Physics-based models consider the deterministic components of the system, while machine learning models consider the probabilistic components of the system.

[0013] In the following description, the object being moved is described as a micro-object, but the same techniques can be applied to control the position of nano-objects. Furthermore, while the hybrid model described below is described as being applied to specific actuation hardware, in further embodiments, the hybrid model can be used to predict the position of a micro- (or nano-)object actuated using a different hardware setup. In yet another embodiment, the hybrid model can be used to control other types of systems. For example, the hybrid model can be applied to the control of a motor-driven mechanical linkage with backlash. The physics of the motor and the rotation of the mechanical linkage can be easily modeled using a physical model that includes the physical parameters of the motor and the linkage, such as the moment of inertia, and the series resistance of the windings. Backlash in the linkage is nonlinear, dependent on previous history, and much more complex. Subjecting the motor system to a random series of motions, controlled at various frequencies and amplitudes, and comparing the measured response with the expected response from the physical model yields the error of the physical model as a function of the input and motor state. This data is used for supervised training of a neural network trained to predict backlash error. Neighboring time errors are forced to be uncorrelated, providing an additional constraint against overfitting. The residual errors are fitted to a stochastic model, which in this case is independent and identically distributed Gaussian noise. The noise variance can be determined from the residuals from the physics and neural network models. This hybrid model is used for system identification, i.e., to predict the system's response as various control actions and past and current states of the system's functions are applied. This prediction can be incorporated into various control algorithms, such as model predictive control, to achieve improved performance over simpler previous control algorithms, such as the typical proportional integral derivative (PID) control algorithm. Still other uses of the hybrid model are possible.

[0014] FIG. 1 is a block diagram illustrating a system 10 for machine learning-based position estimation for use in microassembly control using a digital computer. The system 10 enables cooperative assembly involving several objects 11, such as small objects. The size of the objects 11 varies between the nanoscale (<1 μm) and the microscale (1 μm to hundreds of μm), although other sizes are possible. The objects 11 can be spherical, although other shapes, such as rectangular, are also possible. In one embodiment, the diameter of the spherical objects 11 is 5 μm to 50 μm, although other diameters are also possible. In one embodiment, the dimensions of the rectangular objects 11 can be 200 μm x 300 μm x 100 μm, although other dimensions are also possible. The objects 11 are immersed in a dielectric fluid (not shown) contained within an enclosure (not shown), and a layer of film (not shown) underlies the dielectric fluid contained within the enclosure. In one embodiment, the dielectric fluid is Isopar® M manufactured by ExxonMobil Chemical Company of Spring, Texas, although other dielectric fluids 17 are possible. The dielectric fluid may contain one or more additives, such as di-2-ethylhexyl sulfosuccinate (AOT) charge-inducing molecules, although other additives are possible. The objects 11 may be made of aluminum oxide (AlOx), although other materials are possible. Each of the objects 11 may be a semiconductor chip, an integrated circuit, a particle, a nanodevice, or a structure, or any object that can be physically manipulated by an array of electrodes. In the following description, the objects 11 are referred to as micro-objects 11, although other ways of naming the objects 11 are possible.

[0015] Beneath the suspended micro-object 11, a plurality of programmable electrodes 12 form an array 35, the electrodes 12 configured to generate a dynamic potential energy landscape for manipulating the object with both dielectrophoretic ("DEP") and electrophoretic ("EP") forces. A film is deposited on the electrodes 12. In one embodiment, the electrodes may be square in shape and made of copper, although other shapes and materials are also possible. In one embodiment, the dimensions of the square electrodes 12 may be 16 μm wide and 100 nm thick, although other dimensions are possible in further embodiments. The array 35 may include multiple rows of electrodes 12, with each row including multiple electrodes 12.

[0016] The potential generated by the electrodes 12 is controlled by an array 13 of phototransistors, which comprises an active matrix phototransistor backplane set on glass. A number of phototransistors on the backplane form the array 13, with each phototransistor in the array 13 controlling the potential generated by a single electrode 12. In particular, each phototransistor in the array 13 is attached to one electrode 12. The phototransistor array 13 can have additional characteristics, such as those described in Rupp et al., "Chiplet micro-assembly printer," 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), pp. 1312-1315, May 2019, the disclosure of which is incorporated by reference.

[0017] The array 13 is optically addressed by a video projector 14 to enable dynamic control of the electrostatic energy potential and manipulation of the position of multiple micro-objects 11 simultaneously. In particular, a video projector 14 is used to address each phototransistor control electrode 12, allowing for easy zoom factor changes and stitching into larger arrays. The video projector 14 activates the electrodes 12 by projecting a predetermined pattern, an image 37, which is generated based on a control input: the potential the electrode should generate to achieve the desired motion of the micro-objects 11, as explained further below. The pixel light projected by the projector 14 that constitutes the image charges each individual phototransistor 13 in the array to the required degree, so that the electrode 12 generates the desired potential, as its phototransistor charges the electrode 12. In one embodiment, voltages of up to + / - 200V are controlled by these phototransistors to display dynamically changing complex potential topographies. A single voltage signal is synchronized with the optical addressing of the charge storage capacitors on each pixel. Dynamic electric field patterns are generated by refreshing the projected image pattern (control input). Arrays 13 were constructed with 50 μm, 10 μm and even 3 μm pitch, where the pitch is the distance between the centers of the electrodes on which the phototransistors are attached. A dielectric fluid 17 medium (isopar M) and various additives were used to control the charge.

[0018] The system 10 further includes a high-speed camera 15 that tracks the location of the micro-objects 11 being moved. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16, which can control the electrodes 12 via the projector 14 to induce the formation of a desired micro-object 11 pattern. Other micro-object 11 patterns are possible. In further embodiments, other types of sensors (e.g., using capacitive sensing) can be used to measure and track the position of the micro-objects. In yet another embodiment, the electrode array 35 can have integrated active matrix addressing electronics, and the potential on each electrode of the array is electronically updated as the electrode potential required for the generated desired micro-object movement is generated. Additional characteristics of the array 13, electrodes 12, video projector 14, and camera 15 are described in U.S. Patent Published Application No. 2020 / 0207617, published July 2, 2020, by Plochowietz et al., the disclosure of which is incorporated by reference. Additionally, additional components of the system 10 that interact with the array 14, the electrodes 12, the video projector 14, and the camera 15 are possible.

[0019] The connection between one or more computing devices 16 may be direct, such as via a wire or wireless connection, so as to place one or more computing devices 16 in physical proximity, or the connection may be via internetworking, such as via the internet or a cellular network, so as to place one or more computing devices 16 physically remote. One or more computing devices 16 include one or more circuits that perform data processing, such as a computer processor, although other types of circuits for processing data are possible. In one embodiment, the computer processor may be a central processing unit (CPU), a graphics processing unit (GPU), or a mixture of a CPU and a GPU, although other types of processors or mixtures of processors are possible. In one embodiment, two or more of the processors may perform processing in parallel, as further described in U.S. Patent Application No. 17 / 098,816, filed November 16, 2020, entitled "System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer," to Matei et al., the disclosure of which is incorporated by reference. Additional efficiency and improved scalability can be obtained through parallel computing capabilities for tracking large numbers of particles and for voltage pattern generation. In yet another embodiment, circuits other than a CPU and a GPU can perform the above data processing.

[0020] While one or more computing devices 16 are shown as servers, other types of computer devices are possible. The computing devices 16 may include one or more modules for carrying out the embodiments disclosed herein. The modules may be implemented as computer programs or procedures written as source code in a conventional programming language and presented for execution by a processor as object or bytecode. Alternatively, the modules may also be implemented in hardware, such as integrated circuits or burned into read-only memory components, such that each computing device 16 functions as a specialized computer. For example, if the modules are implemented as hardware, that specific hardware is specialized to perform the above-described calculations and communications, and other computers cannot use it. Furthermore, when the modules are burned into read-only memory components, the computer storing the read-only memory is specialized to perform the above-described operations that other computers cannot. Various implementations of the source code, object code, and bytecode may be maintained on computer-readable storage media, such as floppy disks, hard drives, digital video disks (DVDs), random access memory (RAM), read-only memory (ROM), and similar storage media. Other types of modules and module functions, as well as other physical hardware components, are possible. For example, computing device 16 may include other components found in programmable computing devices, such as input / output ports, network interfaces, and non-volatile storage, although other components are possible. In embodiments in which computing device 16 is a server, the server may also be cloud-based or a dedicated server.

[0021] One or more computing devices 16 are interfaced to a storage device 17 and execute a hybrid model generator 18 that acquires parameters 19 of the system 10 and stores the parameters 19 in the storage device 17. The parameters 19 may include the diameter of the micro-object 11, the dimensions of the electrodes 12, the dielectric fluid constant (e.g., ε=2), the fixed position of the electrodes 12, the materials of the micro-object 11 and the electrodes 12, and the vertical distance between the micro-object 11 and the electrodes 12 (the “height” of the micro-object 11). Other parameters 19 are possible. The hybrid model generator 18 also creates a hybrid model 24 for predicting the position of a particular micro-object 11, which may also be stored in the storage device 17. The hybrid model 24 includes a physics-based model 26 that predicts the position of the micro-object 11 after electrode actuation based on the generated control signal (and consequently the potential generated by the electrodes), the velocity of the micro-object 11 at a particular time, and a starting position 28 (position before actuation) based on a micro-object image 29 captured by the camera 15. An example of a physics-based model is described in "Integrating physics-based modeling with machine learning: A survey," arXiv:2003.04919 [physics.stat] (2020) by J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar, the disclosure of which is incorporated by reference, although other physics-based models 26 are possible. The hybrid model 24 further includes one or more machine learning models 27. The machine learning models may include deep models (machine learning models that utilize neural networks made up of multiple layers), linear models, dense models, autoregressive integrated moving average (ARIMA) models, and recurrent neural network (RNN) models, although other types of machine learning models are possible. In one embodiment, the SimpleRNN layer of the ARIMA model may be changed to an LSTM (long short-term memory, RNN architecture), although other types of ARIMA models are possible.As described above, the physics-based model 26 takes into account the deterministic behavior of the system 10, while the machine learning model 27 takes into account the deterministic aspects of the system 10. Thus, the overall behavior of the system 10 can be given by Y as a function of the physical model f(X, θ), where X is recorded data associated with previous operation of the system 10 (hereinafter referred to as training data 22), θ is a parameter, φ is a noise parameter of the stochastic noise model, and the system behavior is further represented by a neural network function NN(X, W) (associated with the machine learning model 27), which is a function of inputs and trainable weights.

[0022] The hybrid model 24 is constructed using training data 21. The training data describes states associated with the motion of the micro-object 11, including (1) input states of the micro-object 11 (but can also include other states such as orientation and angle) that minimize the position and velocity of the micro-object 11 before the control input activates the electrodes; (2) control actions, such as an illumination pattern applied to the phototransistor control electrodes 12 by a video projector 14 (indicating the potential generated by the electrodes) or another force applied to the micro-object 11; and (3) output states resulting from the input data (the position of the micro-object 11 achieved after activation at a specific time and velocity of the micro-object). The training data 22 is used for supervised training of a series of parametric models that constitute the hybrid model 24, as shown in FIG. 11. The "traditional" models shown in FIG. 11 can be physics-based models 24 and, for example, neural network models of Model 2 (shown in FIG. 11) that run on the residuals and fit additional models to the residuals.

[0023] The hybrid model is tested using interpolated data 33 (also referred to as interpolated data 33) and extrapolated data 23 (also referred to as extrapolated data 23), which are used as test data 101, as described further below. The interpolated data 33 may initially be part of the same data set as the training data 22 and includes input states, control actions, and output states whose values ​​are within the range of values ​​of the training data 21. The extrapolated training data 23 may also be part of the same data set as the training data 21 and includes input states, control actions, and output states, but includes values ​​for at least one of the input states, control actions, and output states that are outside the range of values ​​for the input states, control actions, and output states of the training data 21. For example, at least one of the position, velocity, or action applied to the micro object 11 is for a value that extends beyond the value used in the training data 21.

[0024] 5A and 5B are annotated bright-field images of experimental (recorded) data (21), where the micro-object 11 is highlighted with a white circle and the set of intermediate target positions through which the micro-object transitions is highlighted with a magenta square. Figures 5A and 5B show two different experiments with two different sets of target positions. This experimental data 21 can be used as training data 22. Figure 12 is a diagram of the relationship between the training data 22, extrapolated data 23, and interpolated data 33 when defined for a complex hull. Still other types of training data are possible.

[0025] As described further below, beginning with reference to FIG. 2, the hybrid model 24 is created using gradient boosting, which is implemented in a number of stages during which one or more physics-based models are trained by selecting a loss function, the next stage is implemented by selecting another model (machine learning model 27) using the residuals 31 and previous inputs as new inputs and selecting a loss to boost (train) the parameters of the next stage, and the creation of additional models is repeated in this manner multiple times.

[0026] Thus, during gradient boosting, at least one physics-based model 26 is first defined, and then a loss function 30 (also called a whitening objective function 30) is used to analyze the difference between output training data 22 (the position of the micro-object 11 at a particular time after actuation) and one or more predictions made by using the physics-based model 26 regarding the achieved position of the micro-object 11 at the same time after the same actuation based on the input training data as described by the output training data. The result of the analysis is one or more residuals 31 (also known as pseudo-residuals 31), each describing the difference between a position predicted by the physics-based model 26 based on a piece of input training data 22 and a piece of output training data 22 resulting from the piece of input training data 22 on which the prediction was made. Subsequently, during the next stage, a first machine learning model 27 is constructed using the residuals 31 derived using the initial physics-based model 26, and the first machine learning model 27 is adapted based on the generalized residuals 31 of the physics-based model 26. Subsequently, during the next stage, residuals 31 are generated for a first machine learning model based on the training data 22 using the same loss function 30 described above to process the difference between the predicted micro-object location based on the input training data 22 and the location in the output training data 21. The residuals 31 from the first machine learning model 30 are used to define another deep model, and the residuals 31 of the second deep model are calculated in the same manner as described above. The cascade continues in multiple stages by fitting the generalized residuals of one machine learning model 27 to the next until a desired number of models 26, 27 are generated. (In one embodiment, the desired number of models is based on historical data regarding the accuracy of a hybrid model including a certain number of models 26, 27; in further embodiments, the desired number of models 26, 27 is set based on empirical testing of whether a predetermined number of models produces the desired accuracy.) While the above description illustrates the use of only a single physics-based model 26 as part of the hybrid model 24, in further embodiments, multiple physics-based models 26 can be used as part of the hybrid model.One or more physics-based models 26 in the hybrid model 24 function as weak learners, which are models whose predictions are only slightly better than random guesses, and the trained machine learning model 27 functions as a strong learner, with higher predictive accuracy than the weak learners. All predictions from the models 26, 27 are aggregated into a single prediction for the hybrid model 24, such as by assigning predictions from different model weights and averaging the predictions, although other methods for aggregating predictions are possible. For example, predictions from the various model stages 26, 27 can be aggregated into a combined prediction for the hybrid model 24 as a sum or product refinement according to Equation (2) below, where residuals are expressed as factors rather than variances. Different stages can also be assigned different weights. Mathematically, gradient boosting, as performed by the hybrid modeler 18, can be defined as follows:

[0027] A series of M learners is used to refine the predicted output. The weak (or base) model comprises a physics / deterministic model 26 followed by a machine learning model to capture the predictable yet complex parts.

[0028]

number

[0029] where M is the number of repeats,

[0030]

number

[0031]

number

number

[0032] During the ceremony,

[0033]

number

[0034] In the formula, Ψ i (.,.) is the loss function for Stage I. In summary, sequential methods involve multiple stages in which a model is fitted to the residuals of previous stages using a specific loss function. In the present case, it is the first stage, a physical model that encapsulates the domain knowledge of an assembly or physical system.

[0035] FIG. 11 is a diagram illustrating gradient boosting, according to one embodiment.

[0036] Returning to FIG. 1 , after creating the hybrid model 24, the training data 22 is used to calculate the residuals 31, while the hybrid modeler 18 tests the predictions made by the hybrid model 24 using the extrapolated test data 23 and the interpolated test data 33. First, the hybrid model 24 is tested against the interpolated test data 33. If the location predictions made by the hybrid model 24 based on the input interpolated data 33 are within a certain error threshold from the output interpolated data 33 (indicating that the model 24 is sufficiently trained), the hybrid model 24 remains unchanged. If the location predictions made by the hybrid model 24 are outside the error threshold, the hybrid model 24 is modified. Such testing prevents overfitting of the hybrid model 24. After testing with the interpolated data 33, the hybrid model is tested with the extrapolated test data 23 by comparing the location predictions made by the hybrid model 24 based on the input extrapolated data 23 with the output extrapolated data 23, in the same manner as the interpolated data 33 described above. If the location prediction made by the hybrid model 24 based on the input extrapolated data 23 is within a certain error threshold from the output interpolated data 23, the hybrid model 24 remains unchanged. If the location prediction made by the hybrid model 24 is outside the error threshold, the hybrid model 24 is modified. Such testing using extrapolated data 23 sets helps ensure that the hybrid models 24 have sufficient predictive and extrapolative capabilities, rather than being tied solely to a single training data 22 set.

[0037] As mentioned above, a loss function 30, hereinafter denoted as Ψ(.,.), is used during the calculation of residuals 31 during each gradient boosting stage, and the loss function 30 must be selected by the hybrid model creator 18 for each stage. The loss function 30 is selected based on a probabilistic model 102 (a model that describes aspects of the modeled system and can vary from behavior to behavior) associated with the data that the hybrid model 24 is attempting to predict (and consequently associated with each stage of building the hybrid model 24). For example, when the hybrid model 24 is used to predict continuous data (such as the location of the minute object 11), the probabilistic model 102 is a Gaussian model. If the prediction data is count, the probabilistic model 102 is a Poisson model. If the prediction model is categorical, the probabilistic model 102 is a multinomial model. Still other types of probabilistic models 102 are possible.

[0038] Different loss functions 30 can be used at different stages of gradient boosting. One loss function 30 that can be used while calculating residuals for physics-based models 26 is the mean squared error (MSE).

[0039]

number

[0040]

number

[0041] However, MSE generally does not ensure that the residuals calculated above are compatible with the probabilistic model 102, which typically assumes independent, identically distributed (IID) random variables. Therefore, the hybrid model builder 18 computes the residuals 31 calculated during each stage of gradient boosting using a variance σ 2 To select the loss function 30, the hybrid model builder 18 may use a Ljung-Box (LJB) loss function.

[0042]

number

[0043] The use of the LJB function ensures the uncorrelated residuals 31 generated using the loss function 30 and their conformance with the probabilistic model 102. In equation (7), L is a hyperparameter that is greater than any possible correlation. For each stage of gradient boosting, a loss function is used, which is selected to drive the output residuals 31 toward a desired goal (optimizing the loss function). Scheduling subsequent actuations allows for compensation of control loop latency, thus improving the performance of the system 10. In one embodiment, with a frame rate of 60 Hz (although other frame rates are possible), there is approximately a 2-3 frame control loop delay from when the particle 11 is at a physical location to when the particle is assessed to be at that location (due to the image transfer and computational costs of the image processing analysis), and approximately a 1-frame delay for actuation (the delay from when the particle is determined to be at a location to when an electrode potential is generated to move the particle 11). The control policy 34 compensates for this 3-4 frame delay, allowing for particle position estimation 3-4 frames in advance.

[0044] Because moving a micro-object 11 to a desired location 35 often requires multiple actuations, having an accurate starting position 28 of the micro-object 11 after each actuation is crucial for generating the appropriate potentials during subsequent actuations to most quickly move the micro-object 11 toward the desired location 35. FIG. 2 is a flow diagram illustrating a method 40 for machine learning-based position estimation for use in microassembly control using a digital computer, according to one embodiment. The method 40 can be implemented using the system 10 of FIG. 1. First, parameters 19 of the system 10 are obtained, and a control model 39 is generated using the parameters 19 (step 41). Training data 22 are obtained (step 42), as described further below with reference to FIG. 3. A hybrid model 24 for predicting the position of one or more micro-objects 11 is constructed (step 43), as described further below with reference to FIG. 4. The accuracy of the hybrid model 24 is tested (step 44) using interpolated data 33 as described above with reference to FIG. 1. If the test results are satisfactory and the test passes (step 45), the method proceeds to step 46. If the results of the test are not satisfactory, the routine returns to step 43. The accuracy of the hybrid model 24 is further tested using extrapolated data as described above with reference to Figure 1 (step 46), and if the results of the test are satisfactory and the test passes (step 47), the method proceeds to step 48. If the results of the test are not satisfactory, the routine returns to step 43.

[0045] A desired position for one or more of the micro-objects 11 is received (step 48), e.g., from a user. A starting position 28 for the micro-objects 11 is obtained (step 49) either by using only images 29 captured by the camera 15 if the micro-objects 11 are stationary, or by using a hybrid model 24 (which takes into account data derived from the images 29) if previous actuations have occurred and the micro-objects 11 are now moving (thus, control loop latency is now a factor in determining their positions). A control policy 34 for moving the micro-objects 11 toward a desired position 35 is generated (step 50) using the starting position 28, the control model 39, and the desired position 35, and the electrodes 12 are actuated via the video projector 14 using the control policy 35 (step 51). If the desired position 35 is achieved after actuation (e.g., as determined using the hybrid model 24) (step 52), the method 40 ends. If the desired position is not achieved, the method 40 returns to step 50.

[0046] Recorded data 21 is required for both training and testing of hybrid model 24. Figure 3 is a flow diagram illustrating a routine 60 for generating training data 22 and test data 101 for use in method 40 of Figure 2, according to one embodiment. A set of recorded data 21 is obtained (step 61) and divided into training data 22 and test data 101, the test data including interpolated data 33 (data within the boundaries of the data set as training data 22) and extrapolated data 23 (data outside the boundaries of the data set as training data 22) (step 62), completing the routine. In one embodiment, the data used as training data is 80% of the data set, and the data used as test data 101 is 20% of the data set, although other percentages are possible.

[0047] The hybrid model 24 constructed using gradient boosting combines predictions from at least one physics-based model 26 and one or more machine learning models 27 to achieve a more accurate prediction of the location of the micro-object 11 at a particular time (the time when electrode actuation using the predicted location occurs) by taking into account both the stochastic and deterministic dynamics of the system 10. FIG. 4 is a flow diagram illustrating a routine 70 for constructing the hybrid model 24 for use in the method 40 of FIG. 2 , according to one embodiment. Initially, the multiple models 26, 27 included in the hybrid model 24 are established (step 71) based on user input, past performance of the hybrid model 24 with several models 26, 27, or ongoing testing of the hybrid model 24 created during the method 60. The physics-based model 26 is defined (step 72) using parameters 19, etc. One or more predictions are made regarding the location of one or more micro-objects 11 at a particular time using the physics-based model 26 based on one or more input recorded data 21 (step 73). A probabilistic model 102 and loss function 30 associated with the data to be predicted are selected (step 74), as described above with reference to FIG. 1. One or more residuals 31 are calculated using the selected loss function 30 based on the difference between the prediction made using the physics-based model and the output data (recorded micro-object positions achieved by actuations associated with the input data) associated with the input data used to make the prediction (step 75). An iterative processing loop (steps 76-81) is initiated for all remaining models. Models are constructed (fitted) based on the residuals of the aforementioned models (step 77). A prediction is made regarding the position of one or more micro-objects 11 at a particular time using the model constructed in step 77 based on one or more input recorded data 21 (step 78).A probabilistic model 102 is selected for the data to be predicted as described above with reference to FIG. 1, and a loss function 30 is selected based on the probabilistic model 102 of the model constructed in step 77 as described above with reference to FIG. 1 (step 79). One or more residuals 31 are calculated using the selected loss function 30 based on the difference between the prediction made using the model constructed in step 77 and the output data (recorded micro-object positions achieved by operations associated with the input data simultaneously used for the prediction) associated with the input data used to make the prediction (step 80). The processing loop of steps 76-81 continues until multiple models 26, 27 (including the physics-based model 26 defined in step 62) included in the hybrid model 24 established in step 71 are achieved (step 81), and the routine 70 terminates. In the above description, in one embodiment, all of the models constructed during loop 76-81 may be machine learning models 27. In a further embodiment, at least some of the models constructed during loop 76-81 may be physics-based models 26. In yet another embodiment, machine learning model 27 can be created during step 72 instead of physics-based model 26, and at least one physics-based model can be created during loop 75-81.

[0048] The benefits of using the hybrid model 24 described above have been experimentally tested, and experimental data is presented below for purposes of illustration, not limitation. Model 24 was implemented in system 10 using the frugally-deep cpp library (although other implementations are possible), and the performance of model 24 was compared to what is referred to below as estimator 1.0 (implemented using a system similar to system 10, except that hybrid model 24 is not employed). Estimator 1.0 estimates particle position at time 1 by using the measured position in feedback control at time 1 - #n * 1 / 60 s, an estimated constant for the control cycle delay, #n. The performance of hybrid model 24 was also compared to that of a system similar to system 10, except that instead of hybrid model 24, only machine learning model 26 was used to determine particle position. FIG. 6 shows the test results of Estimator 1.0 compared to using a hybrid model 24 that included an RNN model as the machine learning model 27 ("RNN-Boosted Estimator 1.0"), using only an ARIMA model to predict particle locations, using an ARIMA model boosted with an RNN model (two machine learning models 26) to predict particle locations, and using an RNN model as the machine learning model 27. The hybrid model 24 that included an ARIMA model and the hybrid model that included an RNN model showed more than two times better predictions of the locations (x, y bounded by a box) of micro-objects 11 (referred to as "particles") than Estimator 1.0. These results were obtained using the recorded data 21 shown with reference to FIG. 5A (locations are also referred to as "waypoints").

[0049] Even when extrapolated data 23 is used for testing, the hybrid model 24 performs better than Estimator 1.0. Figure 7 shows the results of using Estimator 1.0 against an ARIMA model and an RNN model in the extrapolated test dataset. The hybrid model 24, which included an ARIMA model, and the hybrid model 24, which included an ARIMA model boosted with an RNN model, showed approximately 1.5 times better predictions of particle (micro-object 11) positions in x and y (enclosed by boxes), while the RNN model performed the worst on the extrapolated test dataset (enclosed by boxes).

[0050] Figure 8 shows a simulation (approximately 50 minutes) of the entire extrapolated dataset using an unsupervised trained System ID model. The hybrid model 24, which included an RNN-boosted ARIMA model, performed best, with a cumulative RMSE error of approximately 10 camera pixels, resulting in a 5-µm inaccuracy in the final position of a 10-µm spherical particle (microobject 11). Estimator 1.0 resulted in a cumulative RMSE error of approximately 200-300 camera pixels (approximately 100-150 µm inaccuracy in the final particle position).

[0051] Figure 9 shows a simulation of the extrapolated dataset using an unsupervised trained System ID model. The hybrid model 24, which included an RNN-boosted ARIMA model, performed best on the true (gray) particle position, predicting particle positions that deviated slightly (black) with a cumulative RMSE error of approximately 10 camera pixels, an inaccuracy of 5 μm relative to the final position of a 10 μm spherical particle (microobject 11). Use of Estimator 1.0 resulted in a cumulative RMSE error of approximately 200-300 camera pixels (an inaccuracy of approximately 100-150 μm relative to the final particle position). The RNN model did not accurately predict particle positions.

[0052] [Table 1]

[0053] The hybrid model 24 also performs better than using a pure machine-based learning model. Table 1 shows the comparative accuracy of the hybrid model 24 (referred to as Estimator 2.0 in Table 1) with a linear model, a dense model, and an Arima model implemented using frugally-deep (cpp). The model was trained on a randomly generated set of waypoints (microobject locations) for 2 hours and then tested on a new set of waypoints for 1 minute in the simulator mode of the MicroAssembler software. The ARIMA model again showed a 2x improvement in predicting particle positions compared to the improved estimator model, Estimator 2.0 (dark box). While the implementation operates faster than 60 Hz (control cycle clock) to predict the positions of hundreds of particles (microobjects 11), in this particular experiment it could not be scaled to 1,000 particles (light box).

[0054] Figures 10A and 10B show the test Estimator 2.0 (hybrid model 24) and linear, dense, and ARIMA models implemented in microassembler software using frugally-deep (cpp). Figure 10A shows the average prediction error rate reduced by camera pixels by >1 for the boosted Arima RNN (LSTM) model compared to the improved Estimator 2.0 model. Figure 10B shows the implementation using deep and RNN models running faster than 60 Hz (internal control cycle clock) to predict the positions of <300 particles (micro-objects 11) in this particular experiment, but not scaled to 1000 particles.

[0055] While the present invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims

1. 1. A method for machine learning enabled microassembly control using a digital computer, comprising: acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects resulting from previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the acts; building a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations using the training data via gradient boosting, the hybrid model comprising a plurality of models including one or more of the physics-based models and one or more machine learning models; obtaining a dataset; splitting the data set into the training data and interpolated test data; extrapolating the previous motion states to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; If the hybrid model passes the test, controlling the closed-loop system, predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the minute objects; performing another one of said operations using the predicted position and the further position; A method wherein said steps are performed by a suitably programmed computer.

2. The gradient boosting method includes multiple stages, one of the models is constructed during each of the stages, and the method comprises: for each of the models, using a loss function to determine a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; 2. The method of claim 1, further comprising: for each of the models other than the model constructed during a first of the stages, defining that model using the residuals associated with the model constructed during the stage prior to the stage in which that model is constructed.

3. 3. The method of claim 2, further comprising: for each of the models based on the stage associated with that model, selecting the loss function used to determine the residual for each of the models based on a probabilistic model.

4. The method of claim 3 , wherein the loss function is a decorrelation function.

5. The method of claim 4 , wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

6. 5. The method of claim 4, wherein the model constructed during the first stage is one of the physics-based models and the model during the last one of the stages is one of the machine learning models.

7. The method of claim 1 , wherein the one or more processors define a recurrent neural network.

8. The method of claim 7 , wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

9. The method of claim 1 , wherein the computer includes at least one of a central processing unit and a graphics processing unit.

10. 1. A system for machine learning enabled microassembly control using a digital computer, comprising: at least one processor, acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects resulting from previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the actuations; building a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations using the training data via gradient boosting, the hybrid model comprising a plurality of models including one or more of the physics-based models and one or more machine learning models; Obtaining a dataset; splitting the data set into the training data and interpolated test data; extrapolating the previous motion state to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; If the hybrid model passes the test, controlling the closed-loop system; predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the minute objects; and performing another one of the operations using the predicted position and the further position.

11. The gradient boosting includes multiple stages, one of the models is constructed during each of the stages, and the at least one processor: for each of the models, using a loss function to determine a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; 11. The system of claim 10, further configured to: for each of the models other than the model constructed during a first one of the stages, define that model using the residuals associated with the model constructed during the stage prior to the stage in which that model is constructed.

12. the at least one processor:

12. The system of claim 11, further configured to, for each of the models based on the stage associated with that model, select the loss function used to determine the residual for each of the models based on a probabilistic model.

13. The system of claim 12 , wherein the loss function is a decorrelation function.

14. The system of claim 13 , wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

15. 14. The system of claim 13, wherein the model constructed during the first stage is one of the physics-based models and the model during the last one of the stages is one of the machine learning models.

16. The system of claim 10 , wherein the one or more processors define a recurrent neural network.

17. 17. The system of claim 16, wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

18. The system of claim 10 , wherein the at least one processor is at least one of a central processing unit and a graphics processing unit.

19. 1. A system for machine learning enabled microassembly control using a digital computer, comprising: at least one processor, acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects resulting from previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the actuations; building a hybrid model for predicting a position of at least one of the micro-objects after one of the acts using the training data via gradient boosting, the hybrid model including a plurality of models including one or more of the physics-based models and one or more machine learning models, the gradient boosting including a plurality of stages, one of the models being built during each of the stages; at least one processor configured to: The at least one processor further comprises: For each of the models based on the stage associated with the model, selecting a loss function based on a probabilistic model, the loss function including a decorrelation function for determining a residual for each of the models; for each of the models, using the loss function to determine the residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; for each of the models other than the model constructed during a first one of the stages, defining that model using the residuals associated with the model constructed during the stage prior to the stage in which the model is constructed; predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the minute objects; performing another one of said operations using the predicted position and the further position.

Images