System and method for machine-learning-based position estimation for use in micro-assembly control with aid of digital computer

Abstract

To provide a system and method for machine-learning-based position estimation for use in microassembly control with the aid of a digital computer.SOLUTION: Control loop latency is taken into account in predicting positions of micro-objects being moved by using a hybrid model that includes both a physics-based model and machine-learning models. The models are fitted based on residuals calculated during a previous stage using gradient boosting and based on comparison with training data. A loss function for each stage is selected based on the model being created. The hybrid model is evaluated on data extrapolated and interpolated from the training data to prevent overfitting and to have a sufficient predictive ability. By including both the physics-based model and the machine-learning model, the hybrid model can account for both deterministic and stochastic components involved in the movement of the micro-object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This application generally relates to microassembly control, and more particularly to systems and methods for machine learning-based position estimation for use in microassembly control using digital computers.

Background Art

[0002] Microscale and nanoscale particle manipulation has gathered the interest of many researchers. The degree to which it is possible to control the assembly of micro-objects, which are objects with dimensions in the micron range, and nano-objects, which are objects with dimensions in the nanometer range, can make a big difference in many technologies, including microfabrication, biology, and medicine. For example, by accurately controlling the positions of micro-objects such as capacitors and resistors so as to be able to manufacture a circuit having a desired behavior, the manufacture of reconfigurable electrical circuits can be improved. Similarly, the production of photovoltaic solar cell arrays can benefit from being able to place photovoltaic cells having specific qualities at specific positions on the array. Such cells are too small to allow their desired placement via human or robotic manipulation and require a different kind of transport mechanism. The microassembly and nanoassembly of particles can also be used in the manipulation of the microstructure of materials such as biological cells that are assembled into tissues. There are many other technical fields in which significant advantages can be brought about by increasing the control of the assembly of micro-objects and nano-objects.

[0003] Many techniques used in microparticle and nanoparticle manipulation involve using electrodes to generate an electrode potential and move the target particle in a desired direction. The initial position of the particle is determined using a sensor, and the electrode potential is generated based on the desired position and the initial position. However, the throughput and accuracy of such systems are significantly limited by the control loop latency, which is the delay between when the particle is at a particular position and when the electrode potential is generated based on that position. The exact length of the delay depends on the software and hardware of the system used for particle manipulation. The delay has multiple components. One component is the operation delay, which is the delay between when the sprite (control signal for the electrode) is generated and when the electrode generates the electrode potential specified by the sprite. Another component of the delay is the detection delay, which is the time between the sensor observing the particle occupying a particular position and that time. The particle position is obtained after the analysis of the sensor signal is complete. Such delays include the time to feed the captured image into memory and store it, the time to send the image data to image analysis to determine its location, and the time to run the image analysis algorithm. These delays can range from 80ms to 100ms and even over 300ms, depending on various latency and processor loads, and can occupy up to five frames at frame rates below 60Hz.

[0004] By predicting delays and subsequently the actual positions of micro- or nano-sized particles that need to move, it will be possible to improve the accuracy and efficiency of microassemblies and nanoassemblies. Such modeling generally utilizes physics-based models, which are models that take advantage of the known physical properties of the operational setup, the environment (such as the assembly being in air, liquid, on the ground, in space, or another environment), and the particles in motion. However, physics-based models of closed-loop systems do not take into account the dynamic changes in control latency that occur during micro-particle or nanoparticle operations, such as particle rotation, target assignment, and transfers that occur in small or large subsets of particles. Such physics-based models also do not take into account the dynamic changes between different settings of operational conditions, such as the use of different types of particles or different lighting conditions. Such physics-based models are also not robust when modeling certain variables within a particular operational setup, such as particle stiction or particle oscillation, or when modeling parameters with unknown values, such as the average number of pixels to which particles transition during a control cycle.

[0005] Therefore, a method is needed to accurately account for the control loop latency required to predict the actual position of particles moved due to the generated electrode potential. [Overview of the project]

[0006] In predicting the position of moving micro-objects, control loop latency can be taken into account by using a hybrid model that includes at least one physics-based model and a machine learning model. The models are coupled using gradient boosting, and the model created during at least one of the stages is fitted based on the residuals calculated during the previous stage, based on a comparison with the training data. The loss function for each stage is selected based on the model being created. The hybrid model is extrapolated from the training data and evaluated on the interpolated data to prevent overfitting and to ensure that the hybrid model has sufficient predictive power. By including both physics-based and machine learning models, the hybrid model can account for both deterministic and probabilistic components involved in the movement of micro-objects, thus increasing the accuracy and throughput of the microassembly.

[0007] In one embodiment, a system and method for machine learning-enabled microassembly control using a digital computer are provided. One or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects are acquired, the system comprising a plurality of programmable electrodes, the electrodes configured to induce the movement of a micro-object when the micro-object is in proximity to the electrode when the electrode is actuated, and each actuated comprising the generation of one or more potentials by one or more of the electrodes. Training data is acquired, which includes states associated with the previous movement of one or more of the plurality of micro-objects resulting from a previous actuated

[0008] Further embodiments of the present invention will be readily apparent to those skilled in the art from the following detailed description, and embodiments of the present invention will be described by illustrating the best mode intended for carrying out the invention. As realized, other different embodiments of the present invention are possible, and some of its details can be modified in various obvious ways, all without departing from the spirit and scope of the invention. Accordingly, the drawings and detailed description should be considered as illustrative and not limiting. [Brief explanation of the drawing]

[0009] [Figure 1]This is a block diagram showing a machine learning-based position estimation system used in microassembly control using a digital computer. [Figure 2] This flowchart illustrates a machine learning-based position estimation method used in microassembly control with digital computers. [Figure 3] This flowchart shows a routine for generating training data used in the method shown in Figure 2, according to one embodiment. [Figure 4] This flowchart shows a routine for constructing a hybrid model used in the method shown in Figure 2, according to one embodiment. [Figure 5A] This figure shows examples of the positions taken by micro-objects in the recorded training data (see Figure 5A) and the positions of micro-objects extrapolated from the recorded training data (see Figure 5B). [Figure 5B] This figure shows examples of the positions taken by micro-objects in the recorded training data (see Figure 5A) and the positions of micro-objects extrapolated from the recorded training data (see Figure 5B). [Figure 6] This figure shows the test results of estimator 1.0, comparing the use of a hybrid model that includes the ARIMA model as the machine learning model with the use of a hybrid model that includes the RNN model as the machine learning mode. [Figure 7] This figure shows the results of using estimator 1.0 for the ARIMA model and RNN model in the extrapolated test dataset. [Figure 8] This figure shows a simulation (approximately 50 minutes) of the entire extrapolated dataset using an unsupervised, trained system ID model. [Figure 9-1] This figure shows a simulation of an extrapolated dataset using an unsupervised, trained system ID model. [Figure 9-2] This figure shows a simulation of an extrapolated dataset using an unsupervised, trained system ID model. [Figure 9-3]This figure shows a simulation of an extrapolated dataset using an unsupervised, trained system ID model. [Figure 10A] This figure shows the Test Estimator 2.0 (hybrid model) and the linear, high-density, and ARIMA models implemented in microassembler software using frugally-deep (cpp). [Figure 10B] This figure shows the Test Estimator 2.0 (hybrid model) and the linear, high-density, and ARIMA models implemented in microassembler software using frugally-deep (cpp). [Figure 11] This figure shows gradient boosting according to one embodiment. [Figure 12] This diagram illustrates the relationship between recorded data, extrapolated data, and interpolated data when defined for a complex ship hull. [Modes for carrying out the invention]

[0010] The systems and methods described below allow for the consideration of control loop latency when generating control signals, and thus can improve the accuracy of positioning achieved using microassemblies or nanoassemblies. As described below, the systems and methods leverage a hybrid model for predicting the position of a micro-object after operation, the hybrid model comprising one or more physics-based models and one or more machine learning models. The physics-based models have the advantages of not requiring any arbitrary data to build, allowing extrapolation beyond the data used to fit the parameters, and having high interpretability of these predictions, whereas they can only capture a small portion of the behavior of the system used to perform the microassembly. The complete behavior of system 10 includes unmodeled degrees of freedom, and these unmodeled degrees of freedom can be captured by using machine learning models. The total system behavior given by Y is a physical model f(X, θ), where X is the previous history and measurements, θ is a parameter, and noise N(ф) is the noise parameter of a probabilistic noise model. The neural network function NN(X, W) is a function of the input and trainable weights W, and captures the unmodeled degrees of freedom according to the following equation.

[0011]

number

[0012] The use of hybrid models allows us to combine the advantages of physics-based models (interpretability, stability, and extrapolation ability) with the flexibility and ability of machine learning models to capture unmodeled degrees of freedom, resulting in more accurate predictions of the positions of micro-objects and, therefore, improved throughput for microassemblies. Physics-based models consider the deterministic components of the system, while machine learning models consider the probabilistic components.

[0013] In the following description, the object to be moved is described as a micro-object, but the same technique can also be applied to control the position of a nano-object. Further, the hybrid model described below is described as being applied to a specific operating hardware, but in further embodiments, the hybrid model can be used to predict the position of a micro (or nano) object starting with a different hardware setup. In yet another embodiment, the hybrid model can be used to control other types of systems. For example, the hybrid model can be applied to the control of a motor-driven mechanical linkage having backlash. The physics of the motor and the rotation of the mechanical linkage can be easily modeled using a physical model including links such as the physical parameters of the motor and the moment of inertia, and the series resistance of the winding. The backlash of the linkage is non-linear, history-dependent, and much more complex. Subjecting the motor system to a random series of motions, various frequencies and amplitudes are controlled, and by comparing the measured response with the response expected from the physical model, the error of the physical model as a function of the input and the state of the motor can be obtained. This data is used for supervised training of a neural network trained to predict the backlash error. The nearby time errors are forced to be uncorrelated, providing further constraints against overfitting. The remaining error is then fitted to a probabilistic model which is independent and identically distributed Gaussian noise in this case. The noise variance can be determined from the residuals from the physics and neural network models. This hybrid model is used for system identification, i.e., to predict the response of the system as various control actions and past and current states of the system's function. This prediction can be incorporated into various control algorithms such as model predictive control to achieve improved performance over simpler previous control algorithms such as the typical proportional integral derivative (PID) control algorithm. Further other uses of the hybrid model are possible.

[0014] Figure 1 is a block diagram showing a system 10 for machine learning-based position estimation used in microassembly control using a digital computer. System 10 enables cooperative assembly of several objects 11, such as micro-objects. The size of the objects 11 varies between nanoscale (<1 μm) and microscale (1 μm to several hundred μm), but other sizes are possible. The objects 11 can be spherical, but other shapes such as rectangles are also possible, and even more shapes are possible. In one embodiment, the diameter of the spherical object 11 is 5 μm to 50 μm, but other diameters are possible. In one embodiment, the dimensions of the rectangular object 11 can be 200 μm x 300 μm x 100 μm, but other dimensions are also possible. The objects 11 are immersed in a dielectric fluid (not shown) contained within a casing (not shown), and a layer of film (not shown) is beneath the dielectric fluid contained within the casing. In one embodiment, the dielectric fluid is Isopar® M manufactured by ExxonMobil Chemical Company in Spring, Texas, but other dielectric fluids 17 are also possible. The dielectric fluid may contain one or more additives, such as di-2-ethylhexyl sulfosuccinate (AOT) charge-inducing molecules, but other additives are also possible. Object 11 can be made from aluminum oxide (AlOx), but other materials are also possible. Each of the objects 11 may be a semiconductor chip, integrated circuit, particle, nanodevice, or structure, or any object that can be physically manipulated, or an array of electrodes. In the following description, object 11 will be referred to as micro-object 11, but other ways of naming object 11 are possible.

[0015] Below the suspended microparticle 11, a plurality of programmable electrodes 12 form an array 35, and the electrodes 12 are configured to generate a dynamic potential energy landscape for manipulating the object with both dielectrophoretic ("DEP") and electrophoretic ("EP") forces. A film is laminated on the electrodes 12. In one embodiment, the electrodes can be square-shaped and made of copper, although other shapes and materials are also possible. In one embodiment, the dimensions of the square electrode 12 can be 16 μm wide and 100 nm thick, although in further embodiments other dimensions are possible. The array 35 can include a number of columns of electrodes 12, and each column includes a number of electrodes 12.

[0016] The potential generated by the electrodes 12 is controlled by an array 13 of optical transistors including an active matrix optical transistor backplane set on glass. A number of optical transistors on the backplane form the array 13, and each of the optical transistors in the array 13 controls the potential generated by a single electrode 12. In particular, each of the optical transistors in the array 13 is attached to one electrode 12. The array 13 of optical transistors can have additional characteristics, such as those described in Rupp et al.'s "Chiplet micro-assembly printer" 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), pages 1312-1315, May 2019, the disclosure of which is incorporated by reference.

[0017] Array 13 is optically addressed by a video projector 14 to enable simultaneous dynamic control of the electrostatic energy potential and manipulation of the positions of numerous micro-objects 11. In particular, the video projector 14 is used to address each optical transistor control electrode 12, allowing for easy changes in the zoom factor and stitching to a larger array. The video projector 14 acts on the electrodes 12 by projecting a predetermined pattern, an image 37, which is generated based on a control input of the potential that the electrodes should generate to achieve the desired motion of the micro-objects 11, as will be further described below. The pixel light projected by the projector 14 that constitutes the image charges each individual optical transistor 13 in the array to the required extent so that the electrodes 12 are charged by their optical transistors, thereby generating the desired potential. In one embodiment, voltages up to + / -200V are controlled by these optical transistors to display a dynamically changing complex potential terrain. A single voltage signal is synchronized with optical addressing to charge storage capacitors on each pixel. The dynamic electric field pattern is generated by refreshing the projected image pattern (control input). Arrays 13 with pitches of 50 μm, 10 μm, and 3 μm are constructed, where the pitch is the distance between the centers of the electrodes to which the phototransistors are attached. A dielectric fluid 17 medium (isopar M) and various additives are used to control the charge.

[0018] System 10 further includes a high-speed camera 15 that tracks the location of a moving micro-object 11. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16, which can control the electrodes 12 via the projector 14 to induce the formation of a desired micro-object 11 pattern. Other micro-object 11 patterns are also possible. In further embodiments, the position of the micro-object can be measured and tracked using a different type of sensor (such as utilizing capacitive sensing). In yet another embodiment, the electrode array 35 may have an integrated active matrix addressing electronic device, and the potential on each electrode of the array is electronically updated when the electrode potential required for the movement of the desired micro-object generated is generated. Additional characteristics of the array 13, electrodes 12, video projector 14, and camera 15 are described in U.S. Patent Publication Application 2020 / 0207617, published 2 July 2020, by Plochowietz et al., whose disclosure is incorporated by reference. Furthermore, additional components of the system 10 that interact with the array 14, electrodes 12, video projector 14, and camera 15 are possible.

[0019] The connection between one or more computing devices 16 may be direct, for example, via wire or wireless connection to physically close the one or more computing devices 16, or the connection may be via internet work to physically distance the one or more computing devices 16 via the internet or cellular network. The one or more computing devices 16 include one or more circuits that perform data processing, such as computer processors, but other types of circuits for processing data are possible. In one embodiment, the computer processor may be a central processing unit (CPU), a graphics processing unit (GPU), or a mixture of a CPU and a GPU, but other types of processors or mixtures of processors are possible. In one embodiment, two or more of the processors may perform processing in parallel, as further described in Matei et al., U.S. Patent Application No. 17 / 098,816, filed November 16, 2020, entitled "System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer," which is incorporated by reference. Additional efficiency and improved scalability can be achieved through parallel computing capabilities for tracking a large number of particles and for generating voltage patterns. In yet another embodiment, circuits other than the CPU and GPU can perform the above data processing.

[0020] While one or more computing devices 16 are shown as servers, other types of computer devices are possible. The computing device 16 may include one or more modules for performing the embodiments disclosed herein. A module may be implemented as a computer program or procedure written as source code in a conventional programming language and presented as object or bytecode for execution by a processor. Alternatively, a module may also be implemented as an integrated circuit or in hardware burned into a read-only memory component, and each of the computing devices 16 may function as a specialized computer. For example, if a module is implemented as hardware, that particular hardware is specialized to perform the calculations and communications described above, and cannot be used by other computers. Furthermore, if a module is burned into a read-only memory component, the computer storing the read-only memory is specialized to perform the operations described above that other computers cannot. Various implementations of source code, object code, and bytecode can be stored on computer-readable storage media such as floppy disks, hard drives, digital video disks (DVDs), random access memory (RAM), read-only memory (ROM), and similar storage media. Other types of modules and module functions, as well as other physical hardware components, are possible. For example, computing device 16 may include, but is not limited to, input / output ports, network interfaces, and other components found in programmable computing devices, such as non-volatile storage devices. In embodiments where computing device 16 is a server, the server may also be cloud-based or a dedicated server.

[0021] One or more computing devices 16 interface to a storage device 17 and run a hybrid model generator 18 that acquires parameters 19 of the system 10 and stores the parameters 19 in the storage device 17. The parameters 19 may include the diameter of the micro-object 11, the dimensions of the electrode 12, the dielectric constant (e.g., ε=2), the fixed position of the electrode 12, the materials of the micro-object 11 and the electrode 12, and the vertical distance between the micro-object 11 and the electrode 12 (the "height" of the micro-object 11). Other parameters 19 are still possible. The hybrid model generator 18 also creates a hybrid model 24 for predicting the position of a particular micro-object 11, which can also be stored in the storage device 17. The hybrid model 24 includes a physics-based model 26 that predicts the position of the micro-object 11 after electrode activation, based on the generated control signals (and consequently the potential generated by the electrode), the velocity of the micro-object 11 at a particular time, and the starting position 28 (position before activation) based on a micro-object image 29 captured by the camera 15. Examples of physics-based models are described by J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar in "Integrating physics-based modeling with machine learning: A survey," arXiv:2003.04919[physics.stat](2020), and their disclosure is incorporated by reference, but other physics-based models26 are possible. Hybrid models24 further include one or more machine learning models27. Machine learning models may include deep models (machine learning models that utilize neural networks constructed with multiple layers), linear models, high-density models, autoregressive integrated moving average (ARIMA) models, and recurrent neural network (RNN) models, but other types of machine learning models are possible. In one embodiment, the SimpleRNN layer of the ARIMA model can be changed to LSTM (Long-Term Short-Term Memory, RNN Architecture), but other types of ARIMA models are possible.As described above, the physics-based model 26 takes into account the deterministic behavior of system 10, while the machine learning model 27 takes into account the deterministic aspects of system 10. Thus, the overall behavior of system 10 can be given by Y as a function of the physical model f(X, θ), where X is recorded data (hereinafter referred to as training data 22) associated with previous operation of system 10, θ is a parameter, and ф is the noise parameter of the probabilistic noise model. The system behavior is further expressed by a neural network function NN(X, W) (associated with machine learning model 27), which is a function of the input and trainable weights.

[0022] The hybrid model 24 is constructed using training data 21. The training data describes states associated with the movement of the micro-object 11, including (1) the input state of the micro-object 11 that minimizes its position and velocity before the control input activates the electrodes (although other states such as orientation and angle may also be included), (2) the control action such as the lighting pattern applied to the phototransistor control electrode 12 by the video projector 14 (indicating the potential generated by the electrodes) or another force applied to the micro-object 11, and (3) the output state resulting from the input data (the position of the micro-object 11 is achieved after activation at a specific time and velocity of the micro-object). The training data 22 is used for supervised training of a set of parametric models that make up the hybrid model 24, as shown in Figure 11. The “conventional” models shown in Figure 11 can be the physics-based model 24 and a neural network model of model 2 (shown in Figure 11) that, for example, uses residuals and further models are fitted to the residuals.

[0023] The hybrid model is tested using interpolated data 33 (also called interpolated data 33) and extrapolated data 23 (also called extrapolated data 23), which are used as test data 101, as will be further described below. The interpolated data 33 can initially be part of the same dataset as the training data 22 and include input states, control actions, and output states whose values ​​are within the range of values ​​in the training data 21. The extrapolated training data 23 can also be part of the same dataset as the training data 21 and include input states, control actions, and output states, but with the condition that at least one of the input states, control actions, and output states is outside the range of values ​​for the input states, control actions, and output states in the training data 21. For example, at least one of the position, velocity, or action applied to the micro-object 11 is for a value that extends beyond the value used in the training data 21.

[0024] Figures 5A and 5B are annotated bright-field images of experimental (21 recorded) data, where micro-objects 11 are highlighted with white circles, and the set of intermediate target positions to which the micro-objects transition is highlighted with magenta squares. Figures 5A and 5B show two different experiments with two different sets of target positions. This experimental data 21 can be used as training data 22. Figure 12 is a diagram of the relationship between the training data 22, extrapolated data 23, and interpolated data 33 when defined with respect to a complex hull. Further other types of training data are possible.

[0025] As further explained below, beginning with a reference to Figure 2, the hybrid model 24 is created using gradient boosting implemented in many stages, in which case one or more physics-based models are trained by selecting a loss function, the next stage is implemented by selecting another model (machine learning model 27) using the residuals 31 and previous inputs as new inputs, and selecting a loss to boost (train) the parameters of the next stage, and the creation of additional models is repeated multiple times in this manner.

[0026] Therefore, during gradient boosting, first, at least one physics-based model 26 is defined, and then a loss function 30 (also called the whitening objective function 30) is used to analyze the difference between output training data 22 (the position of the micro-object 11 at a specific time after operation) and one or more predictions made by using the physics-based model 26 for the achieved position of the micro-object 11 at the same time described by the output training data after the same operation based on the input training data. The result of that analysis is one or more residuals 31 (also known as pseudo-residuals 31), each residual 31 describing the difference between one position predicted by the physics-based model 26 based on one input training data 22 and one output training data 22 resulting from the one input training data 22 on which that prediction was made. Subsequently, during the next stage, a first machine learning model 27 is constructed using the residuals 31 derived using the first physics-based model 26, and the first machine learning model 27 is fitted based on the generalized residuals 31 of the physics-based model 26. Next, during the following stage, residuals 31 are generated for the first machine learning model based on the training data 22 using a loss function 30, which is the same method as above, to process the difference between the predicted micro-object positions based on the input training data 22 and the positions in the output training data 21. The residuals 31 from the first machine learning model 30 are used to define another deep model, and the residuals 31 of the second deep model are calculated in the same method as above. This continues as a cascade in multiple stages by fitting the generalized residuals of one machine learning model 27 to the next machine learning model 27 until a desired number of models 26, 27 are generated (in one embodiment, the desired number of models is based on historical data regarding the accuracy of a hybrid model including a certain number of models 26, 27, and in a further embodiment, the desired number of models 26, 27 is set based on empirical testing of whether a given number of models produce the desired accuracy). In the above description, only a single physics-based model 26 is used as part of the hybrid model 24, but in a further embodiment, multiple physics-based models 26 can be used as part of the hybrid model.One or more physics-based models 26 within the hybrid model 24 act as weak learners, whose predictions are only slightly better than random guesses, while the trained machine learning model 27 acts as a strong learner with higher predictive accuracy than the weak learners. All predictions from models 26, 27 are aggregated into a single prediction in the hybrid model 24 by assigning predictions from the weights of different models and averaging those predictions, although other methods of aggregating predictions are also possible. For example, predictions from various model stages 26, 27 can be aggregated as a sum or as an improvement of the product in the combined prediction of the hybrid model 24 according to equation (2) below, where the residuals are expressed as factors rather than differences. Different weights can also be assigned to the different stages. Mathematically, gradient boosting performed by the hybrid modeler 18 can be defined as follows:

[0027] A series of M learners is used to improve the predictive output. The weak (or base) model includes a physical / deterministic model26 that follows the machine learning model to capture the predictable but complex parts.

[0028]

number

[0029] In the formula, M is the number of repetitions.

[0030]

number

[0031]

number

number

[0032] During the ceremony,

[0033]

number

[0034] In the formula, Ψ i (.,.) is the loss function for stage I. In summary, the sequential method involves multiple stages in which the model is fitted to the residuals of previous stages using a specific loss function. In the current case of a physical model encompassing domain knowledge of an assembly or physical system, this is the first stage.

[0035] Figure 11 shows a gradient boosting according to one embodiment.

[0036] Returning to Figure 1, the training data 22 is used to calculate the residuals 31 after the hybrid model 24 is created, but the hybrid modeler 18 tests the predictions made by the hybrid model 24 using the extrapolated test data 23 and the interpolated test data 33. First, the hybrid model 24 is tested against the interpolated test data 33. If the position prediction made by the hybrid model 24 based on the input interpolated data 33 is within a certain error threshold from the output interpolated data 33 (indicating that the model 24 is well trained), the hybrid model 24 remains unchanged. If the position prediction made by the hybrid model 24 is outside the error threshold, the hybrid model 24 is modified. Such testing prevents overfitting of the hybrid model 24. After testing with the interpolated data 33, the hybrid model is tested with the extrapolated test data 23 by comparing the position prediction made by the hybrid model 24 based on the input extrapolated data 23 with the output extrapolated data 23, in the same manner as with the interpolated data 33 described above. If the position prediction made by the hybrid model 24 based on the input extrapolated data 23 is within a specific error threshold from the output extrapolated data 23, the hybrid model 24 remains unchanged. If the position prediction made by the hybrid model 24 is outside the error threshold, the hybrid model 24 is modified. Such tests using sets of extrapolated data 23 help ensure that those hybrid models 24 have sufficient predictive and extrapolative capabilities, rather than being tied only to a single set of training data 22.

[0037] As described above, the loss function 30, denoted below as Ψ(.,.), is used in calculating the residuals 31 between each of the gradient boosting stages, and the loss function 30 must be selected by the hybrid model builder 18 for each stage. The loss of function 30 is selected based on a probabilistic model 102 (a model that describes the aspects of the modeled system and can vary from action to action) associated with the data that the hybrid model 24 is trying to predict (and consequently associated with each stage that builds the hybrid model 24). For example, when the hybrid model 24 is used to predict continuous data (such as the position of a micro-object 11), the probabilistic model 102 is a Gaussian model. If the predicted data is counted, the probabilistic model 102 is a Poisson model. If the prediction model is categorical, the probabilistic model 102 is a polynomial model. Further other types of probabilistic models 102 are possible.

[0038] Different loss functions 30 can be used at different stages of gradient boosting. One loss function 30 that can be used while calculating the residuals in the case of the physics-based model 26 is the mean squared error (MSE).

[0039]

number

[0040]

number

[0041] However, MSE generally does not guarantee that the residuals calculated as described above are compatible with the probabilistic model 102, which typically assumes independent, identically distributed (IID) random variables. Therefore, the hybrid model generator 18 calculates the residuals 31 between each stage of gradient boosting, and the variance σ 2 This is compared to a probabilistic model 102 that uses an autocorrelation function such as a zero-mean Gaussian one. To select the loss function 30, the hybrid model generator 18 can use the Ljung-Box(Ljung-Box, LJB) loss function.

[0042]

number

[0043] The LJB function ensures the fit of the uncorrelated residuals 31 generated using the loss function 30 to the residuals with the probabilistic model 102. In equation (7), L is a hyperparameter greater than the possible correlation. For each step of gradient boosting, the loss function is selected to drive the output residuals 31 toward the desired target (optimize the loss function). By planning the subsequent operation, it is possible to compensate for the control loop latency and thus improve the performance of the system 10. In one embodiment, with a frame rate of 60 Hz (although other frame rates are possible), there is a control loop of about 2-3 frames from when the particle 11 is at its physical location until it is estimated that the particle is at that location (due to the image transfer and computational costs of the image processing analysis), and a delay of about 1 frame for operation (the delay from when it is determined that the particle is at a certain location until the electrode potential is generated to move the particle 11). The control policy 34 compensates for this 3-4 frame delay and allows the particle location to be estimated 3-4 frames in advance.

[0044] Since moving the micro-object 11 to a desired position 35 almost always requires multiple operations, having the precise starting position 28 of the micro-object 11 after each operation is crucial for generating the appropriate potential between subsequent operations in order to move the micro-object 11 toward the desired position 35 as quickly as possible. Figure 2 is a flowchart showing a method 40 for machine learning-based position estimation used in micro-assembly control using a digital computer, according to one embodiment. Method 40 can be implemented using the system 10 of Figure 1. First, parameters 19 of the system 10 are acquired, and a control model 39 is generated using the parameters 19 (step 41). Training data 22 is acquired, as further described below with reference to Figure 3 (step 42). A hybrid model 24 for predicting the position of one or more micro-objects 11 is constructed, as further described below with reference to Figure 4 (step 43). The accuracy of the hybrid model 24 is tested using the interpolated data 33 as described above, with reference to Figure 1 (step 44), and if the test results are satisfactory and the test passes (step 45), the method proceeds to step 46. If the test results are unsatisfactory, the routine returns to step 43. The accuracy of the hybrid model 24 is further tested using the extrapolated data as described above, with reference to Figure 1 (step 46). If the test results are satisfactory and the test passes (step 47), the method proceeds to step 48. If the test results are unsatisfactory, the routine returns to step 43.

[0045] The user or other party receives one or more desired positions for the micro-objects 11 (step 48). The starting position 28 of the micro-objects 11 is obtained either by using only the image 29 captured by the camera 15 if the micro-objects 11 are stationary, or by using a hybrid model 24 (which takes into account the data derived from the image 29) if a previous operation has been performed and the micro-objects 11 are currently moving (therefore, control loop latency is a factor in determining their positions here) (step 49). A control policy 34 for moving the micro-objects 11 toward a desired position 35 is generated using the starting position 28, the control model 39, and the desired position 35 (step 50), and the electrodes 12 are actuated via the video projector 14 using the control policy 35 (step 51). If the desired position 35 is achieved after the actuation (e.g., determined using the hybrid model 24) (step 52), method 40 terminates. If the desired position is not achieved, method 40 returns to step 50.

[0046] The recorded data 21 is required for both training and testing the hybrid model 24. Figure 3 is a flowchart showing a routine 60 for generating training data 22 and test data 101 for use in method 40 of Figure 2, according to one embodiment. The set of recorded data 21 is acquired (step 61) and split into training data 22 and test data 101, the test data including interpolated data 33 (data within the boundaries of the dataset as training data 22) and extrapolated data 23 (data outside the boundaries of the dataset as training data 22) (step 62), and the routine ends. In one embodiment, the data used as training data is 80% of the dataset and the data used as test data 101 is 20% of the dataset, but other proportions are possible.

[0047] The hybrid model 24, constructed using gradient boosting, combines predictions from at least one physics-based model 26 and one or more machine learning models 27 to achieve a more accurate prediction of the position of micro-objects 11 at a given time (the time when electrode activation using the predicted positions is performed) by taking into account both the probabilistic and deterministic dynamics of the system 10. Figure 4 is a flowchart showing a routine 70 for constructing the hybrid model 24 used in Method 40 of Figure 2, according to one embodiment. First, the multiple models 26, 27 included in the hybrid model 24 are set based on user input, etc., based on the past performance of the hybrid model 24 with some models 26, 27, or based on continuous testing of the hybrid model 24 created during Method 60 (Step 71). The physics-based model 26 is defined using parameters 19, etc. (Step 72). One or more predictions are made for the position of one or more micro-objects 11 at a given time, and are made using the physics-based model 26 based on one or more input recorded data 21 (Step 73). Referring to Figure 1, the probabilistic model 102 and loss function 30 associated with the predicted data are selected as described above (step 74). One or more residuals 31 are calculated using the selected loss function 30 based on the difference between the prediction made using the physics-based model and the output data associated with the input data used to make the prediction (recorded minute object positions achieved by the operation associated with the input data) (step 75). The iterative processing loop (steps 76-81) is started for all remaining models. The models are built (fitted) based on the residuals of the aforementioned models (step 77). Predictions are made for the position of one or more minute objects 11 at a given time, using the model built in step 77 based on one or more input recorded data 21 (step 78).A probabilistic model 102 is selected for the data to be predicted as described above with reference to Figure 1, and the loss function 30 is selected based on the probabilistic model 102 of the model constructed in step 77 as described above with reference to Figure 1 (step 79). One or more residuals 31 are calculated using the selected loss function 30 based on the difference between the prediction made using the model constructed in step 77 and the output data associated with the input data used to make the prediction (recorded minute object positions, achieved simultaneously with the prediction by the operation associated with the input data) (step 80). The processing loop of steps 76-81 moves on to the next model (step 81) and terminates routine 70 until multiple models 26, 27 (including the physics-based model 26 defined in step 62) included in the hybrid model 24 set up in step 71 are achieved. In the above description, in one embodiment, all models created between loops 76-81 can be machine learning models 27. In further embodiments, at least some of the models created between loops 76-81 can be physics-based models 26. In yet another embodiment, the machine learning model 27 can be created during step 72 instead of the physics-based model 26, and at least one physics-based model can be created during loops 75-81.

[0048] The advantage of using the above hybrid model 24 is that it has been experimentally tested, and the experimental data is shown below for illustrative purposes only and not limited to. Model 24 is implemented in system 10 using the frugally-deep cpp library (although other implementations are possible), and the performance of model 24 is compared to what is called estimator 1.0 below (implemented using a system similar to system 10, except that the hybrid model 24 is not employed), which estimates the particle position at time 1 by using the estimated constant for the control cycle delay, the control cycle latency at time 1 - #n*1 / 60s, and the measured position in the feedback control at #n. The performance of hybrid model 24 is also compared to the performance of a system similar to system 10, except that only machine learning model 26 is used to determine the particle position instead of hybrid model 24. Figure 6 shows the test results of estimator 1.0 compared to using a hybrid model 24 that includes an RNN model as machine learning model 27 ("RNN-boosted estimator 1.0"), using only the ARIMA model to predict particle locations, using the ARIMA model boosted with an RNN model (two machine learning models 26) to predict particle locations, and using the RNN model as machine learning model 27. The hybrid model 24 that includes the ARIMA model and the hybrid model that includes the RNN model showed more than twice the prediction of the location (x, y enclosed by boxes) of the micro-objects 11 (referred to as "particles") compared to estimator 1.0. These results were obtained using recorded data 21 (locations also referred to as "waypoints"), as shown with reference to Figure 5A.

[0049] Even when extrapolated data 23 was used in the trial, the hybrid model 24 performed better than estimator 1.0. Figure 7 shows the results of using estimator 1.0 for the ARIMA model and the RNN model in the extrapolated trial dataset. The hybrid model 24 with the ARIMA model, and the hybrid model 24 with the ARIMA model boosted by the RNN model, showed approximately 1.5 times better predictions of the particle (micro-object 11) position in x and y (enclosed by boxes), while the RNN model performed the worst in the extrapolated trial dataset (enclosed by boxes).

[0050] Figure 8 shows a simulation (approximately 50 minutes) of the entire extrapolated dataset using an unsupervised, trained system ID model. Hybrid model 24, which included an RNN-boosted ARIMA model, showed the best results with a cumulative RMSE error of approximately 10 camera pixels and an inaccuracy of 5 μm for the final position of 10 μm spherical particles (micro-objects 11). Estimator 1.0 yielded a cumulative RMSE error of approximately 200–300 camera pixels (inaccuracy of approximately 100–150 μm for the final particle position).

[0051] Figure 9 shows a simulation of an extrapolated dataset using an unsupervised, trained system ID model. A hybrid model 24, including an RNN-boosted ARIMA model, showed the best results in true (gray), predicting particle positions with a cumulative RMSE error of approximately 10 camera pixels and a slight deviation (black) of 5 μm inaccuracy to the final position of a 10 μm spherical particle (micro-object 11). Using estimator 1.0 resulted in a cumulative RMSE error of approximately 200–300 camera pixels (an inaccuracy of approximately 100–150 μm in the final particle position). The RNN model did not accurately predict the particle position.

[0052] [Table 1]

[0053] Hybrid Model 24 also shows better results than using a purely machine-based learning model. Table 1 shows the comparative accuracy of Hybrid Model 24 (referred to as Estimator 2.0 in Table 1) and a linear model, a high-density model, and the Arima model, all implemented using frugally-deep (cpp). The models were trained on a 2-hour randomly generated waypoint set (micro-object positions) and tested for 1 minute with a new waypoint set in the simulator mode of the microassembler software. The ARIMA model again showed a 2x improvement in predicting particle positions compared to the improved Estimator Model Estimator 2.0 (dark boxes). In its implementation form, it operates faster than 60 Hz (control cycle clock) to predict the positions of several hundred particles (micro-objects 11), but in this particular experiment, it could not be scaled up to 1000 particles (light boxes).

[0054] Figures 10A and 10B show the experimental estimator 2.0 (hybrid model 24) and linear, high-density, and ARIMA models implemented in microassembler software using frugally-deep (cpp). Figure 10A shows the mean prediction error rate reduced by camera pixels >1 for the boosted RNN (LSTM) Arima model compared to the improved estimator 2.0 model. Figure 10B shows implementations using deep and RNN models operating faster than 60Hz (internal control cycle clock) to predict the positions of <300 particles (micro-objects 11), although not scaled up to 1000 particles in this particular experiment.

[0055] While the present invention has been shown and described in particular with reference to its embodiments, those skilled in the art will understand that the aforementioned and other modifications of its forms and details can be made therein without departing from the spirit and scope of the invention.

Claims

1. 1. A method for machine learning enabled microassembly control using a digital computer, comprising: acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects resulting from previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the acts; building a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations using the training data via gradient boosting, the hybrid model comprising a plurality of models including one or more of the physics-based models and one or more machine learning models; obtaining a dataset; splitting the data set into the training data and an interpolated test data; extrapolating said previous motion states to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; If the hybrid model passes the test, controlling the closed loop system, predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the micro objects; and performing another one of said operations using the predicted position and the further position; A method wherein said steps are performed by a suitably programmed computer.

2. The gradient boosting includes a plurality of stages, one of the models being constructed during each of the stages, and the method further comprises: for each of the models, using a loss function to determine a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; 2. The method of claim 1, further comprising: for each of the models other than the model constructed during a first one of the stages, defining that model using the residuals associated with the model constructed during the stage prior to the stage in which that model is constructed.

3. 3. The method of claim 2, further comprising: for each of the models based on the stage associated with that model, selecting the loss function used to determine the residual for each of the models based on a probabilistic model.

4. The method of claim 3 , wherein the loss function is a decorrelation function.

5. The method of claim 4 , wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

6. 5. The method of claim 4, wherein the model constructed during the first stage is one of the physics-based models and the model during the last one of the stages is one of the machine learning models.

7. The method of claim 1 , wherein the one or more processors define a recurrent neural network.

8. The method of claim 7 , wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

9. The method of claim 1 , wherein the computer includes at least one of a central processing unit and a graphics processing unit.

10. 1. A system for machine learning enabled microassembly control using a digital computer, comprising: At least one processor, acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects due to previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the acts; building a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations using the training data via gradient boosting, the hybrid model including a plurality of models including one or more of the physics-based models and one or more machine learning models; Obtaining a dataset; splitting the data set into the training data and interpolated test data; extrapolating the previous motion states to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; if the hybrid model passes the test, controlling the closed loop system; predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the micro objects; and performing another one of said operations using the predicted position and the further position.

11. The gradient boosting includes a plurality of stages, one of the models is constructed during each of the stages, and the at least one processor performs: for each of the models, using a loss function to determine a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; 11. The system of claim 10, further configured to: for each of the models other than the model constructed during a first one of the stages, define that model using the residuals associated with the model constructed during the stage prior to the stage in which that model is constructed.

12. The at least one processor:

12. The system of claim 11, further configured to select, for each of the models based on the stage associated with that model, the loss function used to determine the residual for each of the models based on a probabilistic model.

13. The system of claim 12 , wherein the loss function is a decorrelation function.

14. The system of claim 13, wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

15. 14. The system of claim 13, wherein the model constructed during the first stage is one of the physics-based models and the model during the last one of the stages is one of the machine learning models.

16. The system of claim 10 , wherein the one or more processors define a recurrent neural network.

17. 20. The system of claim 16, wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

18. The system of claim 10 , wherein the at least one processor is at least one of a central processing unit and a graphics processing unit.

19. A system for machine learning enabled microassembly control using a digital computer, comprising: At least one processor, acquiring one or more parameters of a closed-loop system for positioning one or more of a plurality of micro-objects, the system including a plurality of programmable electrodes configured to induce movement of the micro-object when the micro-object is in proximity to the electrodes upon actuation of the electrodes, each of the actuations including generation of one or more electrical potentials by one or more of the electrodes; acquiring training data including states associated with previous movements of one or more of the plurality of micro-objects due to previous ones of the operations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the acts; building a hybrid model for predicting a position of at least one of the micro-objects after one of the acts using the training data via gradient boosting, the hybrid model including a plurality of models including one or more of the physics-based models and one or more machine learning models, the gradient boosting including a plurality of stages, one of the models being built during each of the stages; at least one processor configured to: The at least one processor further comprises: selecting, for each of the models based on the stage associated with the models, a loss function including a decorrelation function based on a probabilistic model to determine the residuals of each of the models; for each of the models, using the loss function to determine the residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; for each of the models other than the model constructed during a first one of the stages, defining that model using the residuals associated with the model constructed during the stage prior to the stage in which the model is constructed; predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving additional positions of one or more of the micro objects; and performing another one of said operations using the predicted position and the further position.

Images